2026-02-24T17:53:39.547 INFO:root:teuthology version: 1.2.4.dev9+gfa17720d0 2026-02-24T17:53:39.567 DEBUG:teuthology.report:Pushing job info to https://paddles-paddles.apps.pok.os.sepia.ceph.com 2026-02-24T17:53:39.651 INFO:teuthology.run:Config: archive_path: /home/teuthworker/mnt/teuthology/yuriw-2026-02-24_16:03:54-rados-wip-yuri8-testing-2026-02-23-2128-distro-default-trial/68571 branch: wip-yuri8-testing-2026-02-23-2128 description: rados/singleton-nomsgr/{all/health-warnings mon_election/connectivity rados supported-random-distro$/{ubuntu_latest}} email: yweinste@redhat.com first_in_suite: false flavor: default job_id: '68571' kernel: branch: distro kdb: true sha1: distro ktype: distro last_in_suite: false machine_type: trial name: yuriw-2026-02-24_16:03:54-rados-wip-yuri8-testing-2026-02-23-2128-distro-default-trial no_nested_subset: false os_type: ubuntu os_version: '22.04' overrides: admin_socket: branch: wip-yuri8-testing-2026-02-23-2128 ceph: conf: global: mon election default strategy: 3 mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 30 debug ms: 1 debug paxos: 20 mon scrub interval: 300 osd: bluestore zero block detection: true debug ms: 1 debug osd: 20 osd debug verify cached snaps: true osd debug verify missing on start: true osd mclock override recovery settings: true osd mclock profile: high_recovery_ops osd mclock skip benchmark: true osd op queue: debug_random osd op queue cut off: debug_random flavor: default log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) sha1: 618c318a7a0b287ab5aeff207b7bf1d73ef45044 ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} install: ceph: flavor: default sha1: 618c318a7a0b287ab5aeff207b7bf1d73ef45044 workunit: branch: wip-yuri8-testing-2026-02-23-2128 sha1: 618c318a7a0b287ab5aeff207b7bf1d73ef45044 owner: scheduled_yuriw@soko04.front.sepia.ceph.com priority: 69 repo: https://github.com/ceph/ceph-ci.git roles: - - mon.a - mgr.x - osd.0 - osd.1 - osd.2 - osd.3 - osd.4 - osd.5 - osd.6 - osd.7 - osd.8 - osd.9 - client.0 seed: 6922 sha1: 618c318a7a0b287ab5aeff207b7bf1d73ef45044 sleep_before_teardown: 0 subset: 111/120000 suite: rados suite_branch: wip-yuri8-testing-2026-02-23-2128 suite_path: /home/teuthworker/src/github.com_ceph_ceph-c_618c318a7a0b287ab5aeff207b7bf1d73ef45044/qa suite_relpath: qa suite_repo: https://github.com/ceph/ceph-ci.git suite_sha1: 618c318a7a0b287ab5aeff207b7bf1d73ef45044 targets: trial068.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBJET0we95Ttlwr4RnDY93t7VgPcOzA7uWELNz2LQ0M2zzXpkPpEjMFiROkvBrnMVkgRB6NOSCaSPDtjriH8+gbE= tasks: - install: null - ceph: conf: osd: osd max object name len: 400 osd max object namespace len: 64 log-ignorelist: - but it is still running - overall HEALTH_ - \(OSDMAP_FLAGS\) - \(OSD_ - \(PG_ - \(POOL_APP_NOT_ENABLED\) pre-mgr-commands: - sudo ceph config set mgr mgr_pool false --force - workunit: clients: all: - rados/test_health_warnings.sh teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: fa17720d0088c3ac28e473468bfc79eeaff5cd38 timestamp: 2026-02-24_16:03:54 tube: trial user: yuriw verbose: true worker_log: /home/teuthworker/mnt/teuthology/worker_logs/dispatcher.trial.904699 2026-02-24T17:53:39.651 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/github.com_ceph_ceph-c_618c318a7a0b287ab5aeff207b7bf1d73ef45044/qa; will attempt to use it 2026-02-24T17:53:39.651 INFO:teuthology.run:Found tasks at /home/teuthworker/src/github.com_ceph_ceph-c_618c318a7a0b287ab5aeff207b7bf1d73ef45044/qa/tasks 2026-02-24T17:53:39.652 INFO:teuthology.run_tasks:Running task internal.check_packages... 2026-02-24T17:53:39.653 INFO:teuthology.task.internal:Checking packages... 2026-02-24T17:53:39.653 INFO:teuthology.task.internal:Checking packages for os_type 'ubuntu', flavor 'default' and ceph hash '618c318a7a0b287ab5aeff207b7bf1d73ef45044' 2026-02-24T17:53:39.653 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2026-02-24T17:53:39.653 INFO:teuthology.packaging:ref: None 2026-02-24T17:53:39.653 INFO:teuthology.packaging:tag: None 2026-02-24T17:53:39.653 INFO:teuthology.packaging:branch: wip-yuri8-testing-2026-02-23-2128 2026-02-24T17:53:39.653 INFO:teuthology.packaging:sha1: 618c318a7a0b287ab5aeff207b7bf1d73ef45044 2026-02-24T17:53:39.653 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&ref=wip-yuri8-testing-2026-02-23-2128 2026-02-24T17:53:39.771 INFO:teuthology.task.internal:Found packages for ceph version 20.3.0-5507-g618c318a-1jammy 2026-02-24T17:53:39.772 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2026-02-24T17:53:39.784 INFO:teuthology.task.internal:no buildpackages task found 2026-02-24T17:53:39.784 INFO:teuthology.run_tasks:Running task internal.save_config... 2026-02-24T17:53:39.798 INFO:teuthology.task.internal:Saving configuration 2026-02-24T17:53:39.809 INFO:teuthology.run_tasks:Running task internal.check_lock... 2026-02-24T17:53:39.821 INFO:teuthology.task.internal.check_lock:Checking locks... 2026-02-24T17:53:39.900 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'trial068.front.sepia.ceph.com', 'description': '/home/teuthworker/mnt/teuthology/yuriw-2026-02-24_16:03:54-rados-wip-yuri8-testing-2026-02-23-2128-distro-default-trial/68571', 'up': True, 'machine_type': 'trial', 'is_vm': False, 'vm_host': None, 'os_type': 'ubuntu', 'os_version': '22.04', 'arch': 'x86_64', 'locked': True, 'locked_since': '2026-02-24 17:50:05.816782', 'locked_by': 'scheduled_yuriw@soko04.front.sepia.ceph.com', 'mac_address': None, 'ssh_pub_key': 'ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIKHWvg//Nihw+iEdN1PmNblafXs9XLpmPgXopONP248h'} 2026-02-24T17:53:39.900 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2026-02-24T17:53:39.923 INFO:teuthology.task.internal:roles: ubuntu@trial068.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'osd.8', 'osd.9', 'client.0'] 2026-02-24T17:53:39.923 INFO:teuthology.run_tasks:Running task console_log... 2026-02-24T17:53:40.011 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x77b869c00d60>, signals=[15]) 2026-02-24T17:53:40.012 INFO:teuthology.run_tasks:Running task internal.connect... 2026-02-24T17:53:40.025 INFO:teuthology.task.internal:Opening connections... 2026-02-24T17:53:40.025 DEBUG:teuthology.task.internal:connecting to ubuntu@trial068.front.sepia.ceph.com 2026-02-24T17:53:40.026 DEBUG:teuthology.orchestra.connection:{'hostname': 'trial068.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2026-02-24T17:53:40.090 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2026-02-24T17:53:40.105 DEBUG:teuthology.orchestra.run.trial068:> uname -m 2026-02-24T17:53:40.168 INFO:teuthology.orchestra.run.trial068.stdout:x86_64 2026-02-24T17:53:40.168 DEBUG:teuthology.orchestra.run.trial068:> cat /etc/os-release 2026-02-24T17:53:40.213 INFO:teuthology.orchestra.run.trial068.stdout:PRETTY_NAME="Ubuntu 22.04.5 LTS" 2026-02-24T17:53:40.213 INFO:teuthology.orchestra.run.trial068.stdout:NAME="Ubuntu" 2026-02-24T17:53:40.213 INFO:teuthology.orchestra.run.trial068.stdout:VERSION_ID="22.04" 2026-02-24T17:53:40.213 INFO:teuthology.orchestra.run.trial068.stdout:VERSION="22.04.5 LTS (Jammy Jellyfish)" 2026-02-24T17:53:40.214 INFO:teuthology.orchestra.run.trial068.stdout:VERSION_CODENAME=jammy 2026-02-24T17:53:40.214 INFO:teuthology.orchestra.run.trial068.stdout:ID=ubuntu 2026-02-24T17:53:40.214 INFO:teuthology.orchestra.run.trial068.stdout:ID_LIKE=debian 2026-02-24T17:53:40.214 INFO:teuthology.orchestra.run.trial068.stdout:HOME_URL="https://www.ubuntu.com/" 2026-02-24T17:53:40.214 INFO:teuthology.orchestra.run.trial068.stdout:SUPPORT_URL="https://help.ubuntu.com/" 2026-02-24T17:53:40.214 INFO:teuthology.orchestra.run.trial068.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" 2026-02-24T17:53:40.214 INFO:teuthology.orchestra.run.trial068.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" 2026-02-24T17:53:40.214 INFO:teuthology.orchestra.run.trial068.stdout:UBUNTU_CODENAME=jammy 2026-02-24T17:53:40.215 INFO:teuthology.lock.ops:Updating trial068.front.sepia.ceph.com on lock server 2026-02-24T17:53:40.328 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2026-02-24T17:53:40.428 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2026-02-24T17:53:40.480 INFO:teuthology.task.internal:Checking for old test directory... 2026-02-24T17:53:40.480 DEBUG:teuthology.orchestra.run.trial068:> test '!' -e /home/ubuntu/cephtest 2026-02-24T17:53:40.484 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2026-02-24T17:53:40.501 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2026-02-24T17:53:40.502 DEBUG:teuthology.orchestra.run.trial068:> test -z $(ls -A /var/lib/ceph) 2026-02-24T17:53:40.530 INFO:teuthology.orchestra.run.trial068.stderr:ls: cannot access '/var/lib/ceph': No such file or directory 2026-02-24T17:53:40.530 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2026-02-24T17:53:40.677 INFO:teuthology.run_tasks:Running task kernel... 2026-02-24T17:53:40.697 INFO:teuthology.task.kernel:normalize config orig: {'branch': 'distro', 'kdb': True, 'sha1': 'distro'} 2026-02-24T17:53:40.697 INFO:teuthology.task.kernel:config {'mon.a': {'branch': 'distro', 'kdb': True, 'sha1': 'distro'}}, timeout 300 2026-02-24T17:53:40.698 DEBUG:teuthology.orchestra.run.trial068:> test -f /run/.containerenv -o -f /.dockerenv 2026-02-24T17:53:40.701 DEBUG:teuthology.orchestra.run:got remote process result: 1 2026-02-24T17:53:40.701 DEBUG:teuthology.orchestra.run.trial068:> uname -r 2026-02-24T17:53:40.745 INFO:teuthology.orchestra.run.trial068.stdout:5.15.0-170-generic 2026-02-24T17:53:40.746 INFO:teuthology.task.kernel:Running kernel on trial068: 5.15.0-170-generic 2026-02-24T17:53:40.746 DEBUG:teuthology.orchestra.run.trial068:> sudo apt-get clean 2026-02-24T17:53:40.804 DEBUG:teuthology.orchestra.run.trial068:> sudo apt-get update 2026-02-24T17:53:41.298 INFO:teuthology.orchestra.run.trial068.stdout:Hit:1 https://archive.ubuntu.com/ubuntu jammy InRelease 2026-02-24T17:53:41.388 INFO:teuthology.orchestra.run.trial068.stdout:Get:2 https://archive.ubuntu.com/ubuntu jammy-updates InRelease [128 kB] 2026-02-24T17:53:41.725 INFO:teuthology.orchestra.run.trial068.stdout:Get:3 https://archive.ubuntu.com/ubuntu jammy-security InRelease [129 kB] 2026-02-24T17:53:41.873 INFO:teuthology.orchestra.run.trial068.stdout:Get:4 https://archive.ubuntu.com/ubuntu jammy-backports InRelease [127 kB] 2026-02-24T17:53:42.012 INFO:teuthology.orchestra.run.trial068.stdout:Get:5 https://archive.ubuntu.com/ubuntu jammy-updates/main amd64 Packages [3269 kB] 2026-02-24T17:53:42.413 INFO:teuthology.orchestra.run.trial068.stdout:Get:6 https://archive.ubuntu.com/ubuntu jammy-updates/main Translation-en [497 kB] 2026-02-24T17:53:42.425 INFO:teuthology.orchestra.run.trial068.stdout:Get:7 https://archive.ubuntu.com/ubuntu jammy-updates/main amd64 c-n-f Metadata [19.2 kB] 2026-02-24T17:53:42.425 INFO:teuthology.orchestra.run.trial068.stdout:Get:8 https://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 Packages [1256 kB] 2026-02-24T17:53:42.460 INFO:teuthology.orchestra.run.trial068.stdout:Get:9 https://archive.ubuntu.com/ubuntu jammy-updates/universe Translation-en [314 kB] 2026-02-24T17:53:42.467 INFO:teuthology.orchestra.run.trial068.stdout:Get:10 https://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 c-n-f Metadata [30.4 kB] 2026-02-24T17:53:42.468 INFO:teuthology.orchestra.run.trial068.stdout:Get:11 https://archive.ubuntu.com/ubuntu jammy-updates/restricted amd64 Packages [5379 kB] 2026-02-24T17:53:42.625 INFO:teuthology.orchestra.run.trial068.stdout:Get:12 https://archive.ubuntu.com/ubuntu jammy-updates/restricted Translation-en [1022 kB] 2026-02-24T17:53:42.648 INFO:teuthology.orchestra.run.trial068.stdout:Get:13 https://archive.ubuntu.com/ubuntu jammy-security/main amd64 Packages [3005 kB] 2026-02-24T17:53:42.734 INFO:teuthology.orchestra.run.trial068.stdout:Get:14 https://archive.ubuntu.com/ubuntu jammy-security/main Translation-en [429 kB] 2026-02-24T17:53:42.745 INFO:teuthology.orchestra.run.trial068.stdout:Get:15 https://archive.ubuntu.com/ubuntu jammy-security/main amd64 c-n-f Metadata [14.1 kB] 2026-02-24T17:53:42.745 INFO:teuthology.orchestra.run.trial068.stdout:Get:16 https://archive.ubuntu.com/ubuntu jammy-security/universe amd64 Packages [1018 kB] 2026-02-24T17:53:42.779 INFO:teuthology.orchestra.run.trial068.stdout:Get:17 https://archive.ubuntu.com/ubuntu jammy-security/universe Translation-en [225 kB] 2026-02-24T17:53:42.787 INFO:teuthology.orchestra.run.trial068.stdout:Get:18 https://archive.ubuntu.com/ubuntu jammy-security/universe amd64 c-n-f Metadata [22.7 kB] 2026-02-24T17:53:42.787 INFO:teuthology.orchestra.run.trial068.stdout:Get:19 https://archive.ubuntu.com/ubuntu jammy-security/restricted amd64 Packages [5204 kB] 2026-02-24T17:53:42.936 INFO:teuthology.orchestra.run.trial068.stdout:Get:20 https://archive.ubuntu.com/ubuntu jammy-security/restricted Translation-en [990 kB] 2026-02-24T17:53:46.471 INFO:teuthology.orchestra.run.trial068.stdout:Fetched 23.1 MB in 3s (7835 kB/s) 2026-02-24T17:53:47.012 INFO:teuthology.orchestra.run.trial068.stdout:Reading package lists... 2026-02-24T17:53:47.025 DEBUG:teuthology.orchestra.run.trial068:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2026-02-24T17:53:47.059 INFO:teuthology.orchestra.run.trial068.stdout:Reading package lists... 2026-02-24T17:53:47.172 INFO:teuthology.orchestra.run.trial068.stdout:Building dependency tree... 2026-02-24T17:53:47.172 INFO:teuthology.orchestra.run.trial068.stdout:Reading state information... 2026-02-24T17:53:47.261 INFO:teuthology.orchestra.run.trial068.stdout:The following packages were automatically installed and are no longer required: 2026-02-24T17:53:47.261 INFO:teuthology.orchestra.run.trial068.stdout: kpartx libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 2026-02-24T17:53:47.261 INFO:teuthology.orchestra.run.trial068.stdout: libgfxdr0 libglusterfs0 libiscsi7 libpmemobj1 libsgutils2-2 sg3-utils 2026-02-24T17:53:47.261 INFO:teuthology.orchestra.run.trial068.stdout: sg3-utils-udev 2026-02-24T17:53:47.261 INFO:teuthology.orchestra.run.trial068.stdout:Use 'sudo apt autoremove' to remove them. 2026-02-24T17:53:47.261 INFO:teuthology.orchestra.run.trial068.stdout:The following additional packages will be installed: 2026-02-24T17:53:47.261 INFO:teuthology.orchestra.run.trial068.stdout: linux-generic linux-headers-5.15.0-171 linux-headers-5.15.0-171-generic 2026-02-24T17:53:47.261 INFO:teuthology.orchestra.run.trial068.stdout: linux-headers-generic linux-image-5.15.0-171-generic 2026-02-24T17:53:47.261 INFO:teuthology.orchestra.run.trial068.stdout: linux-modules-5.15.0-171-generic linux-modules-extra-5.15.0-171-generic 2026-02-24T17:53:47.261 INFO:teuthology.orchestra.run.trial068.stdout:Suggested packages: 2026-02-24T17:53:47.261 INFO:teuthology.orchestra.run.trial068.stdout: fdutils linux-doc | linux-source-5.15.0 linux-tools 2026-02-24T17:53:47.276 INFO:teuthology.orchestra.run.trial068.stdout:The following NEW packages will be installed: 2026-02-24T17:53:47.276 INFO:teuthology.orchestra.run.trial068.stdout: linux-headers-5.15.0-171 linux-headers-5.15.0-171-generic 2026-02-24T17:53:47.276 INFO:teuthology.orchestra.run.trial068.stdout: linux-image-5.15.0-171-generic linux-modules-5.15.0-171-generic 2026-02-24T17:53:47.276 INFO:teuthology.orchestra.run.trial068.stdout: linux-modules-extra-5.15.0-171-generic 2026-02-24T17:53:47.277 INFO:teuthology.orchestra.run.trial068.stdout:The following packages will be upgraded: 2026-02-24T17:53:47.277 INFO:teuthology.orchestra.run.trial068.stdout: linux-generic linux-headers-generic linux-image-generic 2026-02-24T17:53:47.374 INFO:teuthology.orchestra.run.trial068.stdout:3 upgraded, 5 newly installed, 0 to remove and 14 not upgraded. 2026-02-24T17:53:47.374 INFO:teuthology.orchestra.run.trial068.stdout:Need to get 113 MB of archives. 2026-02-24T17:53:47.374 INFO:teuthology.orchestra.run.trial068.stdout:After this operation, 585 MB of additional disk space will be used. 2026-02-24T17:53:47.374 INFO:teuthology.orchestra.run.trial068.stdout:Get:1 https://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-modules-5.15.0-171-generic amd64 5.15.0-171.181 [22.7 MB] 2026-02-24T17:53:47.558 INFO:teuthology.orchestra.run.trial068.stdout:Get:2 https://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-image-5.15.0-171-generic amd64 5.15.0-171.181 [11.6 MB] 2026-02-24T17:53:47.602 INFO:teuthology.orchestra.run.trial068.stdout:Get:3 https://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-modules-extra-5.15.0-171-generic amd64 5.15.0-171.181 [63.9 MB] 2026-02-24T17:53:47.843 INFO:teuthology.orchestra.run.trial068.stdout:Get:4 https://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-generic amd64 5.15.0.171.160 [1658 B] 2026-02-24T17:53:47.843 INFO:teuthology.orchestra.run.trial068.stdout:Get:5 https://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-image-generic amd64 5.15.0.171.160 [2486 B] 2026-02-24T17:53:47.843 INFO:teuthology.orchestra.run.trial068.stdout:Get:6 https://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-headers-5.15.0-171 all 5.15.0-171.181 [12.4 MB] 2026-02-24T17:53:47.893 INFO:teuthology.orchestra.run.trial068.stdout:Get:7 https://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-headers-5.15.0-171-generic amd64 5.15.0-171.181 [2849 kB] 2026-02-24T17:53:47.903 INFO:teuthology.orchestra.run.trial068.stdout:Get:8 https://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-headers-generic amd64 5.15.0.171.160 [2332 B] 2026-02-24T17:53:48.102 INFO:teuthology.orchestra.run.trial068.stdout:Fetched 113 MB in 1s (182 MB/s) 2026-02-24T17:53:48.117 INFO:teuthology.orchestra.run.trial068.stdout:Selecting previously unselected package linux-modules-5.15.0-171-generic. 2026-02-24T17:53:48.289 INFO:teuthology.orchestra.run.trial068.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 167442 files and directories currently installed.) 2026-02-24T17:53:48.291 INFO:teuthology.orchestra.run.trial068.stdout:Preparing to unpack .../0-linux-modules-5.15.0-171-generic_5.15.0-171.181_amd64.deb ... 2026-02-24T17:53:48.293 INFO:teuthology.orchestra.run.trial068.stdout:Unpacking linux-modules-5.15.0-171-generic (5.15.0-171.181) ... 2026-02-24T17:53:48.636 INFO:teuthology.orchestra.run.trial068.stdout:Selecting previously unselected package linux-image-5.15.0-171-generic. 2026-02-24T17:53:48.645 INFO:teuthology.orchestra.run.trial068.stdout:Preparing to unpack .../1-linux-image-5.15.0-171-generic_5.15.0-171.181_amd64.deb ... 2026-02-24T17:53:48.663 INFO:teuthology.orchestra.run.trial068.stdout:Unpacking linux-image-5.15.0-171-generic (5.15.0-171.181) ... 2026-02-24T17:53:48.705 INFO:teuthology.orchestra.run.trial068.stdout:Selecting previously unselected package linux-modules-extra-5.15.0-171-generic. 2026-02-24T17:53:48.713 INFO:teuthology.orchestra.run.trial068.stdout:Preparing to unpack .../2-linux-modules-extra-5.15.0-171-generic_5.15.0-171.181_amd64.deb ... 2026-02-24T17:53:48.713 INFO:teuthology.orchestra.run.trial068.stdout:Unpacking linux-modules-extra-5.15.0-171-generic (5.15.0-171.181) ... 2026-02-24T17:53:49.819 INFO:teuthology.orchestra.run.trial068.stdout:Preparing to unpack .../3-linux-generic_5.15.0.171.160_amd64.deb ... 2026-02-24T17:53:49.820 INFO:teuthology.orchestra.run.trial068.stdout:Unpacking linux-generic (5.15.0.171.160) over (5.15.0.170.159) ... 2026-02-24T17:53:49.846 INFO:teuthology.orchestra.run.trial068.stdout:Preparing to unpack .../4-linux-image-generic_5.15.0.171.160_amd64.deb ... 2026-02-24T17:53:49.847 INFO:teuthology.orchestra.run.trial068.stdout:Unpacking linux-image-generic (5.15.0.171.160) over (5.15.0.170.159) ... 2026-02-24T17:53:49.863 INFO:teuthology.orchestra.run.trial068.stdout:Selecting previously unselected package linux-headers-5.15.0-171. 2026-02-24T17:53:49.872 INFO:teuthology.orchestra.run.trial068.stdout:Preparing to unpack .../5-linux-headers-5.15.0-171_5.15.0-171.181_all.deb ... 2026-02-24T17:53:49.873 INFO:teuthology.orchestra.run.trial068.stdout:Unpacking linux-headers-5.15.0-171 (5.15.0-171.181) ... 2026-02-24T17:53:51.195 INFO:teuthology.orchestra.run.trial068.stdout:Selecting previously unselected package linux-headers-5.15.0-171-generic. 2026-02-24T17:53:51.209 INFO:teuthology.orchestra.run.trial068.stdout:Preparing to unpack .../6-linux-headers-5.15.0-171-generic_5.15.0-171.181_amd64.deb ... 2026-02-24T17:53:51.209 INFO:teuthology.orchestra.run.trial068.stdout:Unpacking linux-headers-5.15.0-171-generic (5.15.0-171.181) ... 2026-02-24T17:53:51.805 INFO:teuthology.orchestra.run.trial068.stdout:Preparing to unpack .../7-linux-headers-generic_5.15.0.171.160_amd64.deb ... 2026-02-24T17:53:51.805 INFO:teuthology.orchestra.run.trial068.stdout:Unpacking linux-headers-generic (5.15.0.171.160) over (5.15.0.170.159) ... 2026-02-24T17:53:51.831 INFO:teuthology.orchestra.run.trial068.stdout:Setting up linux-headers-5.15.0-171 (5.15.0-171.181) ... 2026-02-24T17:53:51.833 INFO:teuthology.orchestra.run.trial068.stdout:Setting up linux-headers-5.15.0-171-generic (5.15.0-171.181) ... 2026-02-24T17:53:51.834 INFO:teuthology.orchestra.run.trial068.stdout:Setting up linux-headers-generic (5.15.0.171.160) ... 2026-02-24T17:53:51.836 INFO:teuthology.orchestra.run.trial068.stdout:Setting up linux-image-5.15.0-171-generic (5.15.0-171.181) ... 2026-02-24T17:53:52.630 INFO:teuthology.orchestra.run.trial068.stdout:I: /boot/vmlinuz.old is now a symlink to vmlinuz-5.15.0-170-generic 2026-02-24T17:53:52.630 INFO:teuthology.orchestra.run.trial068.stdout:I: /boot/initrd.img.old is now a symlink to initrd.img-5.15.0-170-generic 2026-02-24T17:53:52.630 INFO:teuthology.orchestra.run.trial068.stdout:I: /boot/vmlinuz is now a symlink to vmlinuz-5.15.0-171-generic 2026-02-24T17:53:52.630 INFO:teuthology.orchestra.run.trial068.stdout:I: /boot/initrd.img is now a symlink to initrd.img-5.15.0-171-generic 2026-02-24T17:53:52.637 INFO:teuthology.orchestra.run.trial068.stdout:Setting up linux-modules-extra-5.15.0-171-generic (5.15.0-171.181) ... 2026-02-24T17:53:53.461 INFO:teuthology.orchestra.run.trial068.stdout:Setting up linux-modules-5.15.0-171-generic (5.15.0-171.181) ... 2026-02-24T17:53:54.274 INFO:teuthology.orchestra.run.trial068.stdout:Setting up linux-image-generic (5.15.0.171.160) ... 2026-02-24T17:53:54.275 INFO:teuthology.orchestra.run.trial068.stdout:Setting up linux-generic (5.15.0.171.160) ... 2026-02-24T17:53:54.276 INFO:teuthology.orchestra.run.trial068.stdout:Processing triggers for linux-image-5.15.0-171-generic (5.15.0-171.181) ... 2026-02-24T17:53:54.282 INFO:teuthology.orchestra.run.trial068.stdout:/etc/kernel/postinst.d/initramfs-tools: 2026-02-24T17:53:54.282 INFO:teuthology.orchestra.run.trial068.stdout:update-initramfs: Generating /boot/initrd.img-5.15.0-171-generic 2026-02-24T17:54:01.389 INFO:teuthology.orchestra.run.trial068.stdout:W: Possible missing firmware /lib/firmware/ast_dp501_fw.bin for module ast 2026-02-24T17:54:08.022 INFO:teuthology.orchestra.run.trial068.stdout:/etc/kernel/postinst.d/zz-update-grub: 2026-02-24T17:54:08.022 INFO:teuthology.orchestra.run.trial068.stdout:Sourcing file `/etc/default/grub' 2026-02-24T17:54:08.042 INFO:teuthology.orchestra.run.trial068.stdout:Sourcing file `/etc/default/grub.d/50-curtin-settings.cfg' 2026-02-24T17:54:08.043 INFO:teuthology.orchestra.run.trial068.stdout:Sourcing file `/etc/default/grub.d/init-select.cfg' 2026-02-24T17:54:08.046 INFO:teuthology.orchestra.run.trial068.stdout:Generating grub configuration file ... 2026-02-24T17:54:08.129 INFO:teuthology.orchestra.run.trial068.stdout:Found linux image: /boot/vmlinuz-5.15.0-171-generic 2026-02-24T17:54:08.136 INFO:teuthology.orchestra.run.trial068.stdout:Found initrd image: /boot/initrd.img-5.15.0-171-generic 2026-02-24T17:54:08.312 INFO:teuthology.orchestra.run.trial068.stdout:Found linux image: /boot/vmlinuz-5.15.0-170-generic 2026-02-24T17:54:08.317 INFO:teuthology.orchestra.run.trial068.stdout:Found initrd image: /boot/initrd.img-5.15.0-170-generic 2026-02-24T17:54:08.361 INFO:teuthology.orchestra.run.trial068.stdout:Found linux image: /boot/vmlinuz-5.15.0-164-generic 2026-02-24T17:54:08.367 INFO:teuthology.orchestra.run.trial068.stdout:Found initrd image: /boot/initrd.img-5.15.0-164-generic 2026-02-24T17:54:08.439 INFO:teuthology.orchestra.run.trial068.stdout:Warning: os-prober will not be executed to detect other bootable partitions. 2026-02-24T17:54:08.439 INFO:teuthology.orchestra.run.trial068.stdout:Systems on them will not be added to the GRUB boot configuration. 2026-02-24T17:54:08.439 INFO:teuthology.orchestra.run.trial068.stdout:Check GRUB_DISABLE_OS_PROBER documentation entry. 2026-02-24T17:54:08.441 INFO:teuthology.orchestra.run.trial068.stdout:Adding boot menu entry for UEFI Firmware Settings ... 2026-02-24T17:54:08.451 INFO:teuthology.orchestra.run.trial068.stdout:done 2026-02-24T17:54:08.863 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-24T17:54:08.863 INFO:teuthology.orchestra.run.trial068.stdout:Pending kernel upgrade! 2026-02-24T17:54:08.863 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-24T17:54:08.863 INFO:teuthology.orchestra.run.trial068.stdout:Running kernel version: 2026-02-24T17:54:08.863 INFO:teuthology.orchestra.run.trial068.stdout: 5.15.0-170-generic 2026-02-24T17:54:08.863 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-24T17:54:08.863 INFO:teuthology.orchestra.run.trial068.stdout:Diagnostics: 2026-02-24T17:54:08.863 INFO:teuthology.orchestra.run.trial068.stdout: The currently running kernel version is not the expected kernel version 5.15.0-171-generic. 2026-02-24T17:54:08.863 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-24T17:54:08.863 INFO:teuthology.orchestra.run.trial068.stdout:Restarting the system to load the new kernel will not be handled automatically, so you should consider rebooting. [Return] 2026-02-24T17:54:08.863 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-24T17:54:08.863 INFO:teuthology.orchestra.run.trial068.stdout:The processor microcode seems to be up-to-date. 2026-02-24T17:54:08.863 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-24T17:54:08.863 INFO:teuthology.orchestra.run.trial068.stdout:No services need to be restarted. 2026-02-24T17:54:08.863 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-24T17:54:08.864 INFO:teuthology.orchestra.run.trial068.stdout:No containers need to be restarted. 2026-02-24T17:54:08.864 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-24T17:54:08.864 INFO:teuthology.orchestra.run.trial068.stdout:No user sessions are running outdated binaries. 2026-02-24T17:54:08.864 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-24T17:54:08.864 INFO:teuthology.orchestra.run.trial068.stdout:No VM guests are running outdated hypervisor (qemu) binaries on this host. 2026-02-24T17:54:09.710 DEBUG:teuthology.orchestra.run.trial068:> dpkg -s linux-image-generic 2026-02-24T17:54:09.721 INFO:teuthology.orchestra.run.trial068.stdout:Package: linux-image-generic 2026-02-24T17:54:09.721 INFO:teuthology.orchestra.run.trial068.stdout:Status: install ok installed 2026-02-24T17:54:09.721 INFO:teuthology.orchestra.run.trial068.stdout:Priority: optional 2026-02-24T17:54:09.721 INFO:teuthology.orchestra.run.trial068.stdout:Section: kernel 2026-02-24T17:54:09.721 INFO:teuthology.orchestra.run.trial068.stdout:Installed-Size: 22 2026-02-24T17:54:09.722 INFO:teuthology.orchestra.run.trial068.stdout:Maintainer: Ubuntu Kernel Team 2026-02-24T17:54:09.722 INFO:teuthology.orchestra.run.trial068.stdout:Architecture: amd64 2026-02-24T17:54:09.722 INFO:teuthology.orchestra.run.trial068.stdout:Source: linux-meta 2026-02-24T17:54:09.722 INFO:teuthology.orchestra.run.trial068.stdout:Version: 5.15.0.171.160 2026-02-24T17:54:09.722 INFO:teuthology.orchestra.run.trial068.stdout:Provides: spl-modules (= 2.1.5-1ubuntu6~22.04.6), v4l2loopback-modules (= 0.12.7-2ubuntu2~22.04.1), virtualbox-guest-modules (= 5.15.0-171), wireguard-modules (= 1.0.0), zfs-modules (= 2.1.5-1ubuntu6~22.04.6) 2026-02-24T17:54:09.722 INFO:teuthology.orchestra.run.trial068.stdout:Depends: linux-image-5.15.0-171-generic, linux-modules-extra-5.15.0-171-generic, linux-firmware, intel-microcode, amd64-microcode 2026-02-24T17:54:09.722 INFO:teuthology.orchestra.run.trial068.stdout:Recommends: thermald 2026-02-24T17:54:09.722 INFO:teuthology.orchestra.run.trial068.stdout:Description: Generic Linux kernel image 2026-02-24T17:54:09.722 INFO:teuthology.orchestra.run.trial068.stdout: This package will always depend on the latest generic kernel image 2026-02-24T17:54:09.722 INFO:teuthology.orchestra.run.trial068.stdout: available. 2026-02-24T17:54:09.723 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.15.0-170-generic Expected: 5.15.0-171-generic 2026-02-24T17:54:09.723 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2026-02-24T17:54:09.723 DEBUG:teuthology.task.kernel:src is distro, skipping download 2026-02-24T17:54:09.723 INFO:teuthology.task.kernel:Installing distro kernel on mon.a... 2026-02-24T17:54:09.723 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@trial068.front.sepia.ceph.com, path=None, version=distro) 2026-02-24T17:54:09.723 DEBUG:teuthology.orchestra.run.trial068:> sudo apt-get clean 2026-02-24T17:54:09.801 DEBUG:teuthology.orchestra.run.trial068:> sudo apt-get update 2026-02-24T17:54:09.994 INFO:teuthology.orchestra.run.trial068.stdout:Hit:1 https://archive.ubuntu.com/ubuntu jammy InRelease 2026-02-24T17:54:09.997 INFO:teuthology.orchestra.run.trial068.stdout:Hit:2 https://archive.ubuntu.com/ubuntu jammy-updates InRelease 2026-02-24T17:54:10.006 INFO:teuthology.orchestra.run.trial068.stdout:Hit:3 https://archive.ubuntu.com/ubuntu jammy-security InRelease 2026-02-24T17:54:10.016 INFO:teuthology.orchestra.run.trial068.stdout:Hit:4 https://archive.ubuntu.com/ubuntu jammy-backports InRelease 2026-02-24T17:54:10.844 INFO:teuthology.orchestra.run.trial068.stdout:Reading package lists... 2026-02-24T17:54:10.858 DEBUG:teuthology.orchestra.run.trial068:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2026-02-24T17:54:10.891 INFO:teuthology.orchestra.run.trial068.stdout:Reading package lists... 2026-02-24T17:54:11.002 INFO:teuthology.orchestra.run.trial068.stdout:Building dependency tree... 2026-02-24T17:54:11.002 INFO:teuthology.orchestra.run.trial068.stdout:Reading state information... 2026-02-24T17:54:11.087 INFO:teuthology.orchestra.run.trial068.stdout:linux-image-generic is already the newest version (5.15.0.171.160). 2026-02-24T17:54:11.087 INFO:teuthology.orchestra.run.trial068.stdout:linux-image-generic set to manually installed. 2026-02-24T17:54:11.087 INFO:teuthology.orchestra.run.trial068.stdout:The following packages were automatically installed and are no longer required: 2026-02-24T17:54:11.088 INFO:teuthology.orchestra.run.trial068.stdout: kpartx libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 2026-02-24T17:54:11.088 INFO:teuthology.orchestra.run.trial068.stdout: libgfxdr0 libglusterfs0 libiscsi7 libpmemobj1 libsgutils2-2 2026-02-24T17:54:11.088 INFO:teuthology.orchestra.run.trial068.stdout: linux-headers-5.15.0-164 linux-headers-5.15.0-164-generic 2026-02-24T17:54:11.088 INFO:teuthology.orchestra.run.trial068.stdout: linux-image-5.15.0-164-generic linux-modules-5.15.0-164-generic 2026-02-24T17:54:11.088 INFO:teuthology.orchestra.run.trial068.stdout: linux-modules-extra-5.15.0-164-generic sg3-utils sg3-utils-udev 2026-02-24T17:54:11.088 INFO:teuthology.orchestra.run.trial068.stdout:Use 'sudo apt autoremove' to remove them. 2026-02-24T17:54:11.106 INFO:teuthology.orchestra.run.trial068.stdout:0 upgraded, 0 newly installed, 0 to remove and 14 not upgraded. 2026-02-24T17:54:11.108 DEBUG:teuthology.orchestra.run.trial068:> dpkg -s linux-image-generic 2026-02-24T17:54:11.159 INFO:teuthology.orchestra.run.trial068.stdout:Package: linux-image-generic 2026-02-24T17:54:11.159 INFO:teuthology.orchestra.run.trial068.stdout:Status: install ok installed 2026-02-24T17:54:11.159 INFO:teuthology.orchestra.run.trial068.stdout:Priority: optional 2026-02-24T17:54:11.159 INFO:teuthology.orchestra.run.trial068.stdout:Section: kernel 2026-02-24T17:54:11.159 INFO:teuthology.orchestra.run.trial068.stdout:Installed-Size: 22 2026-02-24T17:54:11.159 INFO:teuthology.orchestra.run.trial068.stdout:Maintainer: Ubuntu Kernel Team 2026-02-24T17:54:11.159 INFO:teuthology.orchestra.run.trial068.stdout:Architecture: amd64 2026-02-24T17:54:11.159 INFO:teuthology.orchestra.run.trial068.stdout:Source: linux-meta 2026-02-24T17:54:11.159 INFO:teuthology.orchestra.run.trial068.stdout:Version: 5.15.0.171.160 2026-02-24T17:54:11.159 INFO:teuthology.orchestra.run.trial068.stdout:Provides: spl-modules (= 2.1.5-1ubuntu6~22.04.6), v4l2loopback-modules (= 0.12.7-2ubuntu2~22.04.1), virtualbox-guest-modules (= 5.15.0-171), wireguard-modules (= 1.0.0), zfs-modules (= 2.1.5-1ubuntu6~22.04.6) 2026-02-24T17:54:11.159 INFO:teuthology.orchestra.run.trial068.stdout:Depends: linux-image-5.15.0-171-generic, linux-modules-extra-5.15.0-171-generic, linux-firmware, intel-microcode, amd64-microcode 2026-02-24T17:54:11.159 INFO:teuthology.orchestra.run.trial068.stdout:Recommends: thermald 2026-02-24T17:54:11.159 INFO:teuthology.orchestra.run.trial068.stdout:Description: Generic Linux kernel image 2026-02-24T17:54:11.159 INFO:teuthology.orchestra.run.trial068.stdout: This package will always depend on the latest generic kernel image 2026-02-24T17:54:11.159 INFO:teuthology.orchestra.run.trial068.stdout: available. 2026-02-24T17:54:11.160 DEBUG:teuthology.orchestra.run.trial068:> mktemp 2026-02-24T17:54:11.207 INFO:teuthology.orchestra.run.trial068.stdout:/tmp/tmp.fzGqu9ySzY 2026-02-24T17:54:11.207 DEBUG:teuthology.orchestra.run.trial068:> sudo cp /boot/grub/grub.cfg /tmp/tmp.fzGqu9ySzY 2026-02-24T17:54:11.258 DEBUG:teuthology.orchestra.run.trial068:> sudo chmod 0666 /tmp/tmp.fzGqu9ySzY 2026-02-24T17:54:11.365 DEBUG:teuthology.orchestra.remote:trial068:/tmp/tmp.fzGqu9ySzY is 9KB 2026-02-24T17:54:11.373 DEBUG:teuthology.orchestra.run.trial068:> rm -fr /tmp/tmp.fzGqu9ySzY 2026-02-24T17:54:11.378 DEBUG:teuthology.orchestra.run.trial068:> sudo rm -f -- /etc/grub.d/01_ceph_kernel 2026-02-24T17:54:11.426 DEBUG:teuthology.orchestra.run.trial068:> set -ex 2026-02-24T17:54:11.426 DEBUG:teuthology.orchestra.run.trial068:> sudo dd of=/etc/grub.d/01_ceph_kernel 2026-02-24T17:54:11.426 DEBUG:teuthology.orchestra.run.trial068:> sudo chmod 755 /etc/grub.d/01_ceph_kernel 2026-02-24T17:54:11.480 INFO:teuthology.task.kernel:Distro Kernel Version: 5.15.0-171-generic 2026-02-24T17:54:11.481 DEBUG:teuthology.orchestra.run.trial068:> sudo update-grub 2026-02-24T17:54:11.717 INFO:teuthology.orchestra.run.trial068.stderr:Sourcing file `/etc/default/grub' 2026-02-24T17:54:11.736 INFO:teuthology.orchestra.run.trial068.stderr:Sourcing file `/etc/default/grub.d/50-curtin-settings.cfg' 2026-02-24T17:54:11.737 INFO:teuthology.orchestra.run.trial068.stderr:Sourcing file `/etc/default/grub.d/init-select.cfg' 2026-02-24T17:54:11.739 INFO:teuthology.orchestra.run.trial068.stderr:Generating grub configuration file ... 2026-02-24T17:54:11.824 INFO:teuthology.orchestra.run.trial068.stderr:Found linux image: /boot/vmlinuz-5.15.0-171-generic 2026-02-24T17:54:11.830 INFO:teuthology.orchestra.run.trial068.stderr:Found initrd image: /boot/initrd.img-5.15.0-171-generic 2026-02-24T17:54:12.006 INFO:teuthology.orchestra.run.trial068.stderr:Found linux image: /boot/vmlinuz-5.15.0-170-generic 2026-02-24T17:54:12.012 INFO:teuthology.orchestra.run.trial068.stderr:Found initrd image: /boot/initrd.img-5.15.0-170-generic 2026-02-24T17:54:12.053 INFO:teuthology.orchestra.run.trial068.stderr:Found linux image: /boot/vmlinuz-5.15.0-164-generic 2026-02-24T17:54:12.059 INFO:teuthology.orchestra.run.trial068.stderr:Found initrd image: /boot/initrd.img-5.15.0-164-generic 2026-02-24T17:54:12.132 INFO:teuthology.orchestra.run.trial068.stderr:Warning: os-prober will not be executed to detect other bootable partitions. 2026-02-24T17:54:12.132 INFO:teuthology.orchestra.run.trial068.stderr:Systems on them will not be added to the GRUB boot configuration. 2026-02-24T17:54:12.132 INFO:teuthology.orchestra.run.trial068.stderr:Check GRUB_DISABLE_OS_PROBER documentation entry. 2026-02-24T17:54:12.134 INFO:teuthology.orchestra.run.trial068.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2026-02-24T17:54:12.141 INFO:teuthology.orchestra.run.trial068.stderr:done 2026-02-24T17:54:12.143 INFO:teuthology.orchestra.remote:Triggering safe hard reboot via SysRq on trial068 2026-02-24T17:54:12.143 DEBUG:teuthology.orchestra.run.trial068:> sudo bash -c ' 2026-02-24T17:54:12.144 DEBUG:teuthology.orchestra.run.trial068:> echo pci > /sys/kernel/reboot/type && echo 1 > /sys/kernel/reboot/force && sync && echo s > /proc/sysrq-trigger && echo u > /proc/sysrq-trigger && sleep 2 && echo b > /proc/sysrq-trigger 2026-02-24T17:54:12.144 DEBUG:teuthology.orchestra.run.trial068:> ' 2026-02-24T17:54:42.147 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version... 2026-02-24T17:54:42.147 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@trial068.front.sepia.ceph.com' 2026-02-24T17:54:42.148 DEBUG:teuthology.orchestra.connection:{'hostname': 'trial068.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2026-02-24T17:55:42.182 DEBUG:teuthology.orchestra.remote:timed out 2026-02-24T17:55:51.185 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@trial068.front.sepia.ceph.com' 2026-02-24T17:55:51.185 DEBUG:teuthology.orchestra.connection:{'hostname': 'trial068.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2026-02-24T17:55:51.310 DEBUG:teuthology.orchestra.run.trial068:> true 2026-02-24T17:55:51.704 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@trial068.front.sepia.ceph.com' 2026-02-24T17:55:51.704 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "5.15.0-171-generic"... 2026-02-24T17:55:51.704 DEBUG:teuthology.orchestra.run.trial068:> uname -r 2026-02-24T17:55:51.751 INFO:teuthology.orchestra.run.trial068.stdout:5.15.0-171-generic 2026-02-24T17:55:51.751 DEBUG:teuthology.task.kernel:current kernel version is 5.15.0-171-generic vs 5.15.0-171-generic 2026-02-24T17:55:51.751 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2026-02-24T17:55:51.752 DEBUG:teuthology.task.kernel:Distro of this test job: ubuntu 2026-02-24T17:55:52.752 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2026-02-24T17:55:52.752 DEBUG:teuthology.orchestra.run.trial068:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2026-02-24T17:55:52.765 INFO:teuthology.orchestra.run.trial068.stdout:ttyS1 2026-02-24T17:55:52.797 DEBUG:teuthology.parallel:result is None 2026-02-24T17:55:52.797 INFO:teuthology.run_tasks:Running task internal.base... 2026-02-24T17:55:52.847 INFO:teuthology.task.internal:Creating test directory... 2026-02-24T17:55:52.847 DEBUG:teuthology.orchestra.run.trial068:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2026-02-24T17:55:52.851 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2026-02-24T17:55:52.863 INFO:teuthology.run_tasks:Running task internal.archive... 2026-02-24T17:55:52.882 INFO:teuthology.task.internal:Creating archive directory... 2026-02-24T17:55:52.883 DEBUG:teuthology.orchestra.run.trial068:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2026-02-24T17:55:52.903 INFO:teuthology.run_tasks:Running task internal.coredump... 2026-02-24T17:55:52.914 INFO:teuthology.task.internal:Enabling coredump saving... 2026-02-24T17:55:52.915 DEBUG:teuthology.orchestra.run.trial068:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2026-02-24T17:55:52.948 INFO:teuthology.orchestra.run.trial068.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-24T17:55:52.955 INFO:teuthology.orchestra.run.trial068.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-24T17:55:52.956 INFO:teuthology.run_tasks:Running task internal.sudo... 2026-02-24T17:55:52.973 INFO:teuthology.task.internal:Configuring sudo... 2026-02-24T17:55:52.973 DEBUG:teuthology.orchestra.run.trial068:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2026-02-24T17:55:53.008 INFO:teuthology.run_tasks:Running task internal.syslog... 2026-02-24T17:55:53.024 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2026-02-24T17:55:53.024 DEBUG:teuthology.orchestra.run.trial068:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2026-02-24T17:55:53.055 DEBUG:teuthology.orchestra.run.trial068:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2026-02-24T17:55:53.099 DEBUG:teuthology.orchestra.run.trial068:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2026-02-24T17:55:53.144 DEBUG:teuthology.orchestra.run.trial068:> set -ex 2026-02-24T17:55:53.144 DEBUG:teuthology.orchestra.run.trial068:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2026-02-24T17:55:53.197 DEBUG:teuthology.orchestra.run.trial068:> sudo service rsyslog restart 2026-02-24T17:55:53.257 INFO:teuthology.run_tasks:Running task internal.timer... 2026-02-24T17:55:53.272 INFO:teuthology.task.internal:Starting timer... 2026-02-24T17:55:53.272 INFO:teuthology.run_tasks:Running task pcp... 2026-02-24T17:55:53.294 INFO:teuthology.run_tasks:Running task selinux... 2026-02-24T17:55:53.315 DEBUG:teuthology.task.selinux:Excluding trial068: OS 'ubuntu' does not support SELinux 2026-02-24T17:55:53.315 DEBUG:teuthology.task.selinux:Getting current SELinux state 2026-02-24T17:55:53.315 DEBUG:teuthology.task.selinux:Existing SELinux modes: {} 2026-02-24T17:55:53.315 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2026-02-24T17:55:53.316 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2026-02-24T17:55:53.331 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2026-02-24T17:55:53.343 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2026-02-24T17:55:53.345 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit trial068.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2026-02-24T17:58:14.164 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@trial068.front.sepia.ceph.com')] 2026-02-24T17:58:14.165 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@trial068.front.sepia.ceph.com' 2026-02-24T17:58:14.166 DEBUG:teuthology.orchestra.connection:{'hostname': 'trial068.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2026-02-24T17:58:14.259 DEBUG:teuthology.orchestra.run.trial068:> true 2026-02-24T17:58:14.511 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@trial068.front.sepia.ceph.com' 2026-02-24T17:58:14.512 INFO:teuthology.run_tasks:Running task clock... 2026-02-24T17:58:14.536 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2026-02-24T17:58:14.537 INFO:teuthology.orchestra.run:Running command with timeout 360 2026-02-24T17:58:14.538 DEBUG:teuthology.orchestra.run.trial068:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2026-02-24T17:58:14.574 INFO:teuthology.orchestra.run.trial068.stdout:24 Feb 17:58:14 ntpd[8707]: ntpd 4.2.8p15@1.3728-o Wed Feb 16 17:13:02 UTC 2022 (1): Starting 2026-02-24T17:58:14.574 INFO:teuthology.orchestra.run.trial068.stdout:24 Feb 17:58:14 ntpd[8707]: Command line: ntpd -gq 2026-02-24T17:58:14.574 INFO:teuthology.orchestra.run.trial068.stdout:24 Feb 17:58:14 ntpd[8707]: ---------------------------------------------------- 2026-02-24T17:58:14.574 INFO:teuthology.orchestra.run.trial068.stdout:24 Feb 17:58:14 ntpd[8707]: ntp-4 is maintained by Network Time Foundation, 2026-02-24T17:58:14.574 INFO:teuthology.orchestra.run.trial068.stdout:24 Feb 17:58:14 ntpd[8707]: Inc. (NTF), a non-profit 501(c)(3) public-benefit 2026-02-24T17:58:14.574 INFO:teuthology.orchestra.run.trial068.stdout:24 Feb 17:58:14 ntpd[8707]: corporation. Support and training for ntp-4 are 2026-02-24T17:58:14.574 INFO:teuthology.orchestra.run.trial068.stdout:24 Feb 17:58:14 ntpd[8707]: available at https://www.nwtime.org/support 2026-02-24T17:58:14.574 INFO:teuthology.orchestra.run.trial068.stdout:24 Feb 17:58:14 ntpd[8707]: ---------------------------------------------------- 2026-02-24T17:58:14.574 INFO:teuthology.orchestra.run.trial068.stdout:24 Feb 17:58:14 ntpd[8707]: proto: precision = 0.030 usec (-25) 2026-02-24T17:58:14.574 INFO:teuthology.orchestra.run.trial068.stdout:24 Feb 17:58:14 ntpd[8707]: basedate set to 2022-02-04 2026-02-24T17:58:14.574 INFO:teuthology.orchestra.run.trial068.stdout:24 Feb 17:58:14 ntpd[8707]: gps base set to 2022-02-06 (week 2196) 2026-02-24T17:58:14.574 INFO:teuthology.orchestra.run.trial068.stderr:restrict 0.0.0.0: KOD does nothing without LIMITED. 2026-02-24T17:58:14.574 INFO:teuthology.orchestra.run.trial068.stdout:24 Feb 17:58:14 ntpd[8707]: restrict 0.0.0.0: KOD does nothing without LIMITED. 2026-02-24T17:58:14.574 INFO:teuthology.orchestra.run.trial068.stdout:24 Feb 17:58:14 ntpd[8707]: restrict ::: KOD does nothing without LIMITED. 2026-02-24T17:58:14.574 INFO:teuthology.orchestra.run.trial068.stderr:restrict ::: KOD does nothing without LIMITED. 2026-02-24T17:58:14.575 INFO:teuthology.orchestra.run.trial068.stdout:24 Feb 17:58:14 ntpd[8707]: Listen and drop on 0 v6wildcard [::]:123 2026-02-24T17:58:14.575 INFO:teuthology.orchestra.run.trial068.stdout:24 Feb 17:58:14 ntpd[8707]: Listen and drop on 1 v4wildcard 0.0.0.0:123 2026-02-24T17:58:14.575 INFO:teuthology.orchestra.run.trial068.stdout:24 Feb 17:58:14 ntpd[8707]: Listen normally on 2 lo 127.0.0.1:123 2026-02-24T17:58:14.575 INFO:teuthology.orchestra.run.trial068.stdout:24 Feb 17:58:14 ntpd[8707]: Listen normally on 3 enp1s0f0 10.20.193.68:123 2026-02-24T17:58:14.575 INFO:teuthology.orchestra.run.trial068.stdout:24 Feb 17:58:14 ntpd[8707]: Listen normally on 4 lo [::1]:123 2026-02-24T17:58:14.575 INFO:teuthology.orchestra.run.trial068.stdout:24 Feb 17:58:14 ntpd[8707]: Listen normally on 5 enp1s0f0 [fe80::925a:8ff:fe77:60ea%2]:123 2026-02-24T17:58:14.575 INFO:teuthology.orchestra.run.trial068.stdout:24 Feb 17:58:14 ntpd[8707]: Listening on routing socket on fd #22 for interface updates 2026-02-24T17:58:15.575 INFO:teuthology.orchestra.run.trial068.stderr:24 Feb 17:58:15 ntpd[8707]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2026-02-24T17:58:15.575 INFO:teuthology.orchestra.run.trial068.stderr:24 Feb 17:58:15 ntpd[8707]: can't open /var/log/ntpstats/rawstats.20260224: Permission denied 2026-02-24T17:58:15.576 INFO:teuthology.orchestra.run.trial068.stderr:24 Feb 17:58:15 ntpd[8707]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2026-02-24T17:58:15.576 INFO:teuthology.orchestra.run.trial068.stderr:24 Feb 17:58:15 ntpd[8707]: can't open /var/log/ntpstats/rawstats.20260224: Permission denied 2026-02-24T17:58:15.640 INFO:teuthology.orchestra.run.trial068.stderr:24 Feb 17:58:15 ntpd[8707]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2026-02-24T17:58:15.640 INFO:teuthology.orchestra.run.trial068.stderr:24 Feb 17:58:15 ntpd[8707]: can't open /var/log/ntpstats/rawstats.20260224: Permission denied 2026-02-24T17:58:15.640 INFO:teuthology.orchestra.run.trial068.stderr:24 Feb 17:58:15 ntpd[8707]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2026-02-24T17:58:15.640 INFO:teuthology.orchestra.run.trial068.stderr:24 Feb 17:58:15 ntpd[8707]: can't open /var/log/ntpstats/peerstats.20260224: Permission denied 2026-02-24T17:58:15.646 INFO:teuthology.orchestra.run.trial068.stderr:24 Feb 17:58:15 ntpd[8707]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2026-02-24T17:58:15.646 INFO:teuthology.orchestra.run.trial068.stderr:24 Feb 17:58:15 ntpd[8707]: can't open /var/log/ntpstats/rawstats.20260224: Permission denied 2026-02-24T17:58:15.647 INFO:teuthology.orchestra.run.trial068.stderr:24 Feb 17:58:15 ntpd[8707]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2026-02-24T17:58:15.647 INFO:teuthology.orchestra.run.trial068.stderr:24 Feb 17:58:15 ntpd[8707]: can't open /var/log/ntpstats/peerstats.20260224: Permission denied 2026-02-24T17:58:16.574 INFO:teuthology.orchestra.run.trial068.stderr:24 Feb 17:58:16 ntpd[8707]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2026-02-24T17:58:16.574 INFO:teuthology.orchestra.run.trial068.stderr:24 Feb 17:58:16 ntpd[8707]: can't open /var/log/ntpstats/rawstats.20260224: Permission denied 2026-02-24T17:58:16.579 INFO:teuthology.orchestra.run.trial068.stderr:24 Feb 17:58:16 ntpd[8707]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2026-02-24T17:58:16.579 INFO:teuthology.orchestra.run.trial068.stderr:24 Feb 17:58:16 ntpd[8707]: can't open /var/log/ntpstats/rawstats.20260224: Permission denied 2026-02-24T17:58:16.579 INFO:teuthology.orchestra.run.trial068.stderr:24 Feb 17:58:16 ntpd[8707]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2026-02-24T17:58:16.579 INFO:teuthology.orchestra.run.trial068.stderr:24 Feb 17:58:16 ntpd[8707]: can't open /var/log/ntpstats/peerstats.20260224: Permission denied 2026-02-24T17:58:17.574 INFO:teuthology.orchestra.run.trial068.stderr:24 Feb 17:58:17 ntpd[8707]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2026-02-24T17:58:17.574 INFO:teuthology.orchestra.run.trial068.stderr:24 Feb 17:58:17 ntpd[8707]: can't open /var/log/ntpstats/rawstats.20260224: Permission denied 2026-02-24T17:58:17.574 INFO:teuthology.orchestra.run.trial068.stderr:24 Feb 17:58:17 ntpd[8707]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2026-02-24T17:58:17.574 INFO:teuthology.orchestra.run.trial068.stderr:24 Feb 17:58:17 ntpd[8707]: can't open /var/log/ntpstats/rawstats.20260224: Permission denied 2026-02-24T17:58:17.574 INFO:teuthology.orchestra.run.trial068.stderr:24 Feb 17:58:17 ntpd[8707]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2026-02-24T17:58:17.574 INFO:teuthology.orchestra.run.trial068.stderr:24 Feb 17:58:17 ntpd[8707]: can't open /var/log/ntpstats/rawstats.20260224: Permission denied 2026-02-24T17:58:17.596 INFO:teuthology.orchestra.run.trial068.stderr:24 Feb 17:58:17 ntpd[8707]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2026-02-24T17:58:17.596 INFO:teuthology.orchestra.run.trial068.stderr:24 Feb 17:58:17 ntpd[8707]: can't open /var/log/ntpstats/rawstats.20260224: Permission denied 2026-02-24T17:58:17.596 INFO:teuthology.orchestra.run.trial068.stderr:24 Feb 17:58:17 ntpd[8707]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2026-02-24T17:58:17.596 INFO:teuthology.orchestra.run.trial068.stderr:24 Feb 17:58:17 ntpd[8707]: can't open /var/log/ntpstats/peerstats.20260224: Permission denied 2026-02-24T17:58:17.640 INFO:teuthology.orchestra.run.trial068.stderr:24 Feb 17:58:17 ntpd[8707]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2026-02-24T17:58:17.640 INFO:teuthology.orchestra.run.trial068.stderr:24 Feb 17:58:17 ntpd[8707]: can't open /var/log/ntpstats/rawstats.20260224: Permission denied 2026-02-24T17:58:17.640 INFO:teuthology.orchestra.run.trial068.stderr:24 Feb 17:58:17 ntpd[8707]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2026-02-24T17:58:17.640 INFO:teuthology.orchestra.run.trial068.stderr:24 Feb 17:58:17 ntpd[8707]: can't open /var/log/ntpstats/peerstats.20260224: Permission denied 2026-02-24T17:58:17.650 INFO:teuthology.orchestra.run.trial068.stderr:24 Feb 17:58:17 ntpd[8707]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2026-02-24T17:58:17.650 INFO:teuthology.orchestra.run.trial068.stderr:24 Feb 17:58:17 ntpd[8707]: can't open /var/log/ntpstats/rawstats.20260224: Permission denied 2026-02-24T17:58:17.650 INFO:teuthology.orchestra.run.trial068.stderr:24 Feb 17:58:17 ntpd[8707]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2026-02-24T17:58:17.650 INFO:teuthology.orchestra.run.trial068.stderr:24 Feb 17:58:17 ntpd[8707]: can't open /var/log/ntpstats/peerstats.20260224: Permission denied 2026-02-24T17:58:18.574 INFO:teuthology.orchestra.run.trial068.stderr:24 Feb 17:58:18 ntpd[8707]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2026-02-24T17:58:18.574 INFO:teuthology.orchestra.run.trial068.stderr:24 Feb 17:58:18 ntpd[8707]: can't open /var/log/ntpstats/rawstats.20260224: Permission denied 2026-02-24T17:58:18.577 INFO:teuthology.orchestra.run.trial068.stderr:24 Feb 17:58:18 ntpd[8707]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2026-02-24T17:58:18.577 INFO:teuthology.orchestra.run.trial068.stderr:24 Feb 17:58:18 ntpd[8707]: can't open /var/log/ntpstats/rawstats.20260224: Permission denied 2026-02-24T17:58:18.578 INFO:teuthology.orchestra.run.trial068.stderr:24 Feb 17:58:18 ntpd[8707]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2026-02-24T17:58:18.578 INFO:teuthology.orchestra.run.trial068.stderr:24 Feb 17:58:18 ntpd[8707]: can't open /var/log/ntpstats/peerstats.20260224: Permission denied 2026-02-24T17:58:19.574 INFO:teuthology.orchestra.run.trial068.stderr:24 Feb 17:58:19 ntpd[8707]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2026-02-24T17:58:19.574 INFO:teuthology.orchestra.run.trial068.stderr:24 Feb 17:58:19 ntpd[8707]: can't open /var/log/ntpstats/rawstats.20260224: Permission denied 2026-02-24T17:58:19.575 INFO:teuthology.orchestra.run.trial068.stderr:24 Feb 17:58:19 ntpd[8707]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2026-02-24T17:58:19.575 INFO:teuthology.orchestra.run.trial068.stderr:24 Feb 17:58:19 ntpd[8707]: can't open /var/log/ntpstats/rawstats.20260224: Permission denied 2026-02-24T17:58:19.575 INFO:teuthology.orchestra.run.trial068.stderr:24 Feb 17:58:19 ntpd[8707]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2026-02-24T17:58:19.575 INFO:teuthology.orchestra.run.trial068.stderr:24 Feb 17:58:19 ntpd[8707]: can't open /var/log/ntpstats/rawstats.20260224: Permission denied 2026-02-24T17:58:19.596 INFO:teuthology.orchestra.run.trial068.stderr:24 Feb 17:58:19 ntpd[8707]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2026-02-24T17:58:19.597 INFO:teuthology.orchestra.run.trial068.stderr:24 Feb 17:58:19 ntpd[8707]: can't open /var/log/ntpstats/rawstats.20260224: Permission denied 2026-02-24T17:58:19.597 INFO:teuthology.orchestra.run.trial068.stderr:24 Feb 17:58:19 ntpd[8707]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2026-02-24T17:58:19.597 INFO:teuthology.orchestra.run.trial068.stderr:24 Feb 17:58:19 ntpd[8707]: can't open /var/log/ntpstats/peerstats.20260224: Permission denied 2026-02-24T17:58:19.641 INFO:teuthology.orchestra.run.trial068.stderr:24 Feb 17:58:19 ntpd[8707]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2026-02-24T17:58:19.641 INFO:teuthology.orchestra.run.trial068.stderr:24 Feb 17:58:19 ntpd[8707]: can't open /var/log/ntpstats/rawstats.20260224: Permission denied 2026-02-24T17:58:19.641 INFO:teuthology.orchestra.run.trial068.stderr:24 Feb 17:58:19 ntpd[8707]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2026-02-24T17:58:19.641 INFO:teuthology.orchestra.run.trial068.stderr:24 Feb 17:58:19 ntpd[8707]: can't open /var/log/ntpstats/peerstats.20260224: Permission denied 2026-02-24T17:58:19.649 INFO:teuthology.orchestra.run.trial068.stderr:24 Feb 17:58:19 ntpd[8707]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2026-02-24T17:58:19.649 INFO:teuthology.orchestra.run.trial068.stderr:24 Feb 17:58:19 ntpd[8707]: can't open /var/log/ntpstats/rawstats.20260224: Permission denied 2026-02-24T17:58:19.649 INFO:teuthology.orchestra.run.trial068.stderr:24 Feb 17:58:19 ntpd[8707]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2026-02-24T17:58:19.649 INFO:teuthology.orchestra.run.trial068.stderr:24 Feb 17:58:19 ntpd[8707]: can't open /var/log/ntpstats/peerstats.20260224: Permission denied 2026-02-24T17:58:20.574 INFO:teuthology.orchestra.run.trial068.stderr:24 Feb 17:58:20 ntpd[8707]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2026-02-24T17:58:20.574 INFO:teuthology.orchestra.run.trial068.stderr:24 Feb 17:58:20 ntpd[8707]: can't open /var/log/ntpstats/rawstats.20260224: Permission denied 2026-02-24T17:58:20.577 INFO:teuthology.orchestra.run.trial068.stderr:24 Feb 17:58:20 ntpd[8707]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2026-02-24T17:58:20.577 INFO:teuthology.orchestra.run.trial068.stderr:24 Feb 17:58:20 ntpd[8707]: can't open /var/log/ntpstats/rawstats.20260224: Permission denied 2026-02-24T17:58:20.578 INFO:teuthology.orchestra.run.trial068.stderr:24 Feb 17:58:20 ntpd[8707]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2026-02-24T17:58:20.578 INFO:teuthology.orchestra.run.trial068.stderr:24 Feb 17:58:20 ntpd[8707]: can't open /var/log/ntpstats/peerstats.20260224: Permission denied 2026-02-24T17:58:21.575 INFO:teuthology.orchestra.run.trial068.stderr:24 Feb 17:58:21 ntpd[8707]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2026-02-24T17:58:21.575 INFO:teuthology.orchestra.run.trial068.stderr:24 Feb 17:58:21 ntpd[8707]: can't open /var/log/ntpstats/rawstats.20260224: Permission denied 2026-02-24T17:58:21.575 INFO:teuthology.orchestra.run.trial068.stderr:24 Feb 17:58:21 ntpd[8707]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2026-02-24T17:58:21.575 INFO:teuthology.orchestra.run.trial068.stderr:24 Feb 17:58:21 ntpd[8707]: can't open /var/log/ntpstats/rawstats.20260224: Permission denied 2026-02-24T17:58:21.575 INFO:teuthology.orchestra.run.trial068.stderr:24 Feb 17:58:21 ntpd[8707]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2026-02-24T17:58:21.575 INFO:teuthology.orchestra.run.trial068.stderr:24 Feb 17:58:21 ntpd[8707]: can't open /var/log/ntpstats/rawstats.20260224: Permission denied 2026-02-24T17:58:21.597 INFO:teuthology.orchestra.run.trial068.stderr:24 Feb 17:58:21 ntpd[8707]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2026-02-24T17:58:21.597 INFO:teuthology.orchestra.run.trial068.stderr:24 Feb 17:58:21 ntpd[8707]: can't open /var/log/ntpstats/rawstats.20260224: Permission denied 2026-02-24T17:58:21.597 INFO:teuthology.orchestra.run.trial068.stderr:24 Feb 17:58:21 ntpd[8707]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2026-02-24T17:58:21.597 INFO:teuthology.orchestra.run.trial068.stderr:24 Feb 17:58:21 ntpd[8707]: can't open /var/log/ntpstats/peerstats.20260224: Permission denied 2026-02-24T17:58:21.639 INFO:teuthology.orchestra.run.trial068.stderr:24 Feb 17:58:21 ntpd[8707]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2026-02-24T17:58:21.640 INFO:teuthology.orchestra.run.trial068.stderr:24 Feb 17:58:21 ntpd[8707]: can't open /var/log/ntpstats/rawstats.20260224: Permission denied 2026-02-24T17:58:21.640 INFO:teuthology.orchestra.run.trial068.stderr:24 Feb 17:58:21 ntpd[8707]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2026-02-24T17:58:21.640 INFO:teuthology.orchestra.run.trial068.stderr:24 Feb 17:58:21 ntpd[8707]: can't open /var/log/ntpstats/peerstats.20260224: Permission denied 2026-02-24T17:58:21.640 INFO:teuthology.orchestra.run.trial068.stdout:24 Feb 17:58:21 ntpd[8707]: ntpd: time set -0.049042 s 2026-02-24T17:58:21.640 INFO:teuthology.orchestra.run.trial068.stdout:ntpd: time set -0.049042s 2026-02-24T17:58:21.641 INFO:teuthology.orchestra.run.trial068.stderr:24 Feb 17:58:21 ntpd[8707]: couldn't unlink /var/log/ntpstats/loopstats: Permission denied 2026-02-24T17:58:21.641 INFO:teuthology.orchestra.run.trial068.stderr:24 Feb 17:58:21 ntpd[8707]: can't open /var/log/ntpstats/loopstats.20260224: Permission denied 2026-02-24T17:58:21.740 INFO:teuthology.orchestra.run.trial068.stdout: remote refid st t when poll reach delay offset jitter 2026-02-24T17:58:21.740 INFO:teuthology.orchestra.run.trial068.stdout:============================================================================== 2026-02-24T17:58:21.740 INFO:teuthology.orchestra.run.trial068.stdout: ntp.maxhost.io .INIT. 16 u - 64 0 0.000 +0.000 0.000 2026-02-24T17:58:21.740 INFO:teuthology.orchestra.run.trial068.stdout: sushi.ruselabs. .INIT. 16 u - 64 0 0.000 +0.000 0.000 2026-02-24T17:58:21.741 INFO:teuthology.orchestra.run.trial068.stdout: static-72-87-88 .INIT. 16 u - 64 0 0.000 +0.000 0.000 2026-02-24T17:58:21.741 INFO:teuthology.orchestra.run.trial068.stdout: 23.186.168.129 .INIT. 16 u - 64 0 0.000 +0.000 0.000 2026-02-24T17:58:21.741 INFO:teuthology.run_tasks:Running task install... 2026-02-24T17:58:21.882 DEBUG:teuthology.task.install:project ceph 2026-02-24T17:58:21.882 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': '618c318a7a0b287ab5aeff207b7bf1d73ef45044'}} 2026-02-24T17:58:21.882 DEBUG:teuthology.task.install:config {'flavor': 'default', 'sha1': '618c318a7a0b287ab5aeff207b7bf1d73ef45044'} 2026-02-24T17:58:21.883 INFO:teuthology.task.install:Using flavor: default 2026-02-24T17:58:21.892 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-osd', 'ceph-osd-classic', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-osd', 'ceph-osd-classic', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2026-02-24T17:58:21.892 INFO:teuthology.task.install:extra packages: [] 2026-02-24T17:58:21.893 DEBUG:teuthology.orchestra.run.trial068:> sudo apt-key list | grep Ceph 2026-02-24T17:58:21.943 INFO:teuthology.orchestra.run.trial068.stderr:Warning: apt-key is deprecated. Manage keyring files in trusted.gpg.d instead (see apt-key(8)). 2026-02-24T17:58:21.971 INFO:teuthology.orchestra.run.trial068.stdout:uid [ unknown] Ceph automated package build (Ceph automated package build) 2026-02-24T17:58:21.971 INFO:teuthology.orchestra.run.trial068.stdout:uid [ unknown] Ceph.com (release key) 2026-02-24T17:58:21.973 INFO:teuthology.task.install.deb:Installing packages: ceph, cephadm, ceph-mds, ceph-mgr, ceph-osd, ceph-osd-classic, ceph-common, ceph-fuse, ceph-test, ceph-volume, radosgw, python3-rados, python3-rgw, python3-cephfs, python3-rbd, libcephfs2, libcephfs-dev, librados2, librbd1, rbd-fuse on remote deb x86_64 2026-02-24T17:58:21.973 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&sha1=618c318a7a0b287ab5aeff207b7bf1d73ef45044 2026-02-24T17:58:22.152 INFO:teuthology.task.install.deb:Pulling from https://2.chacra.ceph.com/r/ceph/wip-yuri8-testing-2026-02-23-2128/618c318a7a0b287ab5aeff207b7bf1d73ef45044/ubuntu/jammy/flavors/default/ 2026-02-24T17:58:22.152 INFO:teuthology.task.install.deb:Package version is 20.3.0-5507-g618c318a-1jammy 2026-02-24T17:58:22.424 DEBUG:teuthology.orchestra.run.trial068:> set -ex 2026-02-24T17:58:22.424 DEBUG:teuthology.orchestra.run.trial068:> sudo dd of=/etc/apt/sources.list.d/ceph.list 2026-02-24T17:58:22.435 DEBUG:teuthology.orchestra.run.trial068:> sudo apt-get update 2026-02-24T17:58:22.741 INFO:teuthology.orchestra.run.trial068.stdout:Ign:1 https://2.chacra.ceph.com/r/ceph/wip-yuri8-testing-2026-02-23-2128/618c318a7a0b287ab5aeff207b7bf1d73ef45044/ubuntu/jammy/flavors/default jammy InRelease 2026-02-24T17:58:22.787 INFO:teuthology.orchestra.run.trial068.stdout:Get:2 https://2.chacra.ceph.com/r/ceph/wip-yuri8-testing-2026-02-23-2128/618c318a7a0b287ab5aeff207b7bf1d73ef45044/ubuntu/jammy/flavors/default jammy Release [7689 B] 2026-02-24T17:58:22.834 INFO:teuthology.orchestra.run.trial068.stdout:Ign:3 https://2.chacra.ceph.com/r/ceph/wip-yuri8-testing-2026-02-23-2128/618c318a7a0b287ab5aeff207b7bf1d73ef45044/ubuntu/jammy/flavors/default jammy Release.gpg 2026-02-24T17:58:22.881 INFO:teuthology.orchestra.run.trial068.stdout:Get:4 https://2.chacra.ceph.com/r/ceph/wip-yuri8-testing-2026-02-23-2128/618c318a7a0b287ab5aeff207b7bf1d73ef45044/ubuntu/jammy/flavors/default jammy/main amd64 Packages [19.2 kB] 2026-02-24T17:58:22.887 INFO:teuthology.orchestra.run.trial068.stdout:Hit:5 https://archive.ubuntu.com/ubuntu jammy InRelease 2026-02-24T17:58:22.958 INFO:teuthology.orchestra.run.trial068.stdout:Hit:6 https://archive.ubuntu.com/ubuntu jammy-updates InRelease 2026-02-24T17:58:23.033 INFO:teuthology.orchestra.run.trial068.stdout:Hit:7 https://archive.ubuntu.com/ubuntu jammy-security InRelease 2026-02-24T17:58:23.110 INFO:teuthology.orchestra.run.trial068.stdout:Hit:8 https://archive.ubuntu.com/ubuntu jammy-backports InRelease 2026-02-24T17:58:23.251 INFO:teuthology.orchestra.run.trial068.stdout:Fetched 26.9 kB in 1s (41.0 kB/s) 2026-02-24T17:58:23.771 INFO:teuthology.orchestra.run.trial068.stdout:Reading package lists... 2026-02-24T17:58:23.785 DEBUG:teuthology.orchestra.run.trial068:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" install ceph=20.3.0-5507-g618c318a-1jammy cephadm=20.3.0-5507-g618c318a-1jammy ceph-mds=20.3.0-5507-g618c318a-1jammy ceph-mgr=20.3.0-5507-g618c318a-1jammy ceph-osd=20.3.0-5507-g618c318a-1jammy ceph-osd-classic=20.3.0-5507-g618c318a-1jammy ceph-common=20.3.0-5507-g618c318a-1jammy ceph-fuse=20.3.0-5507-g618c318a-1jammy ceph-test=20.3.0-5507-g618c318a-1jammy ceph-volume=20.3.0-5507-g618c318a-1jammy radosgw=20.3.0-5507-g618c318a-1jammy python3-rados=20.3.0-5507-g618c318a-1jammy python3-rgw=20.3.0-5507-g618c318a-1jammy python3-cephfs=20.3.0-5507-g618c318a-1jammy python3-rbd=20.3.0-5507-g618c318a-1jammy libcephfs2=20.3.0-5507-g618c318a-1jammy libcephfs-dev=20.3.0-5507-g618c318a-1jammy librados2=20.3.0-5507-g618c318a-1jammy librbd1=20.3.0-5507-g618c318a-1jammy rbd-fuse=20.3.0-5507-g618c318a-1jammy 2026-02-24T17:58:23.818 INFO:teuthology.orchestra.run.trial068.stdout:Reading package lists... 2026-02-24T17:58:23.926 INFO:teuthology.orchestra.run.trial068.stdout:Building dependency tree... 2026-02-24T17:58:23.926 INFO:teuthology.orchestra.run.trial068.stdout:Reading state information... 2026-02-24T17:58:24.013 INFO:teuthology.orchestra.run.trial068.stdout:The following packages were automatically installed and are no longer required: 2026-02-24T17:58:24.013 INFO:teuthology.orchestra.run.trial068.stdout: kpartx libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 2026-02-24T17:58:24.013 INFO:teuthology.orchestra.run.trial068.stdout: libgfxdr0 libglusterfs0 libiscsi7 libpmemobj1 libsgutils2-2 2026-02-24T17:58:24.013 INFO:teuthology.orchestra.run.trial068.stdout: linux-headers-5.15.0-164 linux-headers-5.15.0-164-generic 2026-02-24T17:58:24.013 INFO:teuthology.orchestra.run.trial068.stdout: linux-image-5.15.0-164-generic linux-modules-5.15.0-164-generic 2026-02-24T17:58:24.013 INFO:teuthology.orchestra.run.trial068.stdout: linux-modules-extra-5.15.0-164-generic sg3-utils sg3-utils-udev 2026-02-24T17:58:24.014 INFO:teuthology.orchestra.run.trial068.stdout:Use 'sudo apt autoremove' to remove them. 2026-02-24T17:58:24.014 INFO:teuthology.orchestra.run.trial068.stdout:The following additional packages will be installed: 2026-02-24T17:58:24.014 INFO:teuthology.orchestra.run.trial068.stdout: ceph-base ceph-mgr-cephadm ceph-mgr-dashboard ceph-mgr-diskprediction-local 2026-02-24T17:58:24.014 INFO:teuthology.orchestra.run.trial068.stdout: ceph-mgr-k8sevents ceph-mgr-modules-core ceph-mon jq libcephfs-daemon 2026-02-24T17:58:24.014 INFO:teuthology.orchestra.run.trial068.stdout: libcephfs-proxy2 libdouble-conversion3 libfuse2 libjq1 liblttng-ust1 libnbd0 2026-02-24T17:58:24.014 INFO:teuthology.orchestra.run.trial068.stdout: liboath0 libonig5 libpcre2-16-0 libqt5core5a libqt5dbus5 libqt5network5 2026-02-24T17:58:24.014 INFO:teuthology.orchestra.run.trial068.stdout: libradosstriper1 librdkafka1 librgw2 libsqlite3-mod-ceph libthrift-0.16.0 2026-02-24T17:58:24.014 INFO:teuthology.orchestra.run.trial068.stdout: nvme-cli python-asyncssh-doc python3-asyncssh python3-cachetools 2026-02-24T17:58:24.014 INFO:teuthology.orchestra.run.trial068.stdout: python3-ceph-argparse python3-ceph-common python3-cheroot python3-cherrypy3 2026-02-24T17:58:24.014 INFO:teuthology.orchestra.run.trial068.stdout: python3-google-auth python3-iniconfig python3-jaraco.classes 2026-02-24T17:58:24.014 INFO:teuthology.orchestra.run.trial068.stdout: python3-jaraco.collections python3-jaraco.functools python3-jaraco.text 2026-02-24T17:58:24.014 INFO:teuthology.orchestra.run.trial068.stdout: python3-joblib python3-kubernetes python3-natsort python3-pluggy 2026-02-24T17:58:24.014 INFO:teuthology.orchestra.run.trial068.stdout: python3-portend python3-prettytable python3-psutil python3-py 2026-02-24T17:58:24.014 INFO:teuthology.orchestra.run.trial068.stdout: python3-pygments python3-pytest python3-repoze.lru python3-requests-oauthlib 2026-02-24T17:58:24.014 INFO:teuthology.orchestra.run.trial068.stdout: python3-routes python3-rsa python3-simplejson python3-sklearn 2026-02-24T17:58:24.014 INFO:teuthology.orchestra.run.trial068.stdout: python3-sklearn-lib python3-tempora python3-threadpoolctl python3-toml 2026-02-24T17:58:24.014 INFO:teuthology.orchestra.run.trial068.stdout: python3-wcwidth python3-webob python3-websocket python3-zc.lockfile 2026-02-24T17:58:24.014 INFO:teuthology.orchestra.run.trial068.stdout: qttranslations5-l10n socat xmlstarlet 2026-02-24T17:58:24.014 INFO:teuthology.orchestra.run.trial068.stdout:Suggested packages: 2026-02-24T17:58:24.014 INFO:teuthology.orchestra.run.trial068.stdout: python3-influxdb liblua5.3-dev luarocks python-natsort-doc python-psutil-doc 2026-02-24T17:58:24.014 INFO:teuthology.orchestra.run.trial068.stdout: subversion python-pygments-doc ttf-bitstream-vera python3-paste python3-dap 2026-02-24T17:58:24.014 INFO:teuthology.orchestra.run.trial068.stdout: python-sklearn-doc ipython3 python-webob-doc 2026-02-24T17:58:24.014 INFO:teuthology.orchestra.run.trial068.stdout:Recommended packages: 2026-02-24T17:58:24.014 INFO:teuthology.orchestra.run.trial068.stdout: btrfs-tools 2026-02-24T17:58:24.048 INFO:teuthology.orchestra.run.trial068.stdout:The following NEW packages will be installed: 2026-02-24T17:58:24.049 INFO:teuthology.orchestra.run.trial068.stdout: ceph ceph-base ceph-common ceph-fuse ceph-mds ceph-mgr ceph-mgr-cephadm 2026-02-24T17:58:24.049 INFO:teuthology.orchestra.run.trial068.stdout: ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-k8sevents 2026-02-24T17:58:24.049 INFO:teuthology.orchestra.run.trial068.stdout: ceph-mgr-modules-core ceph-mon ceph-osd ceph-osd-classic ceph-test 2026-02-24T17:58:24.049 INFO:teuthology.orchestra.run.trial068.stdout: ceph-volume cephadm jq libcephfs-daemon libcephfs-dev libcephfs-proxy2 2026-02-24T17:58:24.049 INFO:teuthology.orchestra.run.trial068.stdout: libcephfs2 libdouble-conversion3 libfuse2 libjq1 liblttng-ust1 libnbd0 2026-02-24T17:58:24.049 INFO:teuthology.orchestra.run.trial068.stdout: liboath0 libonig5 libpcre2-16-0 libqt5core5a libqt5dbus5 libqt5network5 2026-02-24T17:58:24.049 INFO:teuthology.orchestra.run.trial068.stdout: librados2 libradosstriper1 librbd1 librdkafka1 librgw2 libsqlite3-mod-ceph 2026-02-24T17:58:24.049 INFO:teuthology.orchestra.run.trial068.stdout: libthrift-0.16.0 nvme-cli python-asyncssh-doc python3-asyncssh 2026-02-24T17:58:24.049 INFO:teuthology.orchestra.run.trial068.stdout: python3-cachetools python3-ceph-argparse python3-ceph-common python3-cephfs 2026-02-24T17:58:24.049 INFO:teuthology.orchestra.run.trial068.stdout: python3-cheroot python3-cherrypy3 python3-google-auth python3-iniconfig 2026-02-24T17:58:24.049 INFO:teuthology.orchestra.run.trial068.stdout: python3-jaraco.classes python3-jaraco.collections python3-jaraco.functools 2026-02-24T17:58:24.049 INFO:teuthology.orchestra.run.trial068.stdout: python3-jaraco.text python3-joblib python3-kubernetes python3-natsort 2026-02-24T17:58:24.049 INFO:teuthology.orchestra.run.trial068.stdout: python3-pluggy python3-portend python3-prettytable python3-psutil python3-py 2026-02-24T17:58:24.050 INFO:teuthology.orchestra.run.trial068.stdout: python3-pygments python3-pytest python3-rados python3-rbd python3-repoze.lru 2026-02-24T17:58:24.050 INFO:teuthology.orchestra.run.trial068.stdout: python3-requests-oauthlib python3-rgw python3-routes python3-rsa 2026-02-24T17:58:24.050 INFO:teuthology.orchestra.run.trial068.stdout: python3-simplejson python3-sklearn python3-sklearn-lib python3-tempora 2026-02-24T17:58:24.050 INFO:teuthology.orchestra.run.trial068.stdout: python3-threadpoolctl python3-toml python3-wcwidth python3-webob 2026-02-24T17:58:24.050 INFO:teuthology.orchestra.run.trial068.stdout: python3-websocket python3-zc.lockfile qttranslations5-l10n radosgw rbd-fuse 2026-02-24T17:58:24.050 INFO:teuthology.orchestra.run.trial068.stdout: socat xmlstarlet 2026-02-24T17:58:24.315 INFO:teuthology.orchestra.run.trial068.stdout:0 upgraded, 87 newly installed, 0 to remove and 14 not upgraded. 2026-02-24T17:58:24.315 INFO:teuthology.orchestra.run.trial068.stdout:Need to get 693 MB of archives. 2026-02-24T17:58:24.315 INFO:teuthology.orchestra.run.trial068.stdout:After this operation, 2583 MB of additional disk space will be used. 2026-02-24T17:58:24.315 INFO:teuthology.orchestra.run.trial068.stdout:Get:1 https://2.chacra.ceph.com/r/ceph/wip-yuri8-testing-2026-02-23-2128/618c318a7a0b287ab5aeff207b7bf1d73ef45044/ubuntu/jammy/flavors/default jammy/main amd64 librados2 amd64 20.3.0-5507-g618c318a-1jammy [3796 kB] 2026-02-24T17:58:24.442 INFO:teuthology.orchestra.run.trial068.stdout:Get:2 https://archive.ubuntu.com/ubuntu jammy/main amd64 liblttng-ust1 amd64 2.13.1-1ubuntu1 [190 kB] 2026-02-24T17:58:24.815 INFO:teuthology.orchestra.run.trial068.stdout:Get:3 https://2.chacra.ceph.com/r/ceph/wip-yuri8-testing-2026-02-23-2128/618c318a7a0b287ab5aeff207b7bf1d73ef45044/ubuntu/jammy/flavors/default jammy/main amd64 librbd1 amd64 20.3.0-5507-g618c318a-1jammy [2910 kB] 2026-02-24T17:58:24.829 INFO:teuthology.orchestra.run.trial068.stdout:Get:4 https://archive.ubuntu.com/ubuntu jammy/universe amd64 libdouble-conversion3 amd64 3.1.7-4 [39.0 kB] 2026-02-24T17:58:24.841 INFO:teuthology.orchestra.run.trial068.stdout:Get:5 https://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libpcre2-16-0 amd64 10.39-3ubuntu0.1 [203 kB] 2026-02-24T17:58:24.921 INFO:teuthology.orchestra.run.trial068.stdout:Get:6 https://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5core5a amd64 5.15.3+dfsg-2ubuntu0.2 [2006 kB] 2026-02-24T17:58:24.962 INFO:teuthology.orchestra.run.trial068.stdout:Get:7 https://2.chacra.ceph.com/r/ceph/wip-yuri8-testing-2026-02-23-2128/618c318a7a0b287ab5aeff207b7bf1d73ef45044/ubuntu/jammy/flavors/default jammy/main amd64 libcephfs2 amd64 20.3.0-5507-g618c318a-1jammy [883 kB] 2026-02-24T17:58:25.025 INFO:teuthology.orchestra.run.trial068.stdout:Get:8 https://2.chacra.ceph.com/r/ceph/wip-yuri8-testing-2026-02-23-2128/618c318a7a0b287ab5aeff207b7bf1d73ef45044/ubuntu/jammy/flavors/default jammy/main amd64 python3-rados amd64 20.3.0-5507-g618c318a-1jammy [359 kB] 2026-02-24T17:58:25.032 INFO:teuthology.orchestra.run.trial068.stdout:Get:9 https://2.chacra.ceph.com/r/ceph/wip-yuri8-testing-2026-02-23-2128/618c318a7a0b287ab5aeff207b7bf1d73ef45044/ubuntu/jammy/flavors/default jammy/main amd64 python3-ceph-argparse all 20.3.0-5507-g618c318a-1jammy [32.8 kB] 2026-02-24T17:58:25.032 INFO:teuthology.orchestra.run.trial068.stdout:Get:10 https://2.chacra.ceph.com/r/ceph/wip-yuri8-testing-2026-02-23-2128/618c318a7a0b287ab5aeff207b7bf1d73ef45044/ubuntu/jammy/flavors/default jammy/main amd64 python3-cephfs amd64 20.3.0-5507-g618c318a-1jammy [225 kB] 2026-02-24T17:58:25.036 INFO:teuthology.orchestra.run.trial068.stdout:Get:11 https://2.chacra.ceph.com/r/ceph/wip-yuri8-testing-2026-02-23-2128/618c318a7a0b287ab5aeff207b7bf1d73ef45044/ubuntu/jammy/flavors/default jammy/main amd64 python3-ceph-common all 20.3.0-5507-g618c318a-1jammy [88.9 kB] 2026-02-24T17:58:25.085 INFO:teuthology.orchestra.run.trial068.stdout:Get:12 https://2.chacra.ceph.com/r/ceph/wip-yuri8-testing-2026-02-23-2128/618c318a7a0b287ab5aeff207b7bf1d73ef45044/ubuntu/jammy/flavors/default jammy/main amd64 python3-rbd amd64 20.3.0-5507-g618c318a-1jammy [338 kB] 2026-02-24T17:58:25.091 INFO:teuthology.orchestra.run.trial068.stdout:Get:13 https://2.chacra.ceph.com/r/ceph/wip-yuri8-testing-2026-02-23-2128/618c318a7a0b287ab5aeff207b7bf1d73ef45044/ubuntu/jammy/flavors/default jammy/main amd64 librgw2 amd64 20.3.0-5507-g618c318a-1jammy [9130 kB] 2026-02-24T17:58:25.151 INFO:teuthology.orchestra.run.trial068.stdout:Get:14 https://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5dbus5 amd64 5.15.3+dfsg-2ubuntu0.2 [222 kB] 2026-02-24T17:58:25.163 INFO:teuthology.orchestra.run.trial068.stdout:Get:15 https://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5network5 amd64 5.15.3+dfsg-2ubuntu0.2 [731 kB] 2026-02-24T17:58:25.193 INFO:teuthology.orchestra.run.trial068.stdout:Get:16 https://archive.ubuntu.com/ubuntu jammy/universe amd64 libthrift-0.16.0 amd64 0.16.0-2 [267 kB] 2026-02-24T17:58:25.202 INFO:teuthology.orchestra.run.trial068.stdout:Get:17 https://archive.ubuntu.com/ubuntu jammy/universe amd64 libnbd0 amd64 1.10.5-1 [71.3 kB] 2026-02-24T17:58:25.205 INFO:teuthology.orchestra.run.trial068.stdout:Get:18 https://archive.ubuntu.com/ubuntu jammy/main amd64 python3-wcwidth all 0.2.5+dfsg1-1 [21.9 kB] 2026-02-24T17:58:25.206 INFO:teuthology.orchestra.run.trial068.stdout:Get:19 https://archive.ubuntu.com/ubuntu jammy/main amd64 python3-prettytable all 2.5.0-2 [31.3 kB] 2026-02-24T17:58:25.207 INFO:teuthology.orchestra.run.trial068.stdout:Get:20 https://archive.ubuntu.com/ubuntu jammy/universe amd64 librdkafka1 amd64 1.8.0-1build1 [633 kB] 2026-02-24T17:58:25.226 INFO:teuthology.orchestra.run.trial068.stdout:Get:21 https://archive.ubuntu.com/ubuntu jammy-updates/main amd64 liboath0 amd64 2.6.7-3ubuntu0.1 [41.3 kB] 2026-02-24T17:58:25.227 INFO:teuthology.orchestra.run.trial068.stdout:Get:22 https://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.functools all 3.4.0-2 [9030 B] 2026-02-24T17:58:25.236 INFO:teuthology.orchestra.run.trial068.stdout:Get:23 https://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-cheroot all 8.5.2+ds1-1ubuntu3.1 [71.1 kB] 2026-02-24T17:58:25.310 INFO:teuthology.orchestra.run.trial068.stdout:Get:24 https://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.classes all 3.2.1-3 [6452 B] 2026-02-24T17:58:25.310 INFO:teuthology.orchestra.run.trial068.stdout:Get:25 https://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.text all 3.6.0-2 [8716 B] 2026-02-24T17:58:25.310 INFO:teuthology.orchestra.run.trial068.stdout:Get:26 https://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.collections all 3.4.0-2 [11.4 kB] 2026-02-24T17:58:25.310 INFO:teuthology.orchestra.run.trial068.stdout:Get:27 https://archive.ubuntu.com/ubuntu jammy/main amd64 python3-tempora all 4.1.2-1 [14.8 kB] 2026-02-24T17:58:25.311 INFO:teuthology.orchestra.run.trial068.stdout:Get:28 https://archive.ubuntu.com/ubuntu jammy/main amd64 python3-portend all 3.0.0-1 [7240 B] 2026-02-24T17:58:25.311 INFO:teuthology.orchestra.run.trial068.stdout:Get:29 https://archive.ubuntu.com/ubuntu jammy/main amd64 python3-zc.lockfile all 2.0-1 [8980 B] 2026-02-24T17:58:25.311 INFO:teuthology.orchestra.run.trial068.stdout:Get:30 https://archive.ubuntu.com/ubuntu jammy/main amd64 python3-cherrypy3 all 18.6.1-4 [208 kB] 2026-02-24T17:58:25.318 INFO:teuthology.orchestra.run.trial068.stdout:Get:31 https://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-natsort all 8.0.2-1 [35.3 kB] 2026-02-24T17:58:25.319 INFO:teuthology.orchestra.run.trial068.stdout:Get:32 https://archive.ubuntu.com/ubuntu jammy/universe amd64 libfuse2 amd64 2.9.9-5ubuntu3 [90.3 kB] 2026-02-24T17:58:25.396 INFO:teuthology.orchestra.run.trial068.stdout:Get:33 https://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 python3-asyncssh all 2.5.0-1ubuntu0.1 [189 kB] 2026-02-24T17:58:25.399 INFO:teuthology.orchestra.run.trial068.stdout:Get:34 https://archive.ubuntu.com/ubuntu jammy/main amd64 python3-repoze.lru all 0.7-2 [12.1 kB] 2026-02-24T17:58:25.399 INFO:teuthology.orchestra.run.trial068.stdout:Get:35 https://archive.ubuntu.com/ubuntu jammy/main amd64 python3-routes all 2.5.1-1ubuntu1 [89.0 kB] 2026-02-24T17:58:25.400 INFO:teuthology.orchestra.run.trial068.stdout:Get:36 https://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-sklearn-lib amd64 0.23.2-5ubuntu6 [2058 kB] 2026-02-24T17:58:25.428 INFO:teuthology.orchestra.run.trial068.stdout:Get:37 https://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-joblib all 0.17.0-4ubuntu1 [204 kB] 2026-02-24T17:58:25.430 INFO:teuthology.orchestra.run.trial068.stdout:Get:38 https://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-threadpoolctl all 3.1.0-1 [21.3 kB] 2026-02-24T17:58:25.431 INFO:teuthology.orchestra.run.trial068.stdout:Get:39 https://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-sklearn all 0.23.2-5ubuntu6 [1829 kB] 2026-02-24T17:58:25.497 INFO:teuthology.orchestra.run.trial068.stdout:Get:40 https://archive.ubuntu.com/ubuntu jammy/main amd64 python3-cachetools all 5.0.0-1 [9722 B] 2026-02-24T17:58:25.497 INFO:teuthology.orchestra.run.trial068.stdout:Get:41 https://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-rsa all 4.8-1 [28.4 kB] 2026-02-24T17:58:25.497 INFO:teuthology.orchestra.run.trial068.stdout:Get:42 https://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-google-auth all 1.5.1-3 [35.7 kB] 2026-02-24T17:58:25.497 INFO:teuthology.orchestra.run.trial068.stdout:Get:43 https://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-requests-oauthlib all 1.3.0+ds-0.1 [18.7 kB] 2026-02-24T17:58:25.498 INFO:teuthology.orchestra.run.trial068.stdout:Get:44 https://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-websocket all 1.2.3-1 [34.7 kB] 2026-02-24T17:58:25.498 INFO:teuthology.orchestra.run.trial068.stdout:Get:45 https://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-kubernetes all 12.0.1-1ubuntu1 [353 kB] 2026-02-24T17:58:25.555 INFO:teuthology.orchestra.run.trial068.stdout:Get:46 https://2.chacra.ceph.com/r/ceph/wip-yuri8-testing-2026-02-23-2128/618c318a7a0b287ab5aeff207b7bf1d73ef45044/ubuntu/jammy/flavors/default jammy/main amd64 python3-rgw amd64 20.3.0-5507-g618c318a-1jammy [112 kB] 2026-02-24T17:58:25.556 INFO:teuthology.orchestra.run.trial068.stdout:Get:47 https://2.chacra.ceph.com/r/ceph/wip-yuri8-testing-2026-02-23-2128/618c318a7a0b287ab5aeff207b7bf1d73ef45044/ubuntu/jammy/flavors/default jammy/main amd64 libradosstriper1 amd64 20.3.0-5507-g618c318a-1jammy [261 kB] 2026-02-24T17:58:25.569 INFO:teuthology.orchestra.run.trial068.stdout:Get:48 https://archive.ubuntu.com/ubuntu jammy/main amd64 libonig5 amd64 6.9.7.1-2build1 [172 kB] 2026-02-24T17:58:25.571 INFO:teuthology.orchestra.run.trial068.stdout:Get:49 https://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libjq1 amd64 1.6-2.1ubuntu3.1 [133 kB] 2026-02-24T17:58:25.574 INFO:teuthology.orchestra.run.trial068.stdout:Get:50 https://archive.ubuntu.com/ubuntu jammy-updates/main amd64 jq amd64 1.6-2.1ubuntu3.1 [52.5 kB] 2026-02-24T17:58:25.595 INFO:teuthology.orchestra.run.trial068.stdout:Get:51 https://2.chacra.ceph.com/r/ceph/wip-yuri8-testing-2026-02-23-2128/618c318a7a0b287ab5aeff207b7bf1d73ef45044/ubuntu/jammy/flavors/default jammy/main amd64 ceph-common amd64 20.3.0-5507-g618c318a-1jammy [31.6 MB] 2026-02-24T17:58:25.652 INFO:teuthology.orchestra.run.trial068.stdout:Get:52 https://archive.ubuntu.com/ubuntu jammy/main amd64 socat amd64 1.7.4.1-3ubuntu4 [349 kB] 2026-02-24T17:58:25.655 INFO:teuthology.orchestra.run.trial068.stdout:Get:53 https://archive.ubuntu.com/ubuntu jammy/universe amd64 xmlstarlet amd64 1.6.1-2.1 [265 kB] 2026-02-24T17:58:25.658 INFO:teuthology.orchestra.run.trial068.stdout:Get:54 https://archive.ubuntu.com/ubuntu jammy-updates/main amd64 nvme-cli amd64 1.16-3ubuntu0.3 [474 kB] 2026-02-24T17:58:25.662 INFO:teuthology.orchestra.run.trial068.stdout:Get:55 https://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 python-asyncssh-doc all 2.5.0-1ubuntu0.1 [309 kB] 2026-02-24T17:58:25.665 INFO:teuthology.orchestra.run.trial068.stdout:Get:56 https://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-iniconfig all 1.1.1-2 [6024 B] 2026-02-24T17:58:25.665 INFO:teuthology.orchestra.run.trial068.stdout:Get:57 https://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-pluggy all 0.13.0-7.1 [19.0 kB] 2026-02-24T17:58:25.666 INFO:teuthology.orchestra.run.trial068.stdout:Get:58 https://archive.ubuntu.com/ubuntu jammy/main amd64 python3-psutil amd64 5.9.0-1build1 [158 kB] 2026-02-24T17:58:25.735 INFO:teuthology.orchestra.run.trial068.stdout:Get:59 https://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-py all 1.10.0-1 [71.9 kB] 2026-02-24T17:58:25.736 INFO:teuthology.orchestra.run.trial068.stdout:Get:60 https://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-pygments all 2.11.2+dfsg-2ubuntu0.1 [750 kB] 2026-02-24T17:58:25.743 INFO:teuthology.orchestra.run.trial068.stdout:Get:61 https://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-toml all 0.10.2-1 [16.5 kB] 2026-02-24T17:58:25.817 INFO:teuthology.orchestra.run.trial068.stdout:Get:62 https://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-pytest all 6.2.5-1ubuntu2 [214 kB] 2026-02-24T17:58:25.819 INFO:teuthology.orchestra.run.trial068.stdout:Get:63 https://archive.ubuntu.com/ubuntu jammy/main amd64 python3-simplejson amd64 3.17.6-1build1 [54.7 kB] 2026-02-24T17:58:25.820 INFO:teuthology.orchestra.run.trial068.stdout:Get:64 https://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-webob all 1:1.8.6-1.1ubuntu0.1 [86.7 kB] 2026-02-24T17:58:25.821 INFO:teuthology.orchestra.run.trial068.stdout:Get:65 https://archive.ubuntu.com/ubuntu jammy/universe amd64 qttranslations5-l10n all 5.15.3-1 [1983 kB] 2026-02-24T17:58:27.201 INFO:teuthology.orchestra.run.trial068.stdout:Get:66 https://2.chacra.ceph.com/r/ceph/wip-yuri8-testing-2026-02-23-2128/618c318a7a0b287ab5aeff207b7bf1d73ef45044/ubuntu/jammy/flavors/default jammy/main amd64 ceph-base amd64 20.3.0-5507-g618c318a-1jammy [4984 kB] 2026-02-24T17:58:27.446 INFO:teuthology.orchestra.run.trial068.stdout:Get:67 https://2.chacra.ceph.com/r/ceph/wip-yuri8-testing-2026-02-23-2128/618c318a7a0b287ab5aeff207b7bf1d73ef45044/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-modules-core all 20.3.0-5507-g618c318a-1jammy [295 kB] 2026-02-24T17:58:27.464 INFO:teuthology.orchestra.run.trial068.stdout:Get:68 https://2.chacra.ceph.com/r/ceph/wip-yuri8-testing-2026-02-23-2128/618c318a7a0b287ab5aeff207b7bf1d73ef45044/ubuntu/jammy/flavors/default jammy/main amd64 libsqlite3-mod-ceph amd64 20.3.0-5507-g618c318a-1jammy [127 kB] 2026-02-24T17:58:27.466 INFO:teuthology.orchestra.run.trial068.stdout:Get:69 https://2.chacra.ceph.com/r/ceph/wip-yuri8-testing-2026-02-23-2128/618c318a7a0b287ab5aeff207b7bf1d73ef45044/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr amd64 20.3.0-5507-g618c318a-1jammy [934 kB] 2026-02-24T17:58:27.522 INFO:teuthology.orchestra.run.trial068.stdout:Get:70 https://2.chacra.ceph.com/r/ceph/wip-yuri8-testing-2026-02-23-2128/618c318a7a0b287ab5aeff207b7bf1d73ef45044/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mon amd64 20.3.0-5507-g618c318a-1jammy [7550 kB] 2026-02-24T17:58:28.004 INFO:teuthology.orchestra.run.trial068.stdout:Get:71 https://2.chacra.ceph.com/r/ceph/wip-yuri8-testing-2026-02-23-2128/618c318a7a0b287ab5aeff207b7bf1d73ef45044/ubuntu/jammy/flavors/default jammy/main amd64 ceph-osd-classic amd64 20.3.0-5507-g618c318a-1jammy [468 MB] 2026-02-24T17:59:27.244 INFO:teuthology.orchestra.run.trial068.stdout:Get:72 https://2.chacra.ceph.com/r/ceph/wip-yuri8-testing-2026-02-23-2128/618c318a7a0b287ab5aeff207b7bf1d73ef45044/ubuntu/jammy/flavors/default jammy/main amd64 ceph-osd amd64 20.3.0-5507-g618c318a-1jammy [180 kB] 2026-02-24T17:59:27.245 INFO:teuthology.orchestra.run.trial068.stdout:Get:73 https://2.chacra.ceph.com/r/ceph/wip-yuri8-testing-2026-02-23-2128/618c318a7a0b287ab5aeff207b7bf1d73ef45044/ubuntu/jammy/flavors/default jammy/main amd64 ceph amd64 20.3.0-5507-g618c318a-1jammy [14.1 kB] 2026-02-24T17:59:27.245 INFO:teuthology.orchestra.run.trial068.stdout:Get:74 https://2.chacra.ceph.com/r/ceph/wip-yuri8-testing-2026-02-23-2128/618c318a7a0b287ab5aeff207b7bf1d73ef45044/ubuntu/jammy/flavors/default jammy/main amd64 ceph-fuse amd64 20.3.0-5507-g618c318a-1jammy [1030 kB] 2026-02-24T17:59:27.561 INFO:teuthology.orchestra.run.trial068.stdout:Get:75 https://2.chacra.ceph.com/r/ceph/wip-yuri8-testing-2026-02-23-2128/618c318a7a0b287ab5aeff207b7bf1d73ef45044/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mds amd64 20.3.0-5507-g618c318a-1jammy [2437 kB] 2026-02-24T17:59:28.236 INFO:teuthology.orchestra.run.trial068.stdout:Get:76 https://2.chacra.ceph.com/r/ceph/wip-yuri8-testing-2026-02-23-2128/618c318a7a0b287ab5aeff207b7bf1d73ef45044/ubuntu/jammy/flavors/default jammy/main amd64 cephadm amd64 20.3.0-5507-g618c318a-1jammy [1072 kB] 2026-02-24T17:59:28.509 INFO:teuthology.orchestra.run.trial068.stdout:Get:77 https://2.chacra.ceph.com/r/ceph/wip-yuri8-testing-2026-02-23-2128/618c318a7a0b287ab5aeff207b7bf1d73ef45044/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-cephadm all 20.3.0-5507-g618c318a-1jammy [190 kB] 2026-02-24T17:59:28.576 INFO:teuthology.orchestra.run.trial068.stdout:Get:78 https://2.chacra.ceph.com/r/ceph/wip-yuri8-testing-2026-02-23-2128/618c318a7a0b287ab5aeff207b7bf1d73ef45044/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-dashboard all 20.3.0-5507-g618c318a-1jammy [49.0 MB] 2026-02-24T17:59:43.937 INFO:teuthology.orchestra.run.trial068.stdout:Get:79 https://2.chacra.ceph.com/r/ceph/wip-yuri8-testing-2026-02-23-2128/618c318a7a0b287ab5aeff207b7bf1d73ef45044/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-diskprediction-local all 20.3.0-5507-g618c318a-1jammy [8626 kB] 2026-02-24T17:59:46.830 INFO:teuthology.orchestra.run.trial068.stdout:Get:80 https://2.chacra.ceph.com/r/ceph/wip-yuri8-testing-2026-02-23-2128/618c318a7a0b287ab5aeff207b7bf1d73ef45044/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-k8sevents all 20.3.0-5507-g618c318a-1jammy [14.2 kB] 2026-02-24T17:59:46.830 INFO:teuthology.orchestra.run.trial068.stdout:Get:81 https://2.chacra.ceph.com/r/ceph/wip-yuri8-testing-2026-02-23-2128/618c318a7a0b287ab5aeff207b7bf1d73ef45044/ubuntu/jammy/flavors/default jammy/main amd64 ceph-test amd64 20.3.0-5507-g618c318a-1jammy [65.5 MB] 2026-02-24T18:00:04.236 INFO:teuthology.orchestra.run.trial068.stdout:Get:82 https://2.chacra.ceph.com/r/ceph/wip-yuri8-testing-2026-02-23-2128/618c318a7a0b287ab5aeff207b7bf1d73ef45044/ubuntu/jammy/flavors/default jammy/main amd64 ceph-volume all 20.3.0-5507-g618c318a-1jammy [136 kB] 2026-02-24T18:00:04.237 INFO:teuthology.orchestra.run.trial068.stdout:Get:83 https://2.chacra.ceph.com/r/ceph/wip-yuri8-testing-2026-02-23-2128/618c318a7a0b287ab5aeff207b7bf1d73ef45044/ubuntu/jammy/flavors/default jammy/main amd64 libcephfs-daemon amd64 20.3.0-5507-g618c318a-1jammy [47.3 kB] 2026-02-24T18:00:04.237 INFO:teuthology.orchestra.run.trial068.stdout:Get:84 https://2.chacra.ceph.com/r/ceph/wip-yuri8-testing-2026-02-23-2128/618c318a7a0b287ab5aeff207b7bf1d73ef45044/ubuntu/jammy/flavors/default jammy/main amd64 libcephfs-proxy2 amd64 20.3.0-5507-g618c318a-1jammy [33.1 kB] 2026-02-24T18:00:04.239 INFO:teuthology.orchestra.run.trial068.stdout:Get:85 https://2.chacra.ceph.com/r/ceph/wip-yuri8-testing-2026-02-23-2128/618c318a7a0b287ab5aeff207b7bf1d73ef45044/ubuntu/jammy/flavors/default jammy/main amd64 libcephfs-dev amd64 20.3.0-5507-g618c318a-1jammy [44.2 kB] 2026-02-24T18:00:04.241 INFO:teuthology.orchestra.run.trial068.stdout:Get:86 https://2.chacra.ceph.com/r/ceph/wip-yuri8-testing-2026-02-23-2128/618c318a7a0b287ab5aeff207b7bf1d73ef45044/ubuntu/jammy/flavors/default jammy/main amd64 radosgw amd64 20.3.0-5507-g618c318a-1jammy [18.4 MB] 2026-02-24T18:00:06.285 INFO:teuthology.orchestra.run.trial068.stdout:Get:87 https://2.chacra.ceph.com/r/ceph/wip-yuri8-testing-2026-02-23-2128/618c318a7a0b287ab5aeff207b7bf1d73ef45044/ubuntu/jammy/flavors/default jammy/main amd64 rbd-fuse amd64 20.3.0-5507-g618c318a-1jammy [93.7 kB] 2026-02-24T18:00:06.560 INFO:teuthology.orchestra.run.trial068.stdout:Fetched 693 MB in 1min 42s (6780 kB/s) 2026-02-24T18:00:06.580 INFO:teuthology.orchestra.run.trial068.stdout:Selecting previously unselected package liblttng-ust1:amd64. 2026-02-24T18:00:06.747 INFO:teuthology.orchestra.run.trial068.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 202926 files and directories currently installed.) 2026-02-24T18:00:06.750 INFO:teuthology.orchestra.run.trial068.stdout:Preparing to unpack .../00-liblttng-ust1_2.13.1-1ubuntu1_amd64.deb ... 2026-02-24T18:00:06.751 INFO:teuthology.orchestra.run.trial068.stdout:Unpacking liblttng-ust1:amd64 (2.13.1-1ubuntu1) ... 2026-02-24T18:00:06.773 INFO:teuthology.orchestra.run.trial068.stdout:Selecting previously unselected package libdouble-conversion3:amd64. 2026-02-24T18:00:06.787 INFO:teuthology.orchestra.run.trial068.stdout:Preparing to unpack .../01-libdouble-conversion3_3.1.7-4_amd64.deb ... 2026-02-24T18:00:06.788 INFO:teuthology.orchestra.run.trial068.stdout:Unpacking libdouble-conversion3:amd64 (3.1.7-4) ... 2026-02-24T18:00:06.805 INFO:teuthology.orchestra.run.trial068.stdout:Selecting previously unselected package libpcre2-16-0:amd64. 2026-02-24T18:00:06.819 INFO:teuthology.orchestra.run.trial068.stdout:Preparing to unpack .../02-libpcre2-16-0_10.39-3ubuntu0.1_amd64.deb ... 2026-02-24T18:00:06.819 INFO:teuthology.orchestra.run.trial068.stdout:Unpacking libpcre2-16-0:amd64 (10.39-3ubuntu0.1) ... 2026-02-24T18:00:06.838 INFO:teuthology.orchestra.run.trial068.stdout:Selecting previously unselected package libqt5core5a:amd64. 2026-02-24T18:00:06.851 INFO:teuthology.orchestra.run.trial068.stdout:Preparing to unpack .../03-libqt5core5a_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2026-02-24T18:00:06.855 INFO:teuthology.orchestra.run.trial068.stdout:Unpacking libqt5core5a:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2026-02-24T18:00:06.887 INFO:teuthology.orchestra.run.trial068.stdout:Selecting previously unselected package libqt5dbus5:amd64. 2026-02-24T18:00:06.897 INFO:teuthology.orchestra.run.trial068.stdout:Preparing to unpack .../04-libqt5dbus5_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2026-02-24T18:00:06.897 INFO:teuthology.orchestra.run.trial068.stdout:Unpacking libqt5dbus5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2026-02-24T18:00:06.914 INFO:teuthology.orchestra.run.trial068.stdout:Selecting previously unselected package libqt5network5:amd64. 2026-02-24T18:00:06.924 INFO:teuthology.orchestra.run.trial068.stdout:Preparing to unpack .../05-libqt5network5_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2026-02-24T18:00:06.924 INFO:teuthology.orchestra.run.trial068.stdout:Unpacking libqt5network5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2026-02-24T18:00:06.944 INFO:teuthology.orchestra.run.trial068.stdout:Selecting previously unselected package libthrift-0.16.0:amd64. 2026-02-24T18:00:06.953 INFO:teuthology.orchestra.run.trial068.stdout:Preparing to unpack .../06-libthrift-0.16.0_0.16.0-2_amd64.deb ... 2026-02-24T18:00:06.954 INFO:teuthology.orchestra.run.trial068.stdout:Unpacking libthrift-0.16.0:amd64 (0.16.0-2) ... 2026-02-24T18:00:06.970 INFO:teuthology.orchestra.run.trial068.stdout:Selecting previously unselected package librados2. 2026-02-24T18:00:06.980 INFO:teuthology.orchestra.run.trial068.stdout:Preparing to unpack .../07-librados2_20.3.0-5507-g618c318a-1jammy_amd64.deb ... 2026-02-24T18:00:06.981 INFO:teuthology.orchestra.run.trial068.stdout:Unpacking librados2 (20.3.0-5507-g618c318a-1jammy) ... 2026-02-24T18:00:07.024 INFO:teuthology.orchestra.run.trial068.stdout:Selecting previously unselected package libnbd0. 2026-02-24T18:00:07.034 INFO:teuthology.orchestra.run.trial068.stdout:Preparing to unpack .../08-libnbd0_1.10.5-1_amd64.deb ... 2026-02-24T18:00:07.034 INFO:teuthology.orchestra.run.trial068.stdout:Unpacking libnbd0 (1.10.5-1) ... 2026-02-24T18:00:07.050 INFO:teuthology.orchestra.run.trial068.stdout:Selecting previously unselected package librbd1. 2026-02-24T18:00:07.060 INFO:teuthology.orchestra.run.trial068.stdout:Preparing to unpack .../09-librbd1_20.3.0-5507-g618c318a-1jammy_amd64.deb ... 2026-02-24T18:00:07.061 INFO:teuthology.orchestra.run.trial068.stdout:Unpacking librbd1 (20.3.0-5507-g618c318a-1jammy) ... 2026-02-24T18:00:07.097 INFO:teuthology.orchestra.run.trial068.stdout:Selecting previously unselected package libcephfs2. 2026-02-24T18:00:07.107 INFO:teuthology.orchestra.run.trial068.stdout:Preparing to unpack .../10-libcephfs2_20.3.0-5507-g618c318a-1jammy_amd64.deb ... 2026-02-24T18:00:07.108 INFO:teuthology.orchestra.run.trial068.stdout:Unpacking libcephfs2 (20.3.0-5507-g618c318a-1jammy) ... 2026-02-24T18:00:07.129 INFO:teuthology.orchestra.run.trial068.stdout:Selecting previously unselected package python3-rados. 2026-02-24T18:00:07.139 INFO:teuthology.orchestra.run.trial068.stdout:Preparing to unpack .../11-python3-rados_20.3.0-5507-g618c318a-1jammy_amd64.deb ... 2026-02-24T18:00:07.139 INFO:teuthology.orchestra.run.trial068.stdout:Unpacking python3-rados (20.3.0-5507-g618c318a-1jammy) ... 2026-02-24T18:00:07.157 INFO:teuthology.orchestra.run.trial068.stdout:Selecting previously unselected package python3-ceph-argparse. 2026-02-24T18:00:07.167 INFO:teuthology.orchestra.run.trial068.stdout:Preparing to unpack .../12-python3-ceph-argparse_20.3.0-5507-g618c318a-1jammy_all.deb ... 2026-02-24T18:00:07.168 INFO:teuthology.orchestra.run.trial068.stdout:Unpacking python3-ceph-argparse (20.3.0-5507-g618c318a-1jammy) ... 2026-02-24T18:00:07.182 INFO:teuthology.orchestra.run.trial068.stdout:Selecting previously unselected package python3-cephfs. 2026-02-24T18:00:07.192 INFO:teuthology.orchestra.run.trial068.stdout:Preparing to unpack .../13-python3-cephfs_20.3.0-5507-g618c318a-1jammy_amd64.deb ... 2026-02-24T18:00:07.193 INFO:teuthology.orchestra.run.trial068.stdout:Unpacking python3-cephfs (20.3.0-5507-g618c318a-1jammy) ... 2026-02-24T18:00:07.210 INFO:teuthology.orchestra.run.trial068.stdout:Selecting previously unselected package python3-ceph-common. 2026-02-24T18:00:07.219 INFO:teuthology.orchestra.run.trial068.stdout:Preparing to unpack .../14-python3-ceph-common_20.3.0-5507-g618c318a-1jammy_all.deb ... 2026-02-24T18:00:07.220 INFO:teuthology.orchestra.run.trial068.stdout:Unpacking python3-ceph-common (20.3.0-5507-g618c318a-1jammy) ... 2026-02-24T18:00:07.238 INFO:teuthology.orchestra.run.trial068.stdout:Selecting previously unselected package python3-wcwidth. 2026-02-24T18:00:07.248 INFO:teuthology.orchestra.run.trial068.stdout:Preparing to unpack .../15-python3-wcwidth_0.2.5+dfsg1-1_all.deb ... 2026-02-24T18:00:07.249 INFO:teuthology.orchestra.run.trial068.stdout:Unpacking python3-wcwidth (0.2.5+dfsg1-1) ... 2026-02-24T18:00:07.265 INFO:teuthology.orchestra.run.trial068.stdout:Selecting previously unselected package python3-prettytable. 2026-02-24T18:00:07.275 INFO:teuthology.orchestra.run.trial068.stdout:Preparing to unpack .../16-python3-prettytable_2.5.0-2_all.deb ... 2026-02-24T18:00:07.276 INFO:teuthology.orchestra.run.trial068.stdout:Unpacking python3-prettytable (2.5.0-2) ... 2026-02-24T18:00:07.291 INFO:teuthology.orchestra.run.trial068.stdout:Selecting previously unselected package python3-rbd. 2026-02-24T18:00:07.301 INFO:teuthology.orchestra.run.trial068.stdout:Preparing to unpack .../17-python3-rbd_20.3.0-5507-g618c318a-1jammy_amd64.deb ... 2026-02-24T18:00:07.302 INFO:teuthology.orchestra.run.trial068.stdout:Unpacking python3-rbd (20.3.0-5507-g618c318a-1jammy) ... 2026-02-24T18:00:07.320 INFO:teuthology.orchestra.run.trial068.stdout:Selecting previously unselected package librdkafka1:amd64. 2026-02-24T18:00:07.330 INFO:teuthology.orchestra.run.trial068.stdout:Preparing to unpack .../18-librdkafka1_1.8.0-1build1_amd64.deb ... 2026-02-24T18:00:07.330 INFO:teuthology.orchestra.run.trial068.stdout:Unpacking librdkafka1:amd64 (1.8.0-1build1) ... 2026-02-24T18:00:07.349 INFO:teuthology.orchestra.run.trial068.stdout:Selecting previously unselected package librgw2. 2026-02-24T18:00:07.359 INFO:teuthology.orchestra.run.trial068.stdout:Preparing to unpack .../19-librgw2_20.3.0-5507-g618c318a-1jammy_amd64.deb ... 2026-02-24T18:00:07.359 INFO:teuthology.orchestra.run.trial068.stdout:Unpacking librgw2 (20.3.0-5507-g618c318a-1jammy) ... 2026-02-24T18:00:07.459 INFO:teuthology.orchestra.run.trial068.stdout:Selecting previously unselected package python3-rgw. 2026-02-24T18:00:07.470 INFO:teuthology.orchestra.run.trial068.stdout:Preparing to unpack .../20-python3-rgw_20.3.0-5507-g618c318a-1jammy_amd64.deb ... 2026-02-24T18:00:07.471 INFO:teuthology.orchestra.run.trial068.stdout:Unpacking python3-rgw (20.3.0-5507-g618c318a-1jammy) ... 2026-02-24T18:00:07.486 INFO:teuthology.orchestra.run.trial068.stdout:Selecting previously unselected package liboath0:amd64. 2026-02-24T18:00:07.496 INFO:teuthology.orchestra.run.trial068.stdout:Preparing to unpack .../21-liboath0_2.6.7-3ubuntu0.1_amd64.deb ... 2026-02-24T18:00:07.496 INFO:teuthology.orchestra.run.trial068.stdout:Unpacking liboath0:amd64 (2.6.7-3ubuntu0.1) ... 2026-02-24T18:00:07.511 INFO:teuthology.orchestra.run.trial068.stdout:Selecting previously unselected package libradosstriper1. 2026-02-24T18:00:07.521 INFO:teuthology.orchestra.run.trial068.stdout:Preparing to unpack .../22-libradosstriper1_20.3.0-5507-g618c318a-1jammy_amd64.deb ... 2026-02-24T18:00:07.521 INFO:teuthology.orchestra.run.trial068.stdout:Unpacking libradosstriper1 (20.3.0-5507-g618c318a-1jammy) ... 2026-02-24T18:00:07.538 INFO:teuthology.orchestra.run.trial068.stdout:Selecting previously unselected package ceph-common. 2026-02-24T18:00:07.548 INFO:teuthology.orchestra.run.trial068.stdout:Preparing to unpack .../23-ceph-common_20.3.0-5507-g618c318a-1jammy_amd64.deb ... 2026-02-24T18:00:07.548 INFO:teuthology.orchestra.run.trial068.stdout:Unpacking ceph-common (20.3.0-5507-g618c318a-1jammy) ... 2026-02-24T18:00:07.811 INFO:teuthology.orchestra.run.trial068.stdout:Selecting previously unselected package ceph-base. 2026-02-24T18:00:07.821 INFO:teuthology.orchestra.run.trial068.stdout:Preparing to unpack .../24-ceph-base_20.3.0-5507-g618c318a-1jammy_amd64.deb ... 2026-02-24T18:00:07.827 INFO:teuthology.orchestra.run.trial068.stdout:Unpacking ceph-base (20.3.0-5507-g618c318a-1jammy) ... 2026-02-24T18:00:07.885 INFO:teuthology.orchestra.run.trial068.stdout:Selecting previously unselected package python3-jaraco.functools. 2026-02-24T18:00:07.895 INFO:teuthology.orchestra.run.trial068.stdout:Preparing to unpack .../25-python3-jaraco.functools_3.4.0-2_all.deb ... 2026-02-24T18:00:07.896 INFO:teuthology.orchestra.run.trial068.stdout:Unpacking python3-jaraco.functools (3.4.0-2) ... 2026-02-24T18:00:07.911 INFO:teuthology.orchestra.run.trial068.stdout:Selecting previously unselected package python3-cheroot. 2026-02-24T18:00:07.921 INFO:teuthology.orchestra.run.trial068.stdout:Preparing to unpack .../26-python3-cheroot_8.5.2+ds1-1ubuntu3.1_all.deb ... 2026-02-24T18:00:07.921 INFO:teuthology.orchestra.run.trial068.stdout:Unpacking python3-cheroot (8.5.2+ds1-1ubuntu3.1) ... 2026-02-24T18:00:07.940 INFO:teuthology.orchestra.run.trial068.stdout:Selecting previously unselected package python3-jaraco.classes. 2026-02-24T18:00:07.950 INFO:teuthology.orchestra.run.trial068.stdout:Preparing to unpack .../27-python3-jaraco.classes_3.2.1-3_all.deb ... 2026-02-24T18:00:07.950 INFO:teuthology.orchestra.run.trial068.stdout:Unpacking python3-jaraco.classes (3.2.1-3) ... 2026-02-24T18:00:07.965 INFO:teuthology.orchestra.run.trial068.stdout:Selecting previously unselected package python3-jaraco.text. 2026-02-24T18:00:07.976 INFO:teuthology.orchestra.run.trial068.stdout:Preparing to unpack .../28-python3-jaraco.text_3.6.0-2_all.deb ... 2026-02-24T18:00:07.976 INFO:teuthology.orchestra.run.trial068.stdout:Unpacking python3-jaraco.text (3.6.0-2) ... 2026-02-24T18:00:07.991 INFO:teuthology.orchestra.run.trial068.stdout:Selecting previously unselected package python3-jaraco.collections. 2026-02-24T18:00:08.001 INFO:teuthology.orchestra.run.trial068.stdout:Preparing to unpack .../29-python3-jaraco.collections_3.4.0-2_all.deb ... 2026-02-24T18:00:08.001 INFO:teuthology.orchestra.run.trial068.stdout:Unpacking python3-jaraco.collections (3.4.0-2) ... 2026-02-24T18:00:08.015 INFO:teuthology.orchestra.run.trial068.stdout:Selecting previously unselected package python3-tempora. 2026-02-24T18:00:08.025 INFO:teuthology.orchestra.run.trial068.stdout:Preparing to unpack .../30-python3-tempora_4.1.2-1_all.deb ... 2026-02-24T18:00:08.026 INFO:teuthology.orchestra.run.trial068.stdout:Unpacking python3-tempora (4.1.2-1) ... 2026-02-24T18:00:08.042 INFO:teuthology.orchestra.run.trial068.stdout:Selecting previously unselected package python3-portend. 2026-02-24T18:00:08.051 INFO:teuthology.orchestra.run.trial068.stdout:Preparing to unpack .../31-python3-portend_3.0.0-1_all.deb ... 2026-02-24T18:00:08.052 INFO:teuthology.orchestra.run.trial068.stdout:Unpacking python3-portend (3.0.0-1) ... 2026-02-24T18:00:08.066 INFO:teuthology.orchestra.run.trial068.stdout:Selecting previously unselected package python3-zc.lockfile. 2026-02-24T18:00:08.076 INFO:teuthology.orchestra.run.trial068.stdout:Preparing to unpack .../32-python3-zc.lockfile_2.0-1_all.deb ... 2026-02-24T18:00:08.076 INFO:teuthology.orchestra.run.trial068.stdout:Unpacking python3-zc.lockfile (2.0-1) ... 2026-02-24T18:00:08.092 INFO:teuthology.orchestra.run.trial068.stdout:Selecting previously unselected package python3-cherrypy3. 2026-02-24T18:00:08.101 INFO:teuthology.orchestra.run.trial068.stdout:Preparing to unpack .../33-python3-cherrypy3_18.6.1-4_all.deb ... 2026-02-24T18:00:08.102 INFO:teuthology.orchestra.run.trial068.stdout:Unpacking python3-cherrypy3 (18.6.1-4) ... 2026-02-24T18:00:08.125 INFO:teuthology.orchestra.run.trial068.stdout:Selecting previously unselected package python3-natsort. 2026-02-24T18:00:08.135 INFO:teuthology.orchestra.run.trial068.stdout:Preparing to unpack .../34-python3-natsort_8.0.2-1_all.deb ... 2026-02-24T18:00:08.135 INFO:teuthology.orchestra.run.trial068.stdout:Unpacking python3-natsort (8.0.2-1) ... 2026-02-24T18:00:08.152 INFO:teuthology.orchestra.run.trial068.stdout:Selecting previously unselected package ceph-mgr-modules-core. 2026-02-24T18:00:08.161 INFO:teuthology.orchestra.run.trial068.stdout:Preparing to unpack .../35-ceph-mgr-modules-core_20.3.0-5507-g618c318a-1jammy_all.deb ... 2026-02-24T18:00:08.162 INFO:teuthology.orchestra.run.trial068.stdout:Unpacking ceph-mgr-modules-core (20.3.0-5507-g618c318a-1jammy) ... 2026-02-24T18:00:08.193 INFO:teuthology.orchestra.run.trial068.stdout:Selecting previously unselected package libsqlite3-mod-ceph. 2026-02-24T18:00:08.203 INFO:teuthology.orchestra.run.trial068.stdout:Preparing to unpack .../36-libsqlite3-mod-ceph_20.3.0-5507-g618c318a-1jammy_amd64.deb ... 2026-02-24T18:00:08.203 INFO:teuthology.orchestra.run.trial068.stdout:Unpacking libsqlite3-mod-ceph (20.3.0-5507-g618c318a-1jammy) ... 2026-02-24T18:00:08.220 INFO:teuthology.orchestra.run.trial068.stdout:Selecting previously unselected package ceph-mgr. 2026-02-24T18:00:08.229 INFO:teuthology.orchestra.run.trial068.stdout:Preparing to unpack .../37-ceph-mgr_20.3.0-5507-g618c318a-1jammy_amd64.deb ... 2026-02-24T18:00:08.230 INFO:teuthology.orchestra.run.trial068.stdout:Unpacking ceph-mgr (20.3.0-5507-g618c318a-1jammy) ... 2026-02-24T18:00:08.251 INFO:teuthology.orchestra.run.trial068.stdout:Selecting previously unselected package ceph-mon. 2026-02-24T18:00:08.261 INFO:teuthology.orchestra.run.trial068.stdout:Preparing to unpack .../38-ceph-mon_20.3.0-5507-g618c318a-1jammy_amd64.deb ... 2026-02-24T18:00:08.262 INFO:teuthology.orchestra.run.trial068.stdout:Unpacking ceph-mon (20.3.0-5507-g618c318a-1jammy) ... 2026-02-24T18:00:08.325 INFO:teuthology.orchestra.run.trial068.stdout:Selecting previously unselected package libfuse2:amd64. 2026-02-24T18:00:08.335 INFO:teuthology.orchestra.run.trial068.stdout:Preparing to unpack .../39-libfuse2_2.9.9-5ubuntu3_amd64.deb ... 2026-02-24T18:00:08.336 INFO:teuthology.orchestra.run.trial068.stdout:Unpacking libfuse2:amd64 (2.9.9-5ubuntu3) ... 2026-02-24T18:00:08.352 INFO:teuthology.orchestra.run.trial068.stdout:Selecting previously unselected package ceph-osd-classic. 2026-02-24T18:00:08.362 INFO:teuthology.orchestra.run.trial068.stdout:Preparing to unpack .../40-ceph-osd-classic_20.3.0-5507-g618c318a-1jammy_amd64.deb ... 2026-02-24T18:00:08.362 INFO:teuthology.orchestra.run.trial068.stdout:Unpacking ceph-osd-classic (20.3.0-5507-g618c318a-1jammy) ... 2026-02-24T18:00:12.315 INFO:teuthology.orchestra.run.trial068.stdout:Selecting previously unselected package ceph-osd. 2026-02-24T18:00:12.330 INFO:teuthology.orchestra.run.trial068.stdout:Preparing to unpack .../41-ceph-osd_20.3.0-5507-g618c318a-1jammy_amd64.deb ... 2026-02-24T18:00:12.331 INFO:teuthology.orchestra.run.trial068.stdout:Unpacking ceph-osd (20.3.0-5507-g618c318a-1jammy) ... 2026-02-24T18:00:12.351 INFO:teuthology.orchestra.run.trial068.stdout:Selecting previously unselected package ceph. 2026-02-24T18:00:12.367 INFO:teuthology.orchestra.run.trial068.stdout:Preparing to unpack .../42-ceph_20.3.0-5507-g618c318a-1jammy_amd64.deb ... 2026-02-24T18:00:12.367 INFO:teuthology.orchestra.run.trial068.stdout:Unpacking ceph (20.3.0-5507-g618c318a-1jammy) ... 2026-02-24T18:00:12.382 INFO:teuthology.orchestra.run.trial068.stdout:Selecting previously unselected package ceph-fuse. 2026-02-24T18:00:12.398 INFO:teuthology.orchestra.run.trial068.stdout:Preparing to unpack .../43-ceph-fuse_20.3.0-5507-g618c318a-1jammy_amd64.deb ... 2026-02-24T18:00:12.398 INFO:teuthology.orchestra.run.trial068.stdout:Unpacking ceph-fuse (20.3.0-5507-g618c318a-1jammy) ... 2026-02-24T18:00:12.423 INFO:teuthology.orchestra.run.trial068.stdout:Selecting previously unselected package ceph-mds. 2026-02-24T18:00:12.435 INFO:teuthology.orchestra.run.trial068.stdout:Preparing to unpack .../44-ceph-mds_20.3.0-5507-g618c318a-1jammy_amd64.deb ... 2026-02-24T18:00:12.435 INFO:teuthology.orchestra.run.trial068.stdout:Unpacking ceph-mds (20.3.0-5507-g618c318a-1jammy) ... 2026-02-24T18:00:12.469 INFO:teuthology.orchestra.run.trial068.stdout:Selecting previously unselected package cephadm. 2026-02-24T18:00:12.480 INFO:teuthology.orchestra.run.trial068.stdout:Preparing to unpack .../45-cephadm_20.3.0-5507-g618c318a-1jammy_amd64.deb ... 2026-02-24T18:00:12.480 INFO:teuthology.orchestra.run.trial068.stdout:Unpacking cephadm (20.3.0-5507-g618c318a-1jammy) ... 2026-02-24T18:00:12.496 INFO:teuthology.orchestra.run.trial068.stdout:Selecting previously unselected package python3-asyncssh. 2026-02-24T18:00:12.506 INFO:teuthology.orchestra.run.trial068.stdout:Preparing to unpack .../46-python3-asyncssh_2.5.0-1ubuntu0.1_all.deb ... 2026-02-24T18:00:12.506 INFO:teuthology.orchestra.run.trial068.stdout:Unpacking python3-asyncssh (2.5.0-1ubuntu0.1) ... 2026-02-24T18:00:12.528 INFO:teuthology.orchestra.run.trial068.stdout:Selecting previously unselected package ceph-mgr-cephadm. 2026-02-24T18:00:12.539 INFO:teuthology.orchestra.run.trial068.stdout:Preparing to unpack .../47-ceph-mgr-cephadm_20.3.0-5507-g618c318a-1jammy_all.deb ... 2026-02-24T18:00:12.539 INFO:teuthology.orchestra.run.trial068.stdout:Unpacking ceph-mgr-cephadm (20.3.0-5507-g618c318a-1jammy) ... 2026-02-24T18:00:12.560 INFO:teuthology.orchestra.run.trial068.stdout:Selecting previously unselected package python3-repoze.lru. 2026-02-24T18:00:12.570 INFO:teuthology.orchestra.run.trial068.stdout:Preparing to unpack .../48-python3-repoze.lru_0.7-2_all.deb ... 2026-02-24T18:00:12.570 INFO:teuthology.orchestra.run.trial068.stdout:Unpacking python3-repoze.lru (0.7-2) ... 2026-02-24T18:00:12.586 INFO:teuthology.orchestra.run.trial068.stdout:Selecting previously unselected package python3-routes. 2026-02-24T18:00:12.596 INFO:teuthology.orchestra.run.trial068.stdout:Preparing to unpack .../49-python3-routes_2.5.1-1ubuntu1_all.deb ... 2026-02-24T18:00:12.597 INFO:teuthology.orchestra.run.trial068.stdout:Unpacking python3-routes (2.5.1-1ubuntu1) ... 2026-02-24T18:00:12.618 INFO:teuthology.orchestra.run.trial068.stdout:Selecting previously unselected package ceph-mgr-dashboard. 2026-02-24T18:00:12.627 INFO:teuthology.orchestra.run.trial068.stdout:Preparing to unpack .../50-ceph-mgr-dashboard_20.3.0-5507-g618c318a-1jammy_all.deb ... 2026-02-24T18:00:12.628 INFO:teuthology.orchestra.run.trial068.stdout:Unpacking ceph-mgr-dashboard (20.3.0-5507-g618c318a-1jammy) ... 2026-02-24T18:00:13.172 INFO:teuthology.orchestra.run.trial068.stdout:Selecting previously unselected package python3-sklearn-lib:amd64. 2026-02-24T18:00:13.184 INFO:teuthology.orchestra.run.trial068.stdout:Preparing to unpack .../51-python3-sklearn-lib_0.23.2-5ubuntu6_amd64.deb ... 2026-02-24T18:00:13.185 INFO:teuthology.orchestra.run.trial068.stdout:Unpacking python3-sklearn-lib:amd64 (0.23.2-5ubuntu6) ... 2026-02-24T18:00:13.222 INFO:teuthology.orchestra.run.trial068.stdout:Selecting previously unselected package python3-joblib. 2026-02-24T18:00:13.232 INFO:teuthology.orchestra.run.trial068.stdout:Preparing to unpack .../52-python3-joblib_0.17.0-4ubuntu1_all.deb ... 2026-02-24T18:00:13.232 INFO:teuthology.orchestra.run.trial068.stdout:Unpacking python3-joblib (0.17.0-4ubuntu1) ... 2026-02-24T18:00:13.259 INFO:teuthology.orchestra.run.trial068.stdout:Selecting previously unselected package python3-threadpoolctl. 2026-02-24T18:00:13.270 INFO:teuthology.orchestra.run.trial068.stdout:Preparing to unpack .../53-python3-threadpoolctl_3.1.0-1_all.deb ... 2026-02-24T18:00:13.270 INFO:teuthology.orchestra.run.trial068.stdout:Unpacking python3-threadpoolctl (3.1.0-1) ... 2026-02-24T18:00:13.285 INFO:teuthology.orchestra.run.trial068.stdout:Selecting previously unselected package python3-sklearn. 2026-02-24T18:00:13.296 INFO:teuthology.orchestra.run.trial068.stdout:Preparing to unpack .../54-python3-sklearn_0.23.2-5ubuntu6_all.deb ... 2026-02-24T18:00:13.296 INFO:teuthology.orchestra.run.trial068.stdout:Unpacking python3-sklearn (0.23.2-5ubuntu6) ... 2026-02-24T18:00:13.379 INFO:teuthology.orchestra.run.trial068.stdout:Selecting previously unselected package ceph-mgr-diskprediction-local. 2026-02-24T18:00:13.390 INFO:teuthology.orchestra.run.trial068.stdout:Preparing to unpack .../55-ceph-mgr-diskprediction-local_20.3.0-5507-g618c318a-1jammy_all.deb ... 2026-02-24T18:00:13.391 INFO:teuthology.orchestra.run.trial068.stdout:Unpacking ceph-mgr-diskprediction-local (20.3.0-5507-g618c318a-1jammy) ... 2026-02-24T18:00:13.549 INFO:teuthology.orchestra.run.trial068.stdout:Selecting previously unselected package python3-cachetools. 2026-02-24T18:00:13.559 INFO:teuthology.orchestra.run.trial068.stdout:Preparing to unpack .../56-python3-cachetools_5.0.0-1_all.deb ... 2026-02-24T18:00:13.559 INFO:teuthology.orchestra.run.trial068.stdout:Unpacking python3-cachetools (5.0.0-1) ... 2026-02-24T18:00:13.574 INFO:teuthology.orchestra.run.trial068.stdout:Selecting previously unselected package python3-rsa. 2026-02-24T18:00:13.584 INFO:teuthology.orchestra.run.trial068.stdout:Preparing to unpack .../57-python3-rsa_4.8-1_all.deb ... 2026-02-24T18:00:13.585 INFO:teuthology.orchestra.run.trial068.stdout:Unpacking python3-rsa (4.8-1) ... 2026-02-24T18:00:13.603 INFO:teuthology.orchestra.run.trial068.stdout:Selecting previously unselected package python3-google-auth. 2026-02-24T18:00:13.613 INFO:teuthology.orchestra.run.trial068.stdout:Preparing to unpack .../58-python3-google-auth_1.5.1-3_all.deb ... 2026-02-24T18:00:13.614 INFO:teuthology.orchestra.run.trial068.stdout:Unpacking python3-google-auth (1.5.1-3) ... 2026-02-24T18:00:13.632 INFO:teuthology.orchestra.run.trial068.stdout:Selecting previously unselected package python3-requests-oauthlib. 2026-02-24T18:00:13.642 INFO:teuthology.orchestra.run.trial068.stdout:Preparing to unpack .../59-python3-requests-oauthlib_1.3.0+ds-0.1_all.deb ... 2026-02-24T18:00:13.642 INFO:teuthology.orchestra.run.trial068.stdout:Unpacking python3-requests-oauthlib (1.3.0+ds-0.1) ... 2026-02-24T18:00:13.659 INFO:teuthology.orchestra.run.trial068.stdout:Selecting previously unselected package python3-websocket. 2026-02-24T18:00:13.669 INFO:teuthology.orchestra.run.trial068.stdout:Preparing to unpack .../60-python3-websocket_1.2.3-1_all.deb ... 2026-02-24T18:00:13.669 INFO:teuthology.orchestra.run.trial068.stdout:Unpacking python3-websocket (1.2.3-1) ... 2026-02-24T18:00:13.687 INFO:teuthology.orchestra.run.trial068.stdout:Selecting previously unselected package python3-kubernetes. 2026-02-24T18:00:13.697 INFO:teuthology.orchestra.run.trial068.stdout:Preparing to unpack .../61-python3-kubernetes_12.0.1-1ubuntu1_all.deb ... 2026-02-24T18:00:13.698 INFO:teuthology.orchestra.run.trial068.stdout:Unpacking python3-kubernetes (12.0.1-1ubuntu1) ... 2026-02-24T18:00:13.792 INFO:teuthology.orchestra.run.trial068.stdout:Selecting previously unselected package ceph-mgr-k8sevents. 2026-02-24T18:00:13.802 INFO:teuthology.orchestra.run.trial068.stdout:Preparing to unpack .../62-ceph-mgr-k8sevents_20.3.0-5507-g618c318a-1jammy_all.deb ... 2026-02-24T18:00:13.802 INFO:teuthology.orchestra.run.trial068.stdout:Unpacking ceph-mgr-k8sevents (20.3.0-5507-g618c318a-1jammy) ... 2026-02-24T18:00:13.818 INFO:teuthology.orchestra.run.trial068.stdout:Selecting previously unselected package libonig5:amd64. 2026-02-24T18:00:13.828 INFO:teuthology.orchestra.run.trial068.stdout:Preparing to unpack .../63-libonig5_6.9.7.1-2build1_amd64.deb ... 2026-02-24T18:00:13.828 INFO:teuthology.orchestra.run.trial068.stdout:Unpacking libonig5:amd64 (6.9.7.1-2build1) ... 2026-02-24T18:00:13.844 INFO:teuthology.orchestra.run.trial068.stdout:Selecting previously unselected package libjq1:amd64. 2026-02-24T18:00:13.855 INFO:teuthology.orchestra.run.trial068.stdout:Preparing to unpack .../64-libjq1_1.6-2.1ubuntu3.1_amd64.deb ... 2026-02-24T18:00:13.855 INFO:teuthology.orchestra.run.trial068.stdout:Unpacking libjq1:amd64 (1.6-2.1ubuntu3.1) ... 2026-02-24T18:00:13.870 INFO:teuthology.orchestra.run.trial068.stdout:Selecting previously unselected package jq. 2026-02-24T18:00:13.881 INFO:teuthology.orchestra.run.trial068.stdout:Preparing to unpack .../65-jq_1.6-2.1ubuntu3.1_amd64.deb ... 2026-02-24T18:00:13.881 INFO:teuthology.orchestra.run.trial068.stdout:Unpacking jq (1.6-2.1ubuntu3.1) ... 2026-02-24T18:00:13.896 INFO:teuthology.orchestra.run.trial068.stdout:Selecting previously unselected package socat. 2026-02-24T18:00:13.907 INFO:teuthology.orchestra.run.trial068.stdout:Preparing to unpack .../66-socat_1.7.4.1-3ubuntu4_amd64.deb ... 2026-02-24T18:00:13.907 INFO:teuthology.orchestra.run.trial068.stdout:Unpacking socat (1.7.4.1-3ubuntu4) ... 2026-02-24T18:00:13.930 INFO:teuthology.orchestra.run.trial068.stdout:Selecting previously unselected package xmlstarlet. 2026-02-24T18:00:13.940 INFO:teuthology.orchestra.run.trial068.stdout:Preparing to unpack .../67-xmlstarlet_1.6.1-2.1_amd64.deb ... 2026-02-24T18:00:13.941 INFO:teuthology.orchestra.run.trial068.stdout:Unpacking xmlstarlet (1.6.1-2.1) ... 2026-02-24T18:00:13.978 INFO:teuthology.orchestra.run.trial068.stdout:Selecting previously unselected package ceph-test. 2026-02-24T18:00:13.988 INFO:teuthology.orchestra.run.trial068.stdout:Preparing to unpack .../68-ceph-test_20.3.0-5507-g618c318a-1jammy_amd64.deb ... 2026-02-24T18:00:13.988 INFO:teuthology.orchestra.run.trial068.stdout:Unpacking ceph-test (20.3.0-5507-g618c318a-1jammy) ... 2026-02-24T18:00:14.636 INFO:teuthology.orchestra.run.trial068.stdout:Selecting previously unselected package ceph-volume. 2026-02-24T18:00:14.646 INFO:teuthology.orchestra.run.trial068.stdout:Preparing to unpack .../69-ceph-volume_20.3.0-5507-g618c318a-1jammy_all.deb ... 2026-02-24T18:00:14.647 INFO:teuthology.orchestra.run.trial068.stdout:Unpacking ceph-volume (20.3.0-5507-g618c318a-1jammy) ... 2026-02-24T18:00:14.670 INFO:teuthology.orchestra.run.trial068.stdout:Selecting previously unselected package libcephfs-daemon. 2026-02-24T18:00:14.680 INFO:teuthology.orchestra.run.trial068.stdout:Preparing to unpack .../70-libcephfs-daemon_20.3.0-5507-g618c318a-1jammy_amd64.deb ... 2026-02-24T18:00:14.681 INFO:teuthology.orchestra.run.trial068.stdout:Unpacking libcephfs-daemon (20.3.0-5507-g618c318a-1jammy) ... 2026-02-24T18:00:14.697 INFO:teuthology.orchestra.run.trial068.stdout:Selecting previously unselected package libcephfs-proxy2. 2026-02-24T18:00:14.707 INFO:teuthology.orchestra.run.trial068.stdout:Preparing to unpack .../71-libcephfs-proxy2_20.3.0-5507-g618c318a-1jammy_amd64.deb ... 2026-02-24T18:00:14.707 INFO:teuthology.orchestra.run.trial068.stdout:Unpacking libcephfs-proxy2 (20.3.0-5507-g618c318a-1jammy) ... 2026-02-24T18:00:14.722 INFO:teuthology.orchestra.run.trial068.stdout:Selecting previously unselected package libcephfs-dev. 2026-02-24T18:00:14.732 INFO:teuthology.orchestra.run.trial068.stdout:Preparing to unpack .../72-libcephfs-dev_20.3.0-5507-g618c318a-1jammy_amd64.deb ... 2026-02-24T18:00:14.733 INFO:teuthology.orchestra.run.trial068.stdout:Unpacking libcephfs-dev (20.3.0-5507-g618c318a-1jammy) ... 2026-02-24T18:00:14.749 INFO:teuthology.orchestra.run.trial068.stdout:Selecting previously unselected package nvme-cli. 2026-02-24T18:00:14.759 INFO:teuthology.orchestra.run.trial068.stdout:Preparing to unpack .../73-nvme-cli_1.16-3ubuntu0.3_amd64.deb ... 2026-02-24T18:00:14.760 INFO:teuthology.orchestra.run.trial068.stdout:Unpacking nvme-cli (1.16-3ubuntu0.3) ... 2026-02-24T18:00:14.790 INFO:teuthology.orchestra.run.trial068.stdout:Selecting previously unselected package python-asyncssh-doc. 2026-02-24T18:00:14.801 INFO:teuthology.orchestra.run.trial068.stdout:Preparing to unpack .../74-python-asyncssh-doc_2.5.0-1ubuntu0.1_all.deb ... 2026-02-24T18:00:14.801 INFO:teuthology.orchestra.run.trial068.stdout:Unpacking python-asyncssh-doc (2.5.0-1ubuntu0.1) ... 2026-02-24T18:00:14.831 INFO:teuthology.orchestra.run.trial068.stdout:Selecting previously unselected package python3-iniconfig. 2026-02-24T18:00:14.842 INFO:teuthology.orchestra.run.trial068.stdout:Preparing to unpack .../75-python3-iniconfig_1.1.1-2_all.deb ... 2026-02-24T18:00:14.842 INFO:teuthology.orchestra.run.trial068.stdout:Unpacking python3-iniconfig (1.1.1-2) ... 2026-02-24T18:00:14.858 INFO:teuthology.orchestra.run.trial068.stdout:Selecting previously unselected package python3-pluggy. 2026-02-24T18:00:14.868 INFO:teuthology.orchestra.run.trial068.stdout:Preparing to unpack .../76-python3-pluggy_0.13.0-7.1_all.deb ... 2026-02-24T18:00:14.869 INFO:teuthology.orchestra.run.trial068.stdout:Unpacking python3-pluggy (0.13.0-7.1) ... 2026-02-24T18:00:14.885 INFO:teuthology.orchestra.run.trial068.stdout:Selecting previously unselected package python3-psutil. 2026-02-24T18:00:14.896 INFO:teuthology.orchestra.run.trial068.stdout:Preparing to unpack .../77-python3-psutil_5.9.0-1build1_amd64.deb ... 2026-02-24T18:00:14.896 INFO:teuthology.orchestra.run.trial068.stdout:Unpacking python3-psutil (5.9.0-1build1) ... 2026-02-24T18:00:14.915 INFO:teuthology.orchestra.run.trial068.stdout:Selecting previously unselected package python3-py. 2026-02-24T18:00:14.925 INFO:teuthology.orchestra.run.trial068.stdout:Preparing to unpack .../78-python3-py_1.10.0-1_all.deb ... 2026-02-24T18:00:14.925 INFO:teuthology.orchestra.run.trial068.stdout:Unpacking python3-py (1.10.0-1) ... 2026-02-24T18:00:14.946 INFO:teuthology.orchestra.run.trial068.stdout:Selecting previously unselected package python3-pygments. 2026-02-24T18:00:14.956 INFO:teuthology.orchestra.run.trial068.stdout:Preparing to unpack .../79-python3-pygments_2.11.2+dfsg-2ubuntu0.1_all.deb ... 2026-02-24T18:00:14.956 INFO:teuthology.orchestra.run.trial068.stdout:Unpacking python3-pygments (2.11.2+dfsg-2ubuntu0.1) ... 2026-02-24T18:00:15.001 INFO:teuthology.orchestra.run.trial068.stdout:Selecting previously unselected package python3-toml. 2026-02-24T18:00:15.012 INFO:teuthology.orchestra.run.trial068.stdout:Preparing to unpack .../80-python3-toml_0.10.2-1_all.deb ... 2026-02-24T18:00:15.012 INFO:teuthology.orchestra.run.trial068.stdout:Unpacking python3-toml (0.10.2-1) ... 2026-02-24T18:00:15.027 INFO:teuthology.orchestra.run.trial068.stdout:Selecting previously unselected package python3-pytest. 2026-02-24T18:00:15.038 INFO:teuthology.orchestra.run.trial068.stdout:Preparing to unpack .../81-python3-pytest_6.2.5-1ubuntu2_all.deb ... 2026-02-24T18:00:15.038 INFO:teuthology.orchestra.run.trial068.stdout:Unpacking python3-pytest (6.2.5-1ubuntu2) ... 2026-02-24T18:00:15.060 INFO:teuthology.orchestra.run.trial068.stdout:Selecting previously unselected package python3-simplejson. 2026-02-24T18:00:15.071 INFO:teuthology.orchestra.run.trial068.stdout:Preparing to unpack .../82-python3-simplejson_3.17.6-1build1_amd64.deb ... 2026-02-24T18:00:15.071 INFO:teuthology.orchestra.run.trial068.stdout:Unpacking python3-simplejson (3.17.6-1build1) ... 2026-02-24T18:00:15.096 INFO:teuthology.orchestra.run.trial068.stdout:Selecting previously unselected package python3-webob. 2026-02-24T18:00:15.107 INFO:teuthology.orchestra.run.trial068.stdout:Preparing to unpack .../83-python3-webob_1%3a1.8.6-1.1ubuntu0.1_all.deb ... 2026-02-24T18:00:15.107 INFO:teuthology.orchestra.run.trial068.stdout:Unpacking python3-webob (1:1.8.6-1.1ubuntu0.1) ... 2026-02-24T18:00:15.124 INFO:teuthology.orchestra.run.trial068.stdout:Selecting previously unselected package qttranslations5-l10n. 2026-02-24T18:00:15.134 INFO:teuthology.orchestra.run.trial068.stdout:Preparing to unpack .../84-qttranslations5-l10n_5.15.3-1_all.deb ... 2026-02-24T18:00:15.135 INFO:teuthology.orchestra.run.trial068.stdout:Unpacking qttranslations5-l10n (5.15.3-1) ... 2026-02-24T18:00:15.203 INFO:teuthology.orchestra.run.trial068.stdout:Selecting previously unselected package radosgw. 2026-02-24T18:00:15.214 INFO:teuthology.orchestra.run.trial068.stdout:Preparing to unpack .../85-radosgw_20.3.0-5507-g618c318a-1jammy_amd64.deb ... 2026-02-24T18:00:15.214 INFO:teuthology.orchestra.run.trial068.stdout:Unpacking radosgw (20.3.0-5507-g618c318a-1jammy) ... 2026-02-24T18:00:15.371 INFO:teuthology.orchestra.run.trial068.stdout:Selecting previously unselected package rbd-fuse. 2026-02-24T18:00:15.382 INFO:teuthology.orchestra.run.trial068.stdout:Preparing to unpack .../86-rbd-fuse_20.3.0-5507-g618c318a-1jammy_amd64.deb ... 2026-02-24T18:00:15.382 INFO:teuthology.orchestra.run.trial068.stdout:Unpacking rbd-fuse (20.3.0-5507-g618c318a-1jammy) ... 2026-02-24T18:00:15.412 INFO:teuthology.orchestra.run.trial068.stdout:Setting up python3-iniconfig (1.1.1-2) ... 2026-02-24T18:00:15.477 INFO:teuthology.orchestra.run.trial068.stdout:Setting up libdouble-conversion3:amd64 (3.1.7-4) ... 2026-02-24T18:00:15.478 INFO:teuthology.orchestra.run.trial068.stdout:Setting up nvme-cli (1.16-3ubuntu0.3) ... 2026-02-24T18:00:15.551 INFO:teuthology.orchestra.run.trial068.stdout:Created symlink /etc/systemd/system/default.target.wants/nvmefc-boot-connections.service → /lib/systemd/system/nvmefc-boot-connections.service. 2026-02-24T18:00:15.838 INFO:teuthology.orchestra.run.trial068.stdout:Created symlink /etc/systemd/system/default.target.wants/nvmf-autoconnect.service → /lib/systemd/system/nvmf-autoconnect.service. 2026-02-24T18:00:16.232 INFO:teuthology.orchestra.run.trial068.stdout:nvmf-connect.target is a disabled or a static unit, not starting it. 2026-02-24T18:00:16.243 INFO:teuthology.orchestra.run.trial068.stdout:Setting up cephadm (20.3.0-5507-g618c318a-1jammy) ... 2026-02-24T18:00:16.288 INFO:teuthology.orchestra.run.trial068.stdout:Adding system user cephadm....done 2026-02-24T18:00:16.295 INFO:teuthology.orchestra.run.trial068.stdout:usermod: no changes 2026-02-24T18:00:16.301 INFO:teuthology.orchestra.run.trial068.stdout:Setting up python3-jaraco.classes (3.2.1-3) ... 2026-02-24T18:00:16.373 INFO:teuthology.orchestra.run.trial068.stdout:Setting up python-asyncssh-doc (2.5.0-1ubuntu0.1) ... 2026-02-24T18:00:16.374 INFO:teuthology.orchestra.run.trial068.stdout:Setting up python3-jaraco.functools (3.4.0-2) ... 2026-02-24T18:00:16.446 INFO:teuthology.orchestra.run.trial068.stdout:Setting up python3-repoze.lru (0.7-2) ... 2026-02-24T18:00:16.521 INFO:teuthology.orchestra.run.trial068.stdout:Setting up liboath0:amd64 (2.6.7-3ubuntu0.1) ... 2026-02-24T18:00:16.522 INFO:teuthology.orchestra.run.trial068.stdout:Setting up python3-py (1.10.0-1) ... 2026-02-24T18:00:16.616 INFO:teuthology.orchestra.run.trial068.stdout:Setting up python3-joblib (0.17.0-4ubuntu1) ... 2026-02-24T18:00:16.739 INFO:teuthology.orchestra.run.trial068.stdout:Setting up python3-cachetools (5.0.0-1) ... 2026-02-24T18:00:16.812 INFO:teuthology.orchestra.run.trial068.stdout:Setting up python3-threadpoolctl (3.1.0-1) ... 2026-02-24T18:00:16.884 INFO:teuthology.orchestra.run.trial068.stdout:Setting up python3-ceph-argparse (20.3.0-5507-g618c318a-1jammy) ... 2026-02-24T18:00:16.952 INFO:teuthology.orchestra.run.trial068.stdout:Setting up python3-sklearn-lib:amd64 (0.23.2-5ubuntu6) ... 2026-02-24T18:00:16.953 INFO:teuthology.orchestra.run.trial068.stdout:Setting up libnbd0 (1.10.5-1) ... 2026-02-24T18:00:16.954 INFO:teuthology.orchestra.run.trial068.stdout:Setting up libfuse2:amd64 (2.9.9-5ubuntu3) ... 2026-02-24T18:00:16.954 INFO:teuthology.orchestra.run.trial068.stdout:Setting up libpcre2-16-0:amd64 (10.39-3ubuntu0.1) ... 2026-02-24T18:00:16.955 INFO:teuthology.orchestra.run.trial068.stdout:Setting up python3-psutil (5.9.0-1build1) ... 2026-02-24T18:00:17.078 INFO:teuthology.orchestra.run.trial068.stdout:Setting up python3-natsort (8.0.2-1) ... 2026-02-24T18:00:17.156 INFO:teuthology.orchestra.run.trial068.stdout:Setting up libcephfs-proxy2 (20.3.0-5507-g618c318a-1jammy) ... 2026-02-24T18:00:17.156 INFO:teuthology.orchestra.run.trial068.stdout:Setting up python3-routes (2.5.1-1ubuntu1) ... 2026-02-24T18:00:17.231 INFO:teuthology.orchestra.run.trial068.stdout:Setting up python3-simplejson (3.17.6-1build1) ... 2026-02-24T18:00:17.317 INFO:teuthology.orchestra.run.trial068.stdout:Setting up python3-pygments (2.11.2+dfsg-2ubuntu0.1) ... 2026-02-24T18:00:17.565 INFO:teuthology.orchestra.run.trial068.stdout:Setting up qttranslations5-l10n (5.15.3-1) ... 2026-02-24T18:00:17.566 INFO:teuthology.orchestra.run.trial068.stdout:Setting up python3-wcwidth (0.2.5+dfsg1-1) ... 2026-02-24T18:00:17.661 INFO:teuthology.orchestra.run.trial068.stdout:Setting up python3-asyncssh (2.5.0-1ubuntu0.1) ... 2026-02-24T18:00:17.787 INFO:teuthology.orchestra.run.trial068.stdout:Setting up python3-cheroot (8.5.2+ds1-1ubuntu3.1) ... 2026-02-24T18:00:17.867 INFO:teuthology.orchestra.run.trial068.stdout:Setting up python3-jaraco.text (3.6.0-2) ... 2026-02-24T18:00:17.938 INFO:teuthology.orchestra.run.trial068.stdout:Setting up socat (1.7.4.1-3ubuntu4) ... 2026-02-24T18:00:17.939 INFO:teuthology.orchestra.run.trial068.stdout:Setting up python3-ceph-common (20.3.0-5507-g618c318a-1jammy) ... 2026-02-24T18:00:18.029 INFO:teuthology.orchestra.run.trial068.stdout:Setting up python3-sklearn (0.23.2-5ubuntu6) ... 2026-02-24T18:00:18.521 INFO:teuthology.orchestra.run.trial068.stdout:Setting up libqt5core5a:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2026-02-24T18:00:18.526 INFO:teuthology.orchestra.run.trial068.stdout:Setting up python3-toml (0.10.2-1) ... 2026-02-24T18:00:18.606 INFO:teuthology.orchestra.run.trial068.stdout:Setting up librdkafka1:amd64 (1.8.0-1build1) ... 2026-02-24T18:00:18.606 INFO:teuthology.orchestra.run.trial068.stdout:Setting up xmlstarlet (1.6.1-2.1) ... 2026-02-24T18:00:18.607 INFO:teuthology.orchestra.run.trial068.stdout:Setting up python3-pluggy (0.13.0-7.1) ... 2026-02-24T18:00:18.681 INFO:teuthology.orchestra.run.trial068.stdout:Setting up python3-zc.lockfile (2.0-1) ... 2026-02-24T18:00:18.754 INFO:teuthology.orchestra.run.trial068.stdout:Setting up libqt5dbus5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2026-02-24T18:00:18.755 INFO:teuthology.orchestra.run.trial068.stdout:Setting up python3-rsa (4.8-1) ... 2026-02-24T18:00:18.831 INFO:teuthology.orchestra.run.trial068.stdout:Setting up python3-tempora (4.1.2-1) ... 2026-02-24T18:00:18.903 INFO:teuthology.orchestra.run.trial068.stdout:Setting up python3-prettytable (2.5.0-2) ... 2026-02-24T18:00:18.980 INFO:teuthology.orchestra.run.trial068.stdout:Setting up liblttng-ust1:amd64 (2.13.1-1ubuntu1) ... 2026-02-24T18:00:18.981 INFO:teuthology.orchestra.run.trial068.stdout:Setting up python3-websocket (1.2.3-1) ... 2026-02-24T18:00:19.064 INFO:teuthology.orchestra.run.trial068.stdout:Setting up libonig5:amd64 (6.9.7.1-2build1) ... 2026-02-24T18:00:19.065 INFO:teuthology.orchestra.run.trial068.stdout:Setting up python3-requests-oauthlib (1.3.0+ds-0.1) ... 2026-02-24T18:00:19.142 INFO:teuthology.orchestra.run.trial068.stdout:Setting up python3-webob (1:1.8.6-1.1ubuntu0.1) ... 2026-02-24T18:00:19.226 INFO:teuthology.orchestra.run.trial068.stdout:Setting up python3-jaraco.collections (3.4.0-2) ... 2026-02-24T18:00:19.302 INFO:teuthology.orchestra.run.trial068.stdout:Setting up libjq1:amd64 (1.6-2.1ubuntu3.1) ... 2026-02-24T18:00:19.303 INFO:teuthology.orchestra.run.trial068.stdout:Setting up python3-pytest (6.2.5-1ubuntu2) ... 2026-02-24T18:00:19.435 INFO:teuthology.orchestra.run.trial068.stdout:Setting up python3-portend (3.0.0-1) ... 2026-02-24T18:00:19.506 INFO:teuthology.orchestra.run.trial068.stdout:Setting up libqt5network5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2026-02-24T18:00:19.507 INFO:teuthology.orchestra.run.trial068.stdout:Setting up python3-google-auth (1.5.1-3) ... 2026-02-24T18:00:19.588 INFO:teuthology.orchestra.run.trial068.stdout:Setting up jq (1.6-2.1ubuntu3.1) ... 2026-02-24T18:00:19.589 INFO:teuthology.orchestra.run.trial068.stdout:Setting up python3-cherrypy3 (18.6.1-4) ... 2026-02-24T18:00:19.719 INFO:teuthology.orchestra.run.trial068.stdout:Setting up libthrift-0.16.0:amd64 (0.16.0-2) ... 2026-02-24T18:00:19.720 INFO:teuthology.orchestra.run.trial068.stdout:Setting up librados2 (20.3.0-5507-g618c318a-1jammy) ... 2026-02-24T18:00:19.721 INFO:teuthology.orchestra.run.trial068.stdout:Setting up librgw2 (20.3.0-5507-g618c318a-1jammy) ... 2026-02-24T18:00:19.722 INFO:teuthology.orchestra.run.trial068.stdout:Setting up libsqlite3-mod-ceph (20.3.0-5507-g618c318a-1jammy) ... 2026-02-24T18:00:19.723 INFO:teuthology.orchestra.run.trial068.stdout:Setting up python3-kubernetes (12.0.1-1ubuntu1) ... 2026-02-24T18:00:20.234 INFO:teuthology.orchestra.run.trial068.stdout:Setting up libcephfs2 (20.3.0-5507-g618c318a-1jammy) ... 2026-02-24T18:00:20.235 INFO:teuthology.orchestra.run.trial068.stdout:Setting up libradosstriper1 (20.3.0-5507-g618c318a-1jammy) ... 2026-02-24T18:00:20.236 INFO:teuthology.orchestra.run.trial068.stdout:Setting up librbd1 (20.3.0-5507-g618c318a-1jammy) ... 2026-02-24T18:00:20.237 INFO:teuthology.orchestra.run.trial068.stdout:Setting up ceph-mgr-modules-core (20.3.0-5507-g618c318a-1jammy) ... 2026-02-24T18:00:20.238 INFO:teuthology.orchestra.run.trial068.stdout:Setting up ceph-fuse (20.3.0-5507-g618c318a-1jammy) ... 2026-02-24T18:00:20.311 INFO:teuthology.orchestra.run.trial068.stdout:Created symlink /etc/systemd/system/remote-fs.target.wants/ceph-fuse.target → /lib/systemd/system/ceph-fuse.target. 2026-02-24T18:00:20.312 INFO:teuthology.orchestra.run.trial068.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-fuse.target → /lib/systemd/system/ceph-fuse.target. 2026-02-24T18:00:20.720 INFO:teuthology.orchestra.run.trial068.stdout:Setting up libcephfs-dev (20.3.0-5507-g618c318a-1jammy) ... 2026-02-24T18:00:20.721 INFO:teuthology.orchestra.run.trial068.stdout:Setting up python3-rados (20.3.0-5507-g618c318a-1jammy) ... 2026-02-24T18:00:20.722 INFO:teuthology.orchestra.run.trial068.stdout:Setting up libcephfs-daemon (20.3.0-5507-g618c318a-1jammy) ... 2026-02-24T18:00:20.723 INFO:teuthology.orchestra.run.trial068.stdout:Setting up python3-rbd (20.3.0-5507-g618c318a-1jammy) ... 2026-02-24T18:00:20.723 INFO:teuthology.orchestra.run.trial068.stdout:Setting up rbd-fuse (20.3.0-5507-g618c318a-1jammy) ... 2026-02-24T18:00:20.724 INFO:teuthology.orchestra.run.trial068.stdout:Setting up python3-rgw (20.3.0-5507-g618c318a-1jammy) ... 2026-02-24T18:00:20.725 INFO:teuthology.orchestra.run.trial068.stdout:Setting up python3-cephfs (20.3.0-5507-g618c318a-1jammy) ... 2026-02-24T18:00:20.726 INFO:teuthology.orchestra.run.trial068.stdout:Setting up ceph-common (20.3.0-5507-g618c318a-1jammy) ... 2026-02-24T18:00:20.753 INFO:teuthology.orchestra.run.trial068.stdout:Adding group ceph....done 2026-02-24T18:00:20.789 INFO:teuthology.orchestra.run.trial068.stdout:Adding system user ceph....done 2026-02-24T18:00:20.798 INFO:teuthology.orchestra.run.trial068.stdout:Setting system user ceph properties....done 2026-02-24T18:00:20.805 INFO:teuthology.orchestra.run.trial068.stdout:chown: cannot access '/var/log/ceph/*.log*': No such file or directory 2026-02-24T18:00:20.892 INFO:teuthology.orchestra.run.trial068.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /lib/systemd/system/ceph.target. 2026-02-24T18:00:21.173 INFO:teuthology.orchestra.run.trial068.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/rbdmap.service → /lib/systemd/system/rbdmap.service. 2026-02-24T18:00:21.553 INFO:teuthology.orchestra.run.trial068.stdout:Setting up ceph-test (20.3.0-5507-g618c318a-1jammy) ... 2026-02-24T18:00:21.554 INFO:teuthology.orchestra.run.trial068.stdout:Setting up radosgw (20.3.0-5507-g618c318a-1jammy) ... 2026-02-24T18:00:21.814 INFO:teuthology.orchestra.run.trial068.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /lib/systemd/system/ceph-radosgw.target. 2026-02-24T18:00:21.814 INFO:teuthology.orchestra.run.trial068.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /lib/systemd/system/ceph-radosgw.target. 2026-02-24T18:00:22.204 INFO:teuthology.orchestra.run.trial068.stdout:Setting up ceph-base (20.3.0-5507-g618c318a-1jammy) ... 2026-02-24T18:00:22.318 INFO:teuthology.orchestra.run.trial068.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /lib/systemd/system/ceph-crash.service. 2026-02-24T18:00:22.685 INFO:teuthology.orchestra.run.trial068.stdout:Setting up ceph-mds (20.3.0-5507-g618c318a-1jammy) ... 2026-02-24T18:00:22.745 INFO:teuthology.orchestra.run.trial068.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /lib/systemd/system/ceph-mds.target. 2026-02-24T18:00:22.745 INFO:teuthology.orchestra.run.trial068.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /lib/systemd/system/ceph-mds.target. 2026-02-24T18:00:23.135 INFO:teuthology.orchestra.run.trial068.stdout:Setting up ceph-mgr (20.3.0-5507-g618c318a-1jammy) ... 2026-02-24T18:00:23.212 INFO:teuthology.orchestra.run.trial068.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /lib/systemd/system/ceph-mgr.target. 2026-02-24T18:00:23.212 INFO:teuthology.orchestra.run.trial068.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /lib/systemd/system/ceph-mgr.target. 2026-02-24T18:00:23.603 INFO:teuthology.orchestra.run.trial068.stdout:Setting up ceph-mgr-k8sevents (20.3.0-5507-g618c318a-1jammy) ... 2026-02-24T18:00:23.604 INFO:teuthology.orchestra.run.trial068.stdout:Setting up ceph-osd-classic (20.3.0-5507-g618c318a-1jammy) ... 2026-02-24T18:00:23.622 INFO:teuthology.orchestra.run.trial068.stdout:update-alternatives: using /usr/bin/ceph-osd-classic to provide /usr/bin/ceph-osd (ceph-osd) in auto mode 2026-02-24T18:00:23.631 INFO:teuthology.orchestra.run.trial068.stdout:Setting up ceph-mgr-diskprediction-local (20.3.0-5507-g618c318a-1jammy) ... 2026-02-24T18:00:23.642 INFO:teuthology.orchestra.run.trial068.stdout:Setting up ceph-mon (20.3.0-5507-g618c318a-1jammy) ... 2026-02-24T18:00:23.714 INFO:teuthology.orchestra.run.trial068.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /lib/systemd/system/ceph-mon.target. 2026-02-24T18:00:23.714 INFO:teuthology.orchestra.run.trial068.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /lib/systemd/system/ceph-mon.target. 2026-02-24T18:00:24.095 INFO:teuthology.orchestra.run.trial068.stdout:Setting up ceph-mgr-cephadm (20.3.0-5507-g618c318a-1jammy) ... 2026-02-24T18:00:24.105 INFO:teuthology.orchestra.run.trial068.stdout:Setting up ceph-mgr-dashboard (20.3.0-5507-g618c318a-1jammy) ... 2026-02-24T18:00:24.116 INFO:teuthology.orchestra.run.trial068.stdout:Setting up ceph-osd (20.3.0-5507-g618c318a-1jammy) ... 2026-02-24T18:00:24.199 INFO:teuthology.orchestra.run.trial068.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /lib/systemd/system/ceph-osd.target. 2026-02-24T18:00:24.199 INFO:teuthology.orchestra.run.trial068.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /lib/systemd/system/ceph-osd.target. 2026-02-24T18:00:24.604 INFO:teuthology.orchestra.run.trial068.stdout:Setting up ceph (20.3.0-5507-g618c318a-1jammy) ... 2026-02-24T18:00:24.605 INFO:teuthology.orchestra.run.trial068.stdout:Setting up ceph-volume (20.3.0-5507-g618c318a-1jammy) ... 2026-02-24T18:00:24.714 INFO:teuthology.orchestra.run.trial068.stdout:Processing triggers for man-db (2.10.2-1) ... 2026-02-24T18:00:24.799 INFO:teuthology.orchestra.run.trial068.stdout:Processing triggers for libc-bin (2.35-0ubuntu3.13) ... 2026-02-24T18:00:25.489 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-24T18:00:25.489 INFO:teuthology.orchestra.run.trial068.stdout:Running kernel seems to be up-to-date. 2026-02-24T18:00:25.489 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-24T18:00:25.489 INFO:teuthology.orchestra.run.trial068.stdout:The processor microcode seems to be up-to-date. 2026-02-24T18:00:25.489 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-24T18:00:25.489 INFO:teuthology.orchestra.run.trial068.stdout:No services need to be restarted. 2026-02-24T18:00:25.489 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-24T18:00:25.489 INFO:teuthology.orchestra.run.trial068.stdout:No containers need to be restarted. 2026-02-24T18:00:25.489 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-24T18:00:25.489 INFO:teuthology.orchestra.run.trial068.stdout:No user sessions are running outdated binaries. 2026-02-24T18:00:25.489 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-24T18:00:25.489 INFO:teuthology.orchestra.run.trial068.stdout:No VM guests are running outdated hypervisor (qemu) binaries on this host. 2026-02-24T18:00:26.287 INFO:teuthology.orchestra.run.trial068.stderr:W: --force-yes is deprecated, use one of the options starting with --allow instead. 2026-02-24T18:00:26.294 DEBUG:teuthology.parallel:result is None 2026-02-24T18:00:26.295 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&sha1=618c318a7a0b287ab5aeff207b7bf1d73ef45044 2026-02-24T18:00:26.470 DEBUG:teuthology.orchestra.run.trial068:> dpkg-query -W -f '${Version}' ceph 2026-02-24T18:00:26.479 INFO:teuthology.orchestra.run.trial068.stdout:20.3.0-5507-g618c318a-1jammy 2026-02-24T18:00:26.480 INFO:teuthology.packaging:The installed version of ceph is 20.3.0-5507-g618c318a-1jammy 2026-02-24T18:00:26.480 INFO:teuthology.task.install:The correct ceph version 20.3.0-5507-g618c318a-1jammy is installed. 2026-02-24T18:00:26.482 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2026-02-24T18:00:26.482 DEBUG:teuthology.orchestra.run.trial068:> set -ex 2026-02-24T18:00:26.482 DEBUG:teuthology.orchestra.run.trial068:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2026-02-24T18:00:26.530 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2026-02-24T18:00:26.530 DEBUG:teuthology.orchestra.run.trial068:> set -ex 2026-02-24T18:00:26.530 DEBUG:teuthology.orchestra.run.trial068:> sudo dd of=/usr/bin/daemon-helper 2026-02-24T18:00:26.582 DEBUG:teuthology.orchestra.run.trial068:> sudo chmod a=rx -- /usr/bin/daemon-helper 2026-02-24T18:00:26.633 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2026-02-24T18:00:26.633 DEBUG:teuthology.orchestra.run.trial068:> set -ex 2026-02-24T18:00:26.634 DEBUG:teuthology.orchestra.run.trial068:> sudo dd of=/usr/bin/adjust-ulimits 2026-02-24T18:00:26.686 DEBUG:teuthology.orchestra.run.trial068:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2026-02-24T18:00:26.737 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2026-02-24T18:00:26.737 DEBUG:teuthology.orchestra.run.trial068:> set -ex 2026-02-24T18:00:26.737 DEBUG:teuthology.orchestra.run.trial068:> sudo dd of=/usr/bin/stdin-killer 2026-02-24T18:00:26.785 DEBUG:teuthology.orchestra.run.trial068:> sudo chmod a=rx -- /usr/bin/stdin-killer 2026-02-24T18:00:26.836 INFO:teuthology.run_tasks:Running task ceph... 2026-02-24T18:00:26.937 INFO:tasks.ceph:Making ceph log dir writeable by non-root... 2026-02-24T18:00:26.937 DEBUG:teuthology.orchestra.run.trial068:> sudo chmod 777 /var/log/ceph 2026-02-24T18:00:26.944 INFO:tasks.ceph:Disabling ceph logrotate... 2026-02-24T18:00:26.944 DEBUG:teuthology.orchestra.run.trial068:> sudo rm -f -- /etc/logrotate.d/ceph 2026-02-24T18:00:26.992 INFO:tasks.ceph:Creating extra log directories... 2026-02-24T18:00:26.992 DEBUG:teuthology.orchestra.run.trial068:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2026-02-24T18:00:27.048 INFO:tasks.ceph:Creating ceph cluster ceph... 2026-02-24T18:00:27.048 INFO:tasks.ceph:config {'conf': {'osd': {'osd max object name len': 400, 'osd max object namespace len': 64, 'bluestore zero block detection': True, 'debug ms': 1, 'debug osd': 20, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd mclock override recovery settings': True, 'osd mclock profile': 'high_recovery_ops', 'osd mclock skip benchmark': True, 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random'}, 'global': {'mon election default strategy': 3}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 30, 'debug ms': 1, 'debug paxos': 20, 'mon scrub interval': 300}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['but it is still running', 'overall HEALTH_', '\\(OSDMAP_FLAGS\\)', '\\(OSD_', '\\(PG_', '\\(POOL_APP_NOT_ENABLED\\)', '\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)'], 'cpu_profile': set(), 'cluster': 'ceph', 'mon_bind_msgr2': True, 'mon_bind_addrvec': True} 2026-02-24T18:00:27.048 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/mnt/teuthology/yuriw-2026-02-24_16:03:54-rados-wip-yuri8-testing-2026-02-23-2128-distro-default-trial/68571', 'branch': 'wip-yuri8-testing-2026-02-23-2128', 'description': 'rados/singleton-nomsgr/{all/health-warnings mon_election/connectivity rados supported-random-distro$/{ubuntu_latest}}', 'email': 'yweinste@redhat.com', 'first_in_suite': False, 'flavor': 'default', 'job_id': '68571', 'kernel': {'branch': 'distro', 'kdb': True, 'sha1': 'distro'}, 'ktype': 'distro', 'last_in_suite': False, 'machine_type': 'trial', 'name': 'yuriw-2026-02-24_16:03:54-rados-wip-yuri8-testing-2026-02-23-2128-distro-default-trial', 'no_nested_subset': False, 'os_type': 'ubuntu', 'os_version': '22.04', 'overrides': {'admin_socket': {'branch': 'wip-yuri8-testing-2026-02-23-2128'}, 'ceph': {'conf': {'global': {'mon election default strategy': 3}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 30, 'debug ms': 1, 'debug paxos': 20, 'mon scrub interval': 300}, 'osd': {'bluestore zero block detection': True, 'debug ms': 1, 'debug osd': 20, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd mclock override recovery settings': True, 'osd mclock profile': 'high_recovery_ops', 'osd mclock skip benchmark': True, 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random'}}, 'flavor': 'default', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)'], 'sha1': '618c318a7a0b287ab5aeff207b7bf1d73ef45044'}, 'ceph-deploy': {'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {}}}, 'install': {'ceph': {'flavor': 'default', 'sha1': '618c318a7a0b287ab5aeff207b7bf1d73ef45044'}}, 'workunit': {'branch': 'wip-yuri8-testing-2026-02-23-2128', 'sha1': '618c318a7a0b287ab5aeff207b7bf1d73ef45044'}}, 'owner': 'scheduled_yuriw@soko04.front.sepia.ceph.com', 'priority': 69, 'repo': 'https://github.com/ceph/ceph-ci.git', 'roles': [['mon.a', 'mgr.x', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'osd.8', 'osd.9', 'client.0']], 'seed': 6922, 'sha1': '618c318a7a0b287ab5aeff207b7bf1d73ef45044', 'sleep_before_teardown': 0, 'subset': '111/120000', 'suite': 'rados', 'suite_branch': 'wip-yuri8-testing-2026-02-23-2128', 'suite_path': '/home/teuthworker/src/github.com_ceph_ceph-c_618c318a7a0b287ab5aeff207b7bf1d73ef45044/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://github.com/ceph/ceph-ci.git', 'suite_sha1': '618c318a7a0b287ab5aeff207b7bf1d73ef45044', 'targets': {'trial068.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBJET0we95Ttlwr4RnDY93t7VgPcOzA7uWELNz2LQ0M2zzXpkPpEjMFiROkvBrnMVkgRB6NOSCaSPDtjriH8+gbE='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'branch': 'distro', 'kdb': True, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': None}, {'ceph': {'conf': {'osd': {'osd max object name len': 400, 'osd max object namespace len': 64, 'bluestore zero block detection': True, 'debug ms': 1, 'debug osd': 20, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd mclock override recovery settings': True, 'osd mclock profile': 'high_recovery_ops', 'osd mclock skip benchmark': True, 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random'}, 'global': {'mon election default strategy': 3}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 30, 'debug ms': 1, 'debug paxos': 20, 'mon scrub interval': 300}}, 'log-ignorelist': ['but it is still running', 'overall HEALTH_', '\\(OSDMAP_FLAGS\\)', '\\(OSD_', '\\(PG_', '\\(POOL_APP_NOT_ENABLED\\)', '\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)'], 'pre-mgr-commands': ['sudo ceph config set mgr mgr_pool false --force'], 'flavor': 'default', 'sha1': '618c318a7a0b287ab5aeff207b7bf1d73ef45044', 'cluster': 'ceph'}}, {'workunit': {'clients': {'all': ['rados/test_health_warnings.sh']}}}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': []}, 'teuthology_branch': 'main', 'teuthology_sha1': 'fa17720d0088c3ac28e473468bfc79eeaff5cd38', 'timestamp': '2026-02-24_16:03:54', 'tube': 'trial', 'user': 'yuriw', 'verbose': True, 'worker_log': '/home/teuthworker/mnt/teuthology/worker_logs/dispatcher.trial.904699'} 2026-02-24T18:00:27.048 DEBUG:teuthology.orchestra.run.trial068:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2026-02-24T18:00:27.085 DEBUG:teuthology.orchestra.run.trial068:> sudo install -d -m0777 -- /var/run/ceph 2026-02-24T18:00:27.137 DEBUG:teuthology.orchestra.run.trial068:> set -ex 2026-02-24T18:00:27.138 DEBUG:teuthology.orchestra.run.trial068:> dd if=/scratch_devs of=/dev/stdout 2026-02-24T18:00:27.186 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2026-02-24T18:00:27.186 DEBUG:teuthology.orchestra.run.trial068:> stat /dev/vg_nvme/lv_1 2026-02-24T18:00:27.234 INFO:teuthology.orchestra.run.trial068.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2026-02-24T18:00:27.234 INFO:teuthology.orchestra.run.trial068.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2026-02-24T18:00:27.234 INFO:teuthology.orchestra.run.trial068.stdout:Device: 5h/5d Inode: 904 Links: 1 2026-02-24T18:00:27.234 INFO:teuthology.orchestra.run.trial068.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2026-02-24T18:00:27.234 INFO:teuthology.orchestra.run.trial068.stdout:Access: 2026-02-24 17:58:01.942642454 +0000 2026-02-24T18:00:27.234 INFO:teuthology.orchestra.run.trial068.stdout:Modify: 2026-02-24 17:58:01.794644972 +0000 2026-02-24T18:00:27.234 INFO:teuthology.orchestra.run.trial068.stdout:Change: 2026-02-24 17:58:01.794644972 +0000 2026-02-24T18:00:27.234 INFO:teuthology.orchestra.run.trial068.stdout: Birth: - 2026-02-24T18:00:27.234 DEBUG:teuthology.orchestra.run.trial068:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2026-02-24T18:00:27.283 INFO:teuthology.orchestra.run.trial068.stderr:1+0 records in 2026-02-24T18:00:27.284 INFO:teuthology.orchestra.run.trial068.stderr:1+0 records out 2026-02-24T18:00:27.284 INFO:teuthology.orchestra.run.trial068.stderr:512 bytes copied, 0.000183842 s, 2.8 MB/s 2026-02-24T18:00:27.284 DEBUG:teuthology.orchestra.run.trial068:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2026-02-24T18:00:27.330 DEBUG:teuthology.orchestra.run.trial068:> stat /dev/vg_nvme/lv_2 2026-02-24T18:00:27.393 INFO:teuthology.orchestra.run.trial068.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2026-02-24T18:00:27.394 INFO:teuthology.orchestra.run.trial068.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2026-02-24T18:00:27.394 INFO:teuthology.orchestra.run.trial068.stdout:Device: 5h/5d Inode: 928 Links: 1 2026-02-24T18:00:27.394 INFO:teuthology.orchestra.run.trial068.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2026-02-24T18:00:27.394 INFO:teuthology.orchestra.run.trial068.stdout:Access: 2026-02-24 17:58:02.278636742 +0000 2026-02-24T18:00:27.394 INFO:teuthology.orchestra.run.trial068.stdout:Modify: 2026-02-24 17:58:02.098639804 +0000 2026-02-24T18:00:27.395 INFO:teuthology.orchestra.run.trial068.stdout:Change: 2026-02-24 17:58:02.098639804 +0000 2026-02-24T18:00:27.395 INFO:teuthology.orchestra.run.trial068.stdout: Birth: - 2026-02-24T18:00:27.396 DEBUG:teuthology.orchestra.run.trial068:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2026-02-24T18:00:27.444 INFO:teuthology.orchestra.run.trial068.stderr:1+0 records in 2026-02-24T18:00:27.444 INFO:teuthology.orchestra.run.trial068.stderr:1+0 records out 2026-02-24T18:00:27.444 INFO:teuthology.orchestra.run.trial068.stderr:512 bytes copied, 0.00016655 s, 3.1 MB/s 2026-02-24T18:00:27.445 DEBUG:teuthology.orchestra.run.trial068:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2026-02-24T18:00:27.495 DEBUG:teuthology.orchestra.run.trial068:> stat /dev/vg_nvme/lv_3 2026-02-24T18:00:27.541 INFO:teuthology.orchestra.run.trial068.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2026-02-24T18:00:27.541 INFO:teuthology.orchestra.run.trial068.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2026-02-24T18:00:27.542 INFO:teuthology.orchestra.run.trial068.stdout:Device: 5h/5d Inode: 953 Links: 1 2026-02-24T18:00:27.542 INFO:teuthology.orchestra.run.trial068.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2026-02-24T18:00:27.542 INFO:teuthology.orchestra.run.trial068.stdout:Access: 2026-02-24 17:58:02.610631099 +0000 2026-02-24T18:00:27.542 INFO:teuthology.orchestra.run.trial068.stdout:Modify: 2026-02-24 17:58:02.446633888 +0000 2026-02-24T18:00:27.542 INFO:teuthology.orchestra.run.trial068.stdout:Change: 2026-02-24 17:58:02.446633888 +0000 2026-02-24T18:00:27.542 INFO:teuthology.orchestra.run.trial068.stdout: Birth: - 2026-02-24T18:00:27.542 DEBUG:teuthology.orchestra.run.trial068:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2026-02-24T18:00:27.592 INFO:teuthology.orchestra.run.trial068.stderr:1+0 records in 2026-02-24T18:00:27.593 INFO:teuthology.orchestra.run.trial068.stderr:1+0 records out 2026-02-24T18:00:27.593 INFO:teuthology.orchestra.run.trial068.stderr:512 bytes copied, 0.000176888 s, 2.9 MB/s 2026-02-24T18:00:27.594 DEBUG:teuthology.orchestra.run.trial068:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2026-02-24T18:00:27.643 DEBUG:teuthology.orchestra.run.trial068:> stat /dev/vg_nvme/lv_4 2026-02-24T18:00:27.689 INFO:teuthology.orchestra.run.trial068.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2026-02-24T18:00:27.690 INFO:teuthology.orchestra.run.trial068.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2026-02-24T18:00:27.690 INFO:teuthology.orchestra.run.trial068.stdout:Device: 5h/5d Inode: 979 Links: 1 2026-02-24T18:00:27.690 INFO:teuthology.orchestra.run.trial068.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2026-02-24T18:00:27.690 INFO:teuthology.orchestra.run.trial068.stdout:Access: 2026-02-24 17:58:02.934625591 +0000 2026-02-24T18:00:27.690 INFO:teuthology.orchestra.run.trial068.stdout:Modify: 2026-02-24 17:58:02.778628242 +0000 2026-02-24T18:00:27.690 INFO:teuthology.orchestra.run.trial068.stdout:Change: 2026-02-24 17:58:02.778628242 +0000 2026-02-24T18:00:27.690 INFO:teuthology.orchestra.run.trial068.stdout: Birth: - 2026-02-24T18:00:27.691 DEBUG:teuthology.orchestra.run.trial068:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2026-02-24T18:00:27.740 INFO:teuthology.orchestra.run.trial068.stderr:1+0 records in 2026-02-24T18:00:27.740 INFO:teuthology.orchestra.run.trial068.stderr:1+0 records out 2026-02-24T18:00:27.741 INFO:teuthology.orchestra.run.trial068.stderr:512 bytes copied, 0.000173452 s, 3.0 MB/s 2026-02-24T18:00:27.742 DEBUG:teuthology.orchestra.run.trial068:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2026-02-24T18:00:27.791 INFO:tasks.ceph:osd dev map: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2026-02-24T18:00:27.791 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@trial068.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'}} 2026-02-24T18:00:27.792 INFO:tasks.ceph:Generating config... 2026-02-24T18:00:27.794 INFO:tasks.ceph:[osd] osd max object name len = 400 2026-02-24T18:00:27.795 INFO:tasks.ceph:[osd] osd max object namespace len = 64 2026-02-24T18:00:27.795 INFO:tasks.ceph:[osd] bluestore zero block detection = True 2026-02-24T18:00:27.795 INFO:tasks.ceph:[osd] debug ms = 1 2026-02-24T18:00:27.795 INFO:tasks.ceph:[osd] debug osd = 20 2026-02-24T18:00:27.795 INFO:tasks.ceph:[osd] osd debug verify cached snaps = True 2026-02-24T18:00:27.795 INFO:tasks.ceph:[osd] osd debug verify missing on start = True 2026-02-24T18:00:27.795 INFO:tasks.ceph:[osd] osd mclock override recovery settings = True 2026-02-24T18:00:27.795 INFO:tasks.ceph:[osd] osd mclock profile = high_recovery_ops 2026-02-24T18:00:27.795 INFO:tasks.ceph:[osd] osd mclock skip benchmark = True 2026-02-24T18:00:27.796 INFO:tasks.ceph:[osd] osd op queue = debug_random 2026-02-24T18:00:27.796 INFO:tasks.ceph:[osd] osd op queue cut off = debug_random 2026-02-24T18:00:27.796 INFO:tasks.ceph:[global] mon election default strategy = 3 2026-02-24T18:00:27.796 INFO:tasks.ceph:[mgr] debug mgr = 20 2026-02-24T18:00:27.796 INFO:tasks.ceph:[mgr] debug ms = 1 2026-02-24T18:00:27.796 INFO:tasks.ceph:[mon] debug mon = 30 2026-02-24T18:00:27.796 INFO:tasks.ceph:[mon] debug ms = 1 2026-02-24T18:00:27.796 INFO:tasks.ceph:[mon] debug paxos = 20 2026-02-24T18:00:27.796 INFO:tasks.ceph:[mon] mon scrub interval = 300 2026-02-24T18:00:27.797 INFO:tasks.ceph:Setting up mon.a... 2026-02-24T18:00:27.797 DEBUG:teuthology.orchestra.run.trial068:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/ceph.keyring 2026-02-24T18:00:27.849 INFO:teuthology.orchestra.run.trial068.stdout:creating /etc/ceph/ceph.keyring 2026-02-24T18:00:27.852 DEBUG:teuthology.orchestra.run.trial068:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/ceph.keyring 2026-02-24T18:00:27.912 DEBUG:teuthology.orchestra.run.trial068:> sudo chmod 0644 /etc/ceph/ceph.keyring 2026-02-24T18:00:27.961 DEBUG:tasks.ceph:Ceph mon addresses: [('mon.a', '10.20.193.68')] 2026-02-24T18:00:27.962 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'cephx', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=isa technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '10.20.193.68', 'mon election default strategy': 3}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd mclock profile': 'high_recovery_ops', 'osd mclock skip benchmark': True, 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd debug trim objects': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'osd max object name len': 400, 'osd max object namespace len': 64, 'bluestore zero block detection': True, 'debug ms': 1, 'debug osd': 20, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd mclock override recovery settings': True, 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random'}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false'}, 'mon': {'debug ms': 1, 'debug mon': 30, 'debug paxos': 20, 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'mon_down_mkfs_grace': '2m', 'mon_warn_on_filestore_osds': 'false', 'mon scrub interval': 300}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok'}, 'mon.a': {}} 2026-02-24T18:00:27.963 DEBUG:teuthology.orchestra.run.trial068:> set -ex 2026-02-24T18:00:27.963 DEBUG:teuthology.orchestra.run.trial068:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2026-02-24T18:00:28.006 DEBUG:teuthology.orchestra.run.trial068:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --add a 10.20.193.68 --print /home/ubuntu/cephtest/ceph.monmap 2026-02-24T18:00:28.066 INFO:teuthology.orchestra.run.trial068.stdout:monmaptool: monmap file /home/ubuntu/cephtest/ceph.monmap 2026-02-24T18:00:28.066 INFO:teuthology.orchestra.run.trial068.stdout:monmaptool: generated fsid 85c6d86c-aad0-4a30-9c50-f24175e8fcf5 2026-02-24T18:00:28.066 INFO:teuthology.orchestra.run.trial068.stdout:setting min_mon_release = tentacle 2026-02-24T18:00:28.067 INFO:teuthology.orchestra.run.trial068.stdout:epoch 0 2026-02-24T18:00:28.067 INFO:teuthology.orchestra.run.trial068.stdout:fsid 85c6d86c-aad0-4a30-9c50-f24175e8fcf5 2026-02-24T18:00:28.067 INFO:teuthology.orchestra.run.trial068.stdout:last_changed 2026-02-24T18:00:28.063188+0000 2026-02-24T18:00:28.067 INFO:teuthology.orchestra.run.trial068.stdout:created 2026-02-24T18:00:28.063188+0000 2026-02-24T18:00:28.067 INFO:teuthology.orchestra.run.trial068.stdout:min_mon_release 20 (tentacle) 2026-02-24T18:00:28.067 INFO:teuthology.orchestra.run.trial068.stdout:election_strategy: 3 2026-02-24T18:00:28.067 INFO:teuthology.orchestra.run.trial068.stdout:0: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.a 2026-02-24T18:00:28.067 INFO:teuthology.orchestra.run.trial068.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/ceph.monmap (1 monitors) 2026-02-24T18:00:28.070 DEBUG:teuthology.orchestra.run.trial068:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2026-02-24T18:00:28.112 INFO:tasks.ceph:Writing /etc/ceph/ceph.conf for FSID 85c6d86c-aad0-4a30-9c50-f24175e8fcf5... 2026-02-24T18:00:28.113 DEBUG:teuthology.orchestra.run.trial068:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2026-02-24T18:00:28.172 INFO:teuthology.orchestra.run.trial068.stdout:[global] 2026-02-24T18:00:28.172 INFO:teuthology.orchestra.run.trial068.stdout: chdir = "" 2026-02-24T18:00:28.172 INFO:teuthology.orchestra.run.trial068.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2026-02-24T18:00:28.172 INFO:teuthology.orchestra.run.trial068.stdout: auth supported = cephx 2026-02-24T18:00:28.172 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-24T18:00:28.172 INFO:teuthology.orchestra.run.trial068.stdout: filestore xattr use omap = true 2026-02-24T18:00:28.172 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-24T18:00:28.172 INFO:teuthology.orchestra.run.trial068.stdout: mon clock drift allowed = 1.000 2026-02-24T18:00:28.172 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-24T18:00:28.172 INFO:teuthology.orchestra.run.trial068.stdout: osd crush chooseleaf type = 0 2026-02-24T18:00:28.172 INFO:teuthology.orchestra.run.trial068.stdout: auth debug = true 2026-02-24T18:00:28.172 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-24T18:00:28.173 INFO:teuthology.orchestra.run.trial068.stdout: ms die on old message = true 2026-02-24T18:00:28.173 INFO:teuthology.orchestra.run.trial068.stdout: ms die on bug = true 2026-02-24T18:00:28.173 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-24T18:00:28.173 INFO:teuthology.orchestra.run.trial068.stdout: mon max pg per osd = 10000 # >= luminous 2026-02-24T18:00:28.173 INFO:teuthology.orchestra.run.trial068.stdout: mon pg warn max object skew = 0 2026-02-24T18:00:28.173 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-24T18:00:28.173 INFO:teuthology.orchestra.run.trial068.stdout: # disable pg_autoscaler by default for new pools 2026-02-24T18:00:28.173 INFO:teuthology.orchestra.run.trial068.stdout: osd_pool_default_pg_autoscale_mode = off 2026-02-24T18:00:28.173 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-24T18:00:28.173 INFO:teuthology.orchestra.run.trial068.stdout: osd pool default size = 2 2026-02-24T18:00:28.173 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-24T18:00:28.173 INFO:teuthology.orchestra.run.trial068.stdout: mon osd allow primary affinity = true 2026-02-24T18:00:28.173 INFO:teuthology.orchestra.run.trial068.stdout: mon osd allow pg remap = true 2026-02-24T18:00:28.173 INFO:teuthology.orchestra.run.trial068.stdout: mon warn on legacy crush tunables = false 2026-02-24T18:00:28.173 INFO:teuthology.orchestra.run.trial068.stdout: mon warn on crush straw calc version zero = false 2026-02-24T18:00:28.173 INFO:teuthology.orchestra.run.trial068.stdout: mon warn on no sortbitwise = false 2026-02-24T18:00:28.173 INFO:teuthology.orchestra.run.trial068.stdout: mon warn on osd down out interval zero = false 2026-02-24T18:00:28.173 INFO:teuthology.orchestra.run.trial068.stdout: mon warn on too few osds = false 2026-02-24T18:00:28.173 INFO:teuthology.orchestra.run.trial068.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2026-02-24T18:00:28.174 INFO:teuthology.orchestra.run.trial068.stdout: mon_warn_on_pool_no_redundancy = false 2026-02-24T18:00:28.174 INFO:teuthology.orchestra.run.trial068.stdout: mon_allow_pool_size_one = true 2026-02-24T18:00:28.174 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-24T18:00:28.174 INFO:teuthology.orchestra.run.trial068.stdout: osd pool default erasure code profile = plugin=isa technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2026-02-24T18:00:28.174 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-24T18:00:28.174 INFO:teuthology.orchestra.run.trial068.stdout: osd default data pool replay window = 5 2026-02-24T18:00:28.174 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-24T18:00:28.174 INFO:teuthology.orchestra.run.trial068.stdout: mon allow pool delete = true 2026-02-24T18:00:28.174 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-24T18:00:28.174 INFO:teuthology.orchestra.run.trial068.stdout: mon cluster log file level = debug 2026-02-24T18:00:28.174 INFO:teuthology.orchestra.run.trial068.stdout: debug asserts on shutdown = true 2026-02-24T18:00:28.174 INFO:teuthology.orchestra.run.trial068.stdout: mon health detail to clog = false 2026-02-24T18:00:28.174 INFO:teuthology.orchestra.run.trial068.stdout: mon host = 10.20.193.68 2026-02-24T18:00:28.174 INFO:teuthology.orchestra.run.trial068.stdout: mon election default strategy = 3 2026-02-24T18:00:28.174 INFO:teuthology.orchestra.run.trial068.stdout: fsid = 85c6d86c-aad0-4a30-9c50-f24175e8fcf5 2026-02-24T18:00:28.174 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-24T18:00:28.174 INFO:teuthology.orchestra.run.trial068.stdout:[osd] 2026-02-24T18:00:28.174 INFO:teuthology.orchestra.run.trial068.stdout: osd journal size = 100 2026-02-24T18:00:28.174 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-24T18:00:28.175 INFO:teuthology.orchestra.run.trial068.stdout: osd scrub load threshold = 5.0 2026-02-24T18:00:28.175 INFO:teuthology.orchestra.run.trial068.stdout: osd scrub max interval = 600 2026-02-24T18:00:28.175 INFO:teuthology.orchestra.run.trial068.stdout: osd mclock profile = high_recovery_ops 2026-02-24T18:00:28.175 INFO:teuthology.orchestra.run.trial068.stdout: osd mclock skip benchmark = True 2026-02-24T18:00:28.175 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-24T18:00:28.175 INFO:teuthology.orchestra.run.trial068.stdout: osd recover clone overlap = true 2026-02-24T18:00:28.175 INFO:teuthology.orchestra.run.trial068.stdout: osd recovery max chunk = 1048576 2026-02-24T18:00:28.175 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-24T18:00:28.175 INFO:teuthology.orchestra.run.trial068.stdout: osd debug shutdown = true 2026-02-24T18:00:28.175 INFO:teuthology.orchestra.run.trial068.stdout: osd debug op order = true 2026-02-24T18:00:28.175 INFO:teuthology.orchestra.run.trial068.stdout: osd debug verify stray on activate = true 2026-02-24T18:00:28.175 INFO:teuthology.orchestra.run.trial068.stdout: osd debug trim objects = true 2026-02-24T18:00:28.175 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-24T18:00:28.175 INFO:teuthology.orchestra.run.trial068.stdout: osd open classes on start = true 2026-02-24T18:00:28.175 INFO:teuthology.orchestra.run.trial068.stdout: osd debug pg log writeout = true 2026-02-24T18:00:28.175 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-24T18:00:28.175 INFO:teuthology.orchestra.run.trial068.stdout: osd deep scrub update digest min age = 30 2026-02-24T18:00:28.176 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-24T18:00:28.176 INFO:teuthology.orchestra.run.trial068.stdout: osd map max advance = 10 2026-02-24T18:00:28.176 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-24T18:00:28.176 INFO:teuthology.orchestra.run.trial068.stdout: journal zero on create = true 2026-02-24T18:00:28.176 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-24T18:00:28.176 INFO:teuthology.orchestra.run.trial068.stdout: filestore ondisk finisher threads = 3 2026-02-24T18:00:28.176 INFO:teuthology.orchestra.run.trial068.stdout: filestore apply finisher threads = 3 2026-02-24T18:00:28.176 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-24T18:00:28.176 INFO:teuthology.orchestra.run.trial068.stdout: bdev debug aio = true 2026-02-24T18:00:28.176 INFO:teuthology.orchestra.run.trial068.stdout: osd debug misdirected ops = true 2026-02-24T18:00:28.176 INFO:teuthology.orchestra.run.trial068.stdout: osd max object name len = 400 2026-02-24T18:00:28.176 INFO:teuthology.orchestra.run.trial068.stdout: osd max object namespace len = 64 2026-02-24T18:00:28.176 INFO:teuthology.orchestra.run.trial068.stdout: bluestore zero block detection = True 2026-02-24T18:00:28.176 INFO:teuthology.orchestra.run.trial068.stdout: debug ms = 1 2026-02-24T18:00:28.176 INFO:teuthology.orchestra.run.trial068.stdout: debug osd = 20 2026-02-24T18:00:28.176 INFO:teuthology.orchestra.run.trial068.stdout: osd debug verify cached snaps = True 2026-02-24T18:00:28.176 INFO:teuthology.orchestra.run.trial068.stdout: osd debug verify missing on start = True 2026-02-24T18:00:28.176 INFO:teuthology.orchestra.run.trial068.stdout: osd mclock override recovery settings = True 2026-02-24T18:00:28.176 INFO:teuthology.orchestra.run.trial068.stdout: osd op queue = debug_random 2026-02-24T18:00:28.176 INFO:teuthology.orchestra.run.trial068.stdout: osd op queue cut off = debug_random 2026-02-24T18:00:28.177 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-24T18:00:28.177 INFO:teuthology.orchestra.run.trial068.stdout:[mgr] 2026-02-24T18:00:28.177 INFO:teuthology.orchestra.run.trial068.stdout: debug ms = 1 2026-02-24T18:00:28.177 INFO:teuthology.orchestra.run.trial068.stdout: debug mgr = 20 2026-02-24T18:00:28.177 INFO:teuthology.orchestra.run.trial068.stdout: debug mon = 20 2026-02-24T18:00:28.177 INFO:teuthology.orchestra.run.trial068.stdout: debug auth = 20 2026-02-24T18:00:28.177 INFO:teuthology.orchestra.run.trial068.stdout: mon reweight min pgs per osd = 4 2026-02-24T18:00:28.177 INFO:teuthology.orchestra.run.trial068.stdout: mon reweight min bytes per osd = 10 2026-02-24T18:00:28.177 INFO:teuthology.orchestra.run.trial068.stdout: mgr/telemetry/nag = false 2026-02-24T18:00:28.177 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-24T18:00:28.177 INFO:teuthology.orchestra.run.trial068.stdout:[mon] 2026-02-24T18:00:28.177 INFO:teuthology.orchestra.run.trial068.stdout: debug ms = 1 2026-02-24T18:00:28.177 INFO:teuthology.orchestra.run.trial068.stdout: debug mon = 30 2026-02-24T18:00:28.177 INFO:teuthology.orchestra.run.trial068.stdout: debug paxos = 20 2026-02-24T18:00:28.177 INFO:teuthology.orchestra.run.trial068.stdout: debug auth = 20 2026-02-24T18:00:28.177 INFO:teuthology.orchestra.run.trial068.stdout: mon data avail warn = 5 2026-02-24T18:00:28.177 INFO:teuthology.orchestra.run.trial068.stdout: mon mgr mkfs grace = 240 2026-02-24T18:00:28.177 INFO:teuthology.orchestra.run.trial068.stdout: mon reweight min pgs per osd = 4 2026-02-24T18:00:28.178 INFO:teuthology.orchestra.run.trial068.stdout: mon osd reporter subtree level = osd 2026-02-24T18:00:28.178 INFO:teuthology.orchestra.run.trial068.stdout: mon osd prime pg temp = true 2026-02-24T18:00:28.178 INFO:teuthology.orchestra.run.trial068.stdout: mon reweight min bytes per osd = 10 2026-02-24T18:00:28.178 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-24T18:00:28.178 INFO:teuthology.orchestra.run.trial068.stdout: # rotate auth tickets quickly to exercise renewal paths 2026-02-24T18:00:28.178 INFO:teuthology.orchestra.run.trial068.stdout: auth mon ticket ttl = 660 # 11m 2026-02-24T18:00:28.178 INFO:teuthology.orchestra.run.trial068.stdout: auth service ticket ttl = 240 # 4m 2026-02-24T18:00:28.178 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-24T18:00:28.178 INFO:teuthology.orchestra.run.trial068.stdout: # don't complain about insecure global_id in the test suite 2026-02-24T18:00:28.178 INFO:teuthology.orchestra.run.trial068.stdout: mon_warn_on_insecure_global_id_reclaim = false 2026-02-24T18:00:28.178 INFO:teuthology.orchestra.run.trial068.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2026-02-24T18:00:28.178 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-24T18:00:28.178 INFO:teuthology.orchestra.run.trial068.stdout: # 1m isn't quite enough 2026-02-24T18:00:28.178 INFO:teuthology.orchestra.run.trial068.stdout: mon_down_mkfs_grace = 2m 2026-02-24T18:00:28.178 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-24T18:00:28.178 INFO:teuthology.orchestra.run.trial068.stdout: mon_warn_on_filestore_osds = false 2026-02-24T18:00:28.178 INFO:teuthology.orchestra.run.trial068.stdout: mon scrub interval = 300 2026-02-24T18:00:28.179 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-24T18:00:28.179 INFO:teuthology.orchestra.run.trial068.stdout:[client] 2026-02-24T18:00:28.179 INFO:teuthology.orchestra.run.trial068.stdout: rgw cache enabled = true 2026-02-24T18:00:28.179 INFO:teuthology.orchestra.run.trial068.stdout: rgw enable ops log = true 2026-02-24T18:00:28.179 INFO:teuthology.orchestra.run.trial068.stdout: rgw enable usage log = true 2026-02-24T18:00:28.179 INFO:teuthology.orchestra.run.trial068.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2026-02-24T18:00:28.179 INFO:teuthology.orchestra.run.trial068.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2026-02-24T18:00:28.179 INFO:teuthology.orchestra.run.trial068.stdout:[mon.a] 2026-02-24T18:00:28.180 INFO:tasks.ceph:Creating admin key on mon.a... 2026-02-24T18:00:28.180 DEBUG:teuthology.orchestra.run.trial068:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/ceph.keyring 2026-02-24T18:00:28.245 INFO:tasks.ceph:Copying monmap to all nodes... 2026-02-24T18:00:28.246 DEBUG:teuthology.orchestra.run.trial068:> set -ex 2026-02-24T18:00:28.246 DEBUG:teuthology.orchestra.run.trial068:> dd if=/etc/ceph/ceph.keyring of=/dev/stdout 2026-02-24T18:00:28.296 DEBUG:teuthology.orchestra.run.trial068:> set -ex 2026-02-24T18:00:28.296 DEBUG:teuthology.orchestra.run.trial068:> dd if=/home/ubuntu/cephtest/ceph.monmap of=/dev/stdout 2026-02-24T18:00:28.345 INFO:tasks.ceph:Sending monmap to node ubuntu@trial068.front.sepia.ceph.com 2026-02-24T18:00:28.345 DEBUG:teuthology.orchestra.run.trial068:> set -ex 2026-02-24T18:00:28.345 DEBUG:teuthology.orchestra.run.trial068:> sudo dd of=/etc/ceph/ceph.keyring 2026-02-24T18:00:28.345 DEBUG:teuthology.orchestra.run.trial068:> sudo chmod 0644 /etc/ceph/ceph.keyring 2026-02-24T18:00:28.398 DEBUG:teuthology.orchestra.run.trial068:> set -ex 2026-02-24T18:00:28.399 DEBUG:teuthology.orchestra.run.trial068:> dd of=/home/ubuntu/cephtest/ceph.monmap 2026-02-24T18:00:28.444 INFO:tasks.ceph:Setting up mon nodes... 2026-02-24T18:00:28.445 INFO:tasks.ceph:Setting up mgr nodes... 2026-02-24T18:00:28.445 DEBUG:teuthology.orchestra.run.trial068:> sudo mkdir -p /var/lib/ceph/mgr/ceph-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/ceph-x/keyring 2026-02-24T18:00:28.513 INFO:teuthology.orchestra.run.trial068.stdout:creating /var/lib/ceph/mgr/ceph-x/keyring 2026-02-24T18:00:28.516 INFO:tasks.ceph:Setting up mds nodes... 2026-02-24T18:00:28.516 INFO:tasks.ceph_client:Setting up client nodes... 2026-02-24T18:00:28.516 DEBUG:teuthology.orchestra.run.trial068:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/ceph.client.0.keyring && sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2026-02-24T18:00:28.579 INFO:teuthology.orchestra.run.trial068.stdout:creating /etc/ceph/ceph.client.0.keyring 2026-02-24T18:00:28.586 INFO:tasks.ceph:Running mkfs on osd nodes... 2026-02-24T18:00:28.586 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@trial068.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'}} 2026-02-24T18:00:28.586 DEBUG:teuthology.orchestra.run.trial068:> sudo mkdir -p /var/lib/ceph/osd/ceph-0 2026-02-24T18:00:28.638 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2026-02-24T18:00:28.638 INFO:tasks.ceph:role: osd.0 2026-02-24T18:00:28.638 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@trial068.front.sepia.ceph.com 2026-02-24T18:00:28.638 DEBUG:teuthology.orchestra.run.trial068:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2026-02-24T18:00:28.690 INFO:teuthology.orchestra.run.trial068.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=45875200 blks 2026-02-24T18:00:28.691 INFO:teuthology.orchestra.run.trial068.stdout: = sectsz=4096 attr=2, projid32bit=1 2026-02-24T18:00:28.691 INFO:teuthology.orchestra.run.trial068.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2026-02-24T18:00:28.691 INFO:teuthology.orchestra.run.trial068.stdout: = reflink=1 bigtime=0 inobtcount=0 2026-02-24T18:00:28.691 INFO:teuthology.orchestra.run.trial068.stdout:data = bsize=4096 blocks=183500800, imaxpct=25 2026-02-24T18:00:28.691 INFO:teuthology.orchestra.run.trial068.stdout: = sunit=0 swidth=0 blks 2026-02-24T18:00:28.691 INFO:teuthology.orchestra.run.trial068.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2026-02-24T18:00:28.691 INFO:teuthology.orchestra.run.trial068.stdout:log =internal log bsize=4096 blocks=89600, version=2 2026-02-24T18:00:28.691 INFO:teuthology.orchestra.run.trial068.stdout: = sectsz=4096 sunit=1 blks, lazy-count=1 2026-02-24T18:00:28.691 INFO:teuthology.orchestra.run.trial068.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2026-02-24T18:00:29.284 INFO:teuthology.orchestra.run.trial068.stdout:Discarding blocks...Done. 2026-02-24T18:00:29.285 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@trial068.front.sepia.ceph.com -o noatime 2026-02-24T18:00:29.285 DEBUG:teuthology.orchestra.run.trial068:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-0 2026-02-24T18:00:29.359 DEBUG:teuthology.orchestra.run.trial068:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-0 2026-02-24T18:00:29.426 INFO:teuthology.orchestra.run.trial068.stderr:sudo: /sbin/restorecon: command not found 2026-02-24T18:00:29.426 DEBUG:teuthology.orchestra.run:got remote process result: 1 2026-02-24T18:00:29.426 DEBUG:teuthology.orchestra.run.trial068:> sudo mkdir -p /var/lib/ceph/osd/ceph-1 2026-02-24T18:00:29.481 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2026-02-24T18:00:29.481 INFO:tasks.ceph:role: osd.1 2026-02-24T18:00:29.481 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@trial068.front.sepia.ceph.com 2026-02-24T18:00:29.481 DEBUG:teuthology.orchestra.run.trial068:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2026-02-24T18:00:29.530 INFO:teuthology.orchestra.run.trial068.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=45875200 blks 2026-02-24T18:00:29.530 INFO:teuthology.orchestra.run.trial068.stdout: = sectsz=4096 attr=2, projid32bit=1 2026-02-24T18:00:29.531 INFO:teuthology.orchestra.run.trial068.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2026-02-24T18:00:29.531 INFO:teuthology.orchestra.run.trial068.stdout: = reflink=1 bigtime=0 inobtcount=0 2026-02-24T18:00:29.531 INFO:teuthology.orchestra.run.trial068.stdout:data = bsize=4096 blocks=183500800, imaxpct=25 2026-02-24T18:00:29.531 INFO:teuthology.orchestra.run.trial068.stdout: = sunit=0 swidth=0 blks 2026-02-24T18:00:29.531 INFO:teuthology.orchestra.run.trial068.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2026-02-24T18:00:29.531 INFO:teuthology.orchestra.run.trial068.stdout:log =internal log bsize=4096 blocks=89600, version=2 2026-02-24T18:00:29.531 INFO:teuthology.orchestra.run.trial068.stdout: = sectsz=4096 sunit=1 blks, lazy-count=1 2026-02-24T18:00:29.531 INFO:teuthology.orchestra.run.trial068.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2026-02-24T18:00:30.098 INFO:teuthology.orchestra.run.trial068.stdout:Discarding blocks...Done. 2026-02-24T18:00:30.099 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@trial068.front.sepia.ceph.com -o noatime 2026-02-24T18:00:30.100 DEBUG:teuthology.orchestra.run.trial068:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-1 2026-02-24T18:00:30.155 DEBUG:teuthology.orchestra.run.trial068:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-1 2026-02-24T18:00:30.207 DEBUG:teuthology.orchestra.run:got remote process result: 1 2026-02-24T18:00:30.208 INFO:teuthology.orchestra.run.trial068.stderr:sudo: /sbin/restorecon: command not found 2026-02-24T18:00:30.208 DEBUG:teuthology.orchestra.run.trial068:> sudo mkdir -p /var/lib/ceph/osd/ceph-2 2026-02-24T18:00:30.257 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2026-02-24T18:00:30.258 INFO:tasks.ceph:role: osd.2 2026-02-24T18:00:30.258 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@trial068.front.sepia.ceph.com 2026-02-24T18:00:30.258 DEBUG:teuthology.orchestra.run.trial068:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2026-02-24T18:00:30.310 INFO:teuthology.orchestra.run.trial068.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=45875200 blks 2026-02-24T18:00:30.311 INFO:teuthology.orchestra.run.trial068.stdout: = sectsz=4096 attr=2, projid32bit=1 2026-02-24T18:00:30.311 INFO:teuthology.orchestra.run.trial068.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2026-02-24T18:00:30.311 INFO:teuthology.orchestra.run.trial068.stdout: = reflink=1 bigtime=0 inobtcount=0 2026-02-24T18:00:30.311 INFO:teuthology.orchestra.run.trial068.stdout:data = bsize=4096 blocks=183500800, imaxpct=25 2026-02-24T18:00:30.311 INFO:teuthology.orchestra.run.trial068.stdout: = sunit=0 swidth=0 blks 2026-02-24T18:00:30.311 INFO:teuthology.orchestra.run.trial068.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2026-02-24T18:00:30.311 INFO:teuthology.orchestra.run.trial068.stdout:log =internal log bsize=4096 blocks=89600, version=2 2026-02-24T18:00:30.311 INFO:teuthology.orchestra.run.trial068.stdout: = sectsz=4096 sunit=1 blks, lazy-count=1 2026-02-24T18:00:30.311 INFO:teuthology.orchestra.run.trial068.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2026-02-24T18:00:30.889 INFO:teuthology.orchestra.run.trial068.stdout:Discarding blocks...Done. 2026-02-24T18:00:30.891 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@trial068.front.sepia.ceph.com -o noatime 2026-02-24T18:00:30.891 DEBUG:teuthology.orchestra.run.trial068:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-2 2026-02-24T18:00:30.946 DEBUG:teuthology.orchestra.run.trial068:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-2 2026-02-24T18:00:31.010 DEBUG:teuthology.orchestra.run:got remote process result: 1 2026-02-24T18:00:31.012 INFO:teuthology.orchestra.run.trial068.stderr:sudo: /sbin/restorecon: command not found 2026-02-24T18:00:31.013 DEBUG:teuthology.orchestra.run.trial068:> sudo mkdir -p /var/lib/ceph/osd/ceph-3 2026-02-24T18:00:31.061 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2026-02-24T18:00:31.062 INFO:tasks.ceph:role: osd.3 2026-02-24T18:00:31.062 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@trial068.front.sepia.ceph.com 2026-02-24T18:00:31.062 DEBUG:teuthology.orchestra.run.trial068:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2026-02-24T18:00:31.115 INFO:teuthology.orchestra.run.trial068.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=45875200 blks 2026-02-24T18:00:31.115 INFO:teuthology.orchestra.run.trial068.stdout: = sectsz=4096 attr=2, projid32bit=1 2026-02-24T18:00:31.115 INFO:teuthology.orchestra.run.trial068.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2026-02-24T18:00:31.116 INFO:teuthology.orchestra.run.trial068.stdout: = reflink=1 bigtime=0 inobtcount=0 2026-02-24T18:00:31.116 INFO:teuthology.orchestra.run.trial068.stdout:data = bsize=4096 blocks=183500800, imaxpct=25 2026-02-24T18:00:31.116 INFO:teuthology.orchestra.run.trial068.stdout: = sunit=0 swidth=0 blks 2026-02-24T18:00:31.116 INFO:teuthology.orchestra.run.trial068.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2026-02-24T18:00:31.116 INFO:teuthology.orchestra.run.trial068.stdout:log =internal log bsize=4096 blocks=89600, version=2 2026-02-24T18:00:31.116 INFO:teuthology.orchestra.run.trial068.stdout: = sectsz=4096 sunit=1 blks, lazy-count=1 2026-02-24T18:00:31.116 INFO:teuthology.orchestra.run.trial068.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2026-02-24T18:00:31.679 INFO:teuthology.orchestra.run.trial068.stdout:Discarding blocks...Done. 2026-02-24T18:00:31.680 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@trial068.front.sepia.ceph.com -o noatime 2026-02-24T18:00:31.680 DEBUG:teuthology.orchestra.run.trial068:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-3 2026-02-24T18:00:31.734 DEBUG:teuthology.orchestra.run.trial068:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-3 2026-02-24T18:00:31.785 DEBUG:teuthology.orchestra.run:got remote process result: 1 2026-02-24T18:00:31.785 INFO:teuthology.orchestra.run.trial068.stderr:sudo: /sbin/restorecon: command not found 2026-02-24T18:00:31.786 DEBUG:teuthology.orchestra.run.trial068:> sudo mkdir -p /var/lib/ceph/osd/ceph-4 2026-02-24T18:00:31.837 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2026-02-24T18:00:31.837 INFO:tasks.ceph:role: osd.4 2026-02-24T18:00:31.838 DEBUG:teuthology.orchestra.run.trial068:> sudo mkdir -p /var/lib/ceph/osd/ceph-5 2026-02-24T18:00:31.894 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2026-02-24T18:00:31.895 INFO:tasks.ceph:role: osd.5 2026-02-24T18:00:31.895 DEBUG:teuthology.orchestra.run.trial068:> sudo mkdir -p /var/lib/ceph/osd/ceph-6 2026-02-24T18:00:31.945 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2026-02-24T18:00:31.946 INFO:tasks.ceph:role: osd.6 2026-02-24T18:00:31.946 DEBUG:teuthology.orchestra.run.trial068:> sudo mkdir -p /var/lib/ceph/osd/ceph-7 2026-02-24T18:00:31.997 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2026-02-24T18:00:31.997 INFO:tasks.ceph:role: osd.7 2026-02-24T18:00:31.998 DEBUG:teuthology.orchestra.run.trial068:> sudo mkdir -p /var/lib/ceph/osd/ceph-8 2026-02-24T18:00:32.049 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2026-02-24T18:00:32.049 INFO:tasks.ceph:role: osd.8 2026-02-24T18:00:32.050 DEBUG:teuthology.orchestra.run.trial068:> sudo mkdir -p /var/lib/ceph/osd/ceph-9 2026-02-24T18:00:32.105 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2026-02-24T18:00:32.105 INFO:tasks.ceph:role: osd.9 2026-02-24T18:00:32.106 DEBUG:teuthology.orchestra.run.trial068:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/ceph.monmap 2026-02-24T18:00:32.180 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-24T18:00:32.173+0000 7f722174ca40 -1 auth: error reading file: /var/lib/ceph/osd/ceph-0/keyring: can't open /var/lib/ceph/osd/ceph-0/keyring: (2) No such file or directory 2026-02-24T18:00:32.181 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-24T18:00:32.173+0000 7f722174ca40 -1 created new key in keyring /var/lib/ceph/osd/ceph-0/keyring 2026-02-24T18:00:32.181 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-24T18:00:32.173+0000 7f722174ca40 -1 bdev(0x55b55501d800 /var/lib/ceph/osd/ceph-0/block) open stat got: (1) Operation not permitted 2026-02-24T18:00:32.181 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-24T18:00:32.173+0000 7f722174ca40 -1 bluestore(/var/lib/ceph/osd/ceph-0) _read_fsid unparsable uuid 2026-02-24T18:00:32.822 DEBUG:teuthology.orchestra.run.trial068:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2026-02-24T18:00:32.874 DEBUG:teuthology.orchestra.run.trial068:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/ceph.monmap 2026-02-24T18:00:32.938 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-24T18:00:32.933+0000 7f0dff7d9a40 -1 auth: error reading file: /var/lib/ceph/osd/ceph-1/keyring: can't open /var/lib/ceph/osd/ceph-1/keyring: (2) No such file or directory 2026-02-24T18:00:32.939 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-24T18:00:32.933+0000 7f0dff7d9a40 -1 created new key in keyring /var/lib/ceph/osd/ceph-1/keyring 2026-02-24T18:00:32.939 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-24T18:00:32.933+0000 7f0dff7d9a40 -1 bdev(0x5586e2d01800 /var/lib/ceph/osd/ceph-1/block) open stat got: (1) Operation not permitted 2026-02-24T18:00:32.939 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-24T18:00:32.933+0000 7f0dff7d9a40 -1 bluestore(/var/lib/ceph/osd/ceph-1) _read_fsid unparsable uuid 2026-02-24T18:00:33.580 DEBUG:teuthology.orchestra.run.trial068:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2026-02-24T18:00:33.630 DEBUG:teuthology.orchestra.run.trial068:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/ceph.monmap 2026-02-24T18:00:33.699 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-24T18:00:33.693+0000 7f57efbcca40 -1 auth: error reading file: /var/lib/ceph/osd/ceph-2/keyring: can't open /var/lib/ceph/osd/ceph-2/keyring: (2) No such file or directory 2026-02-24T18:00:33.700 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-24T18:00:33.693+0000 7f57efbcca40 -1 created new key in keyring /var/lib/ceph/osd/ceph-2/keyring 2026-02-24T18:00:33.700 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-24T18:00:33.693+0000 7f57efbcca40 -1 bdev(0x559701105800 /var/lib/ceph/osd/ceph-2/block) open stat got: (1) Operation not permitted 2026-02-24T18:00:33.700 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-24T18:00:33.693+0000 7f57efbcca40 -1 bluestore(/var/lib/ceph/osd/ceph-2) _read_fsid unparsable uuid 2026-02-24T18:00:34.314 DEBUG:teuthology.orchestra.run.trial068:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2026-02-24T18:00:34.366 DEBUG:teuthology.orchestra.run.trial068:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 3 --monmap /home/ubuntu/cephtest/ceph.monmap 2026-02-24T18:00:34.434 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-24T18:00:34.429+0000 7ff0b84d9a40 -1 auth: error reading file: /var/lib/ceph/osd/ceph-3/keyring: can't open /var/lib/ceph/osd/ceph-3/keyring: (2) No such file or directory 2026-02-24T18:00:34.435 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-24T18:00:34.429+0000 7ff0b84d9a40 -1 created new key in keyring /var/lib/ceph/osd/ceph-3/keyring 2026-02-24T18:00:34.435 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-24T18:00:34.429+0000 7ff0b84d9a40 -1 bdev(0x55f681267800 /var/lib/ceph/osd/ceph-3/block) open stat got: (1) Operation not permitted 2026-02-24T18:00:34.435 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-24T18:00:34.429+0000 7ff0b84d9a40 -1 bluestore(/var/lib/ceph/osd/ceph-3) _read_fsid unparsable uuid 2026-02-24T18:00:35.038 DEBUG:teuthology.orchestra.run.trial068:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2026-02-24T18:00:35.094 DEBUG:teuthology.orchestra.run.trial068:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 4 --monmap /home/ubuntu/cephtest/ceph.monmap 2026-02-24T18:00:35.164 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-24T18:00:35.157+0000 7f11a0d81a40 -1 auth: error reading file: /var/lib/ceph/osd/ceph-4/keyring: can't open /var/lib/ceph/osd/ceph-4/keyring: (2) No such file or directory 2026-02-24T18:00:35.164 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-24T18:00:35.157+0000 7f11a0d81a40 -1 created new key in keyring /var/lib/ceph/osd/ceph-4/keyring 2026-02-24T18:00:35.164 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-24T18:00:35.157+0000 7f11a0d81a40 -1 bdev(0x557509f1f800 /var/lib/ceph/osd/ceph-4/block) open stat got: (1) Operation not permitted 2026-02-24T18:00:35.165 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-24T18:00:35.157+0000 7f11a0d81a40 -1 bluestore(/var/lib/ceph/osd/ceph-4) _read_fsid unparsable uuid 2026-02-24T18:00:35.743 DEBUG:teuthology.orchestra.run.trial068:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2026-02-24T18:00:35.798 DEBUG:teuthology.orchestra.run.trial068:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 5 --monmap /home/ubuntu/cephtest/ceph.monmap 2026-02-24T18:00:35.866 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-24T18:00:35.861+0000 7f9f4979ba40 -1 auth: error reading file: /var/lib/ceph/osd/ceph-5/keyring: can't open /var/lib/ceph/osd/ceph-5/keyring: (2) No such file or directory 2026-02-24T18:00:35.867 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-24T18:00:35.861+0000 7f9f4979ba40 -1 created new key in keyring /var/lib/ceph/osd/ceph-5/keyring 2026-02-24T18:00:35.867 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-24T18:00:35.861+0000 7f9f4979ba40 -1 bdev(0x56327e16b800 /var/lib/ceph/osd/ceph-5/block) open stat got: (1) Operation not permitted 2026-02-24T18:00:35.867 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-24T18:00:35.861+0000 7f9f4979ba40 -1 bluestore(/var/lib/ceph/osd/ceph-5) _read_fsid unparsable uuid 2026-02-24T18:00:36.510 DEBUG:teuthology.orchestra.run.trial068:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2026-02-24T18:00:36.566 DEBUG:teuthology.orchestra.run.trial068:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 6 --monmap /home/ubuntu/cephtest/ceph.monmap 2026-02-24T18:00:36.640 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-24T18:00:36.633+0000 7f6bcb409a40 -1 auth: error reading file: /var/lib/ceph/osd/ceph-6/keyring: can't open /var/lib/ceph/osd/ceph-6/keyring: (2) No such file or directory 2026-02-24T18:00:36.640 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-24T18:00:36.633+0000 7f6bcb409a40 -1 created new key in keyring /var/lib/ceph/osd/ceph-6/keyring 2026-02-24T18:00:36.641 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-24T18:00:36.633+0000 7f6bcb409a40 -1 bdev(0x556ab7393800 /var/lib/ceph/osd/ceph-6/block) open stat got: (1) Operation not permitted 2026-02-24T18:00:36.641 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-24T18:00:36.633+0000 7f6bcb409a40 -1 bluestore(/var/lib/ceph/osd/ceph-6) _read_fsid unparsable uuid 2026-02-24T18:00:37.271 DEBUG:teuthology.orchestra.run.trial068:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-6 2026-02-24T18:00:37.321 DEBUG:teuthology.orchestra.run.trial068:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 7 --monmap /home/ubuntu/cephtest/ceph.monmap 2026-02-24T18:00:37.386 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-24T18:00:37.381+0000 7f991eac7a40 -1 auth: error reading file: /var/lib/ceph/osd/ceph-7/keyring: can't open /var/lib/ceph/osd/ceph-7/keyring: (2) No such file or directory 2026-02-24T18:00:37.386 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-24T18:00:37.381+0000 7f991eac7a40 -1 created new key in keyring /var/lib/ceph/osd/ceph-7/keyring 2026-02-24T18:00:37.387 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-24T18:00:37.381+0000 7f991eac7a40 -1 bdev(0x561747895800 /var/lib/ceph/osd/ceph-7/block) open stat got: (1) Operation not permitted 2026-02-24T18:00:37.387 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-24T18:00:37.381+0000 7f991eac7a40 -1 bluestore(/var/lib/ceph/osd/ceph-7) _read_fsid unparsable uuid 2026-02-24T18:00:38.029 DEBUG:teuthology.orchestra.run.trial068:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-7 2026-02-24T18:00:38.078 DEBUG:teuthology.orchestra.run.trial068:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 8 --monmap /home/ubuntu/cephtest/ceph.monmap 2026-02-24T18:00:38.142 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-24T18:00:38.137+0000 7f1320408a40 -1 auth: error reading file: /var/lib/ceph/osd/ceph-8/keyring: can't open /var/lib/ceph/osd/ceph-8/keyring: (2) No such file or directory 2026-02-24T18:00:38.142 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-24T18:00:38.137+0000 7f1320408a40 -1 created new key in keyring /var/lib/ceph/osd/ceph-8/keyring 2026-02-24T18:00:38.142 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-24T18:00:38.137+0000 7f1320408a40 -1 bdev(0x55ea74dbf800 /var/lib/ceph/osd/ceph-8/block) open stat got: (1) Operation not permitted 2026-02-24T18:00:38.142 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-24T18:00:38.137+0000 7f1320408a40 -1 bluestore(/var/lib/ceph/osd/ceph-8) _read_fsid unparsable uuid 2026-02-24T18:00:38.770 DEBUG:teuthology.orchestra.run.trial068:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-8 2026-02-24T18:00:38.822 DEBUG:teuthology.orchestra.run.trial068:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 9 --monmap /home/ubuntu/cephtest/ceph.monmap 2026-02-24T18:00:38.886 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-24T18:00:38.881+0000 7f815b0fda40 -1 auth: error reading file: /var/lib/ceph/osd/ceph-9/keyring: can't open /var/lib/ceph/osd/ceph-9/keyring: (2) No such file or directory 2026-02-24T18:00:38.886 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-24T18:00:38.881+0000 7f815b0fda40 -1 created new key in keyring /var/lib/ceph/osd/ceph-9/keyring 2026-02-24T18:00:38.886 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-24T18:00:38.881+0000 7f815b0fda40 -1 bdev(0x5612d4fe9800 /var/lib/ceph/osd/ceph-9/block) open stat got: (1) Operation not permitted 2026-02-24T18:00:38.886 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-24T18:00:38.881+0000 7f815b0fda40 -1 bluestore(/var/lib/ceph/osd/ceph-9) _read_fsid unparsable uuid 2026-02-24T18:00:39.491 DEBUG:teuthology.orchestra.run.trial068:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-9 2026-02-24T18:00:39.542 INFO:tasks.ceph:Reading keys from all nodes... 2026-02-24T18:00:39.543 DEBUG:teuthology.orchestra.run.trial068:> set -ex 2026-02-24T18:00:39.543 DEBUG:teuthology.orchestra.run.trial068:> sudo dd if=/var/lib/ceph/mgr/ceph-x/keyring of=/dev/stdout 2026-02-24T18:00:39.594 DEBUG:teuthology.orchestra.run.trial068:> set -ex 2026-02-24T18:00:39.595 DEBUG:teuthology.orchestra.run.trial068:> sudo dd if=/var/lib/ceph/osd/ceph-0/keyring of=/dev/stdout 2026-02-24T18:00:39.646 DEBUG:teuthology.orchestra.run.trial068:> set -ex 2026-02-24T18:00:39.647 DEBUG:teuthology.orchestra.run.trial068:> sudo dd if=/var/lib/ceph/osd/ceph-1/keyring of=/dev/stdout 2026-02-24T18:00:39.698 DEBUG:teuthology.orchestra.run.trial068:> set -ex 2026-02-24T18:00:39.699 DEBUG:teuthology.orchestra.run.trial068:> sudo dd if=/var/lib/ceph/osd/ceph-2/keyring of=/dev/stdout 2026-02-24T18:00:39.750 DEBUG:teuthology.orchestra.run.trial068:> set -ex 2026-02-24T18:00:39.751 DEBUG:teuthology.orchestra.run.trial068:> sudo dd if=/var/lib/ceph/osd/ceph-3/keyring of=/dev/stdout 2026-02-24T18:00:39.802 DEBUG:teuthology.orchestra.run.trial068:> set -ex 2026-02-24T18:00:39.802 DEBUG:teuthology.orchestra.run.trial068:> sudo dd if=/var/lib/ceph/osd/ceph-4/keyring of=/dev/stdout 2026-02-24T18:00:39.854 DEBUG:teuthology.orchestra.run.trial068:> set -ex 2026-02-24T18:00:39.854 DEBUG:teuthology.orchestra.run.trial068:> sudo dd if=/var/lib/ceph/osd/ceph-5/keyring of=/dev/stdout 2026-02-24T18:00:39.905 DEBUG:teuthology.orchestra.run.trial068:> set -ex 2026-02-24T18:00:39.905 DEBUG:teuthology.orchestra.run.trial068:> sudo dd if=/var/lib/ceph/osd/ceph-6/keyring of=/dev/stdout 2026-02-24T18:00:39.958 DEBUG:teuthology.orchestra.run.trial068:> set -ex 2026-02-24T18:00:39.958 DEBUG:teuthology.orchestra.run.trial068:> sudo dd if=/var/lib/ceph/osd/ceph-7/keyring of=/dev/stdout 2026-02-24T18:00:40.008 DEBUG:teuthology.orchestra.run.trial068:> set -ex 2026-02-24T18:00:40.008 DEBUG:teuthology.orchestra.run.trial068:> sudo dd if=/var/lib/ceph/osd/ceph-8/keyring of=/dev/stdout 2026-02-24T18:00:40.057 DEBUG:teuthology.orchestra.run.trial068:> set -ex 2026-02-24T18:00:40.057 DEBUG:teuthology.orchestra.run.trial068:> sudo dd if=/var/lib/ceph/osd/ceph-9/keyring of=/dev/stdout 2026-02-24T18:00:40.104 DEBUG:teuthology.orchestra.run.trial068:> set -ex 2026-02-24T18:00:40.105 DEBUG:teuthology.orchestra.run.trial068:> dd if=/etc/ceph/ceph.client.0.keyring of=/dev/stdout 2026-02-24T18:00:40.149 INFO:tasks.ceph:Adding keys to all mons... 2026-02-24T18:00:40.149 DEBUG:teuthology.orchestra.run.trial068:> sudo tee -a /etc/ceph/ceph.keyring 2026-02-24T18:00:40.200 INFO:teuthology.orchestra.run.trial068.stdout:[mgr.x] 2026-02-24T18:00:40.200 INFO:teuthology.orchestra.run.trial068.stdout: key = AQA8551pOkpsHhAACjyBQ9uS7h6uj2j/Uokhhw== 2026-02-24T18:00:40.200 INFO:teuthology.orchestra.run.trial068.stdout:[osd.0] 2026-02-24T18:00:40.200 INFO:teuthology.orchestra.run.trial068.stdout: key = AQBA551pKYqcChAAiBgCwcBq3HLHV/uByaDK8w== 2026-02-24T18:00:40.200 INFO:teuthology.orchestra.run.trial068.stdout:[osd.1] 2026-02-24T18:00:40.200 INFO:teuthology.orchestra.run.trial068.stdout: key = AQBA551pSgbGNxAAeec7JGyTx1aAITBSE7wkaA== 2026-02-24T18:00:40.200 INFO:teuthology.orchestra.run.trial068.stdout:[osd.2] 2026-02-24T18:00:40.200 INFO:teuthology.orchestra.run.trial068.stdout: key = AQBB551pz0WAKRAATASAd7aMeb5kJJEUfi3NKQ== 2026-02-24T18:00:40.200 INFO:teuthology.orchestra.run.trial068.stdout:[osd.3] 2026-02-24T18:00:40.200 INFO:teuthology.orchestra.run.trial068.stdout: key = AQBC551pG2a5GRAAV6R1leUsnY31hMtkofVY0A== 2026-02-24T18:00:40.200 INFO:teuthology.orchestra.run.trial068.stdout:[osd.4] 2026-02-24T18:00:40.200 INFO:teuthology.orchestra.run.trial068.stdout: key = AQBD551pQRuaCRAATfD+HSwn9+x5RICw7T8Ylw== 2026-02-24T18:00:40.201 INFO:teuthology.orchestra.run.trial068.stdout:[osd.5] 2026-02-24T18:00:40.201 INFO:teuthology.orchestra.run.trial068.stdout: key = AQBD551pTIt2MxAAwjwv6H/DPBZyf6Ie4Xq4vw== 2026-02-24T18:00:40.201 INFO:teuthology.orchestra.run.trial068.stdout:[osd.6] 2026-02-24T18:00:40.201 INFO:teuthology.orchestra.run.trial068.stdout: key = AQBE551pXqH1JRAAtUOSKT2t/wXuwT2BRCBV7Q== 2026-02-24T18:00:40.201 INFO:teuthology.orchestra.run.trial068.stdout:[osd.7] 2026-02-24T18:00:40.201 INFO:teuthology.orchestra.run.trial068.stdout: key = AQBF551p/nHWFhAAr9W8A+Gmii7tonIXC4en3w== 2026-02-24T18:00:40.201 INFO:teuthology.orchestra.run.trial068.stdout:[osd.8] 2026-02-24T18:00:40.201 INFO:teuthology.orchestra.run.trial068.stdout: key = AQBG551pBkxOCBAAyaWUD0MBVKMx2L5zbcilrw== 2026-02-24T18:00:40.201 INFO:teuthology.orchestra.run.trial068.stdout:[osd.9] 2026-02-24T18:00:40.201 INFO:teuthology.orchestra.run.trial068.stdout: key = AQBG551pgluiNBAAawpAVScAL1Tftsj9ltx/FQ== 2026-02-24T18:00:40.201 INFO:teuthology.orchestra.run.trial068.stdout:[client.0] 2026-02-24T18:00:40.201 INFO:teuthology.orchestra.run.trial068.stdout: key = AQA8551pYgFbIhAAq9vzMDpcOkP3mUqC+CQ4XQ== 2026-02-24T18:00:40.202 DEBUG:teuthology.orchestra.run.trial068:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2026-02-24T18:00:40.267 DEBUG:teuthology.orchestra.run.trial068:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2026-02-24T18:00:40.330 DEBUG:teuthology.orchestra.run.trial068:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2026-02-24T18:00:40.408 DEBUG:teuthology.orchestra.run.trial068:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2026-02-24T18:00:40.480 DEBUG:teuthology.orchestra.run.trial068:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2026-02-24T18:00:40.542 DEBUG:teuthology.orchestra.run.trial068:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2026-02-24T18:00:40.606 DEBUG:teuthology.orchestra.run.trial068:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2026-02-24T18:00:40.670 DEBUG:teuthology.orchestra.run.trial068:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2026-02-24T18:00:40.735 DEBUG:teuthology.orchestra.run.trial068:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2026-02-24T18:00:40.799 DEBUG:teuthology.orchestra.run.trial068:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.8 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2026-02-24T18:00:40.863 DEBUG:teuthology.orchestra.run.trial068:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.9 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2026-02-24T18:00:40.925 DEBUG:teuthology.orchestra.run.trial068:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2026-02-24T18:00:40.991 INFO:tasks.ceph:Running mkfs on mon nodes... 2026-02-24T18:00:40.991 DEBUG:teuthology.orchestra.run.trial068:> sudo mkdir -p /var/lib/ceph/mon/ceph-a 2026-02-24T18:00:41.041 DEBUG:teuthology.orchestra.run.trial068:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i a --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2026-02-24T18:00:41.106 DEBUG:teuthology.orchestra.run.trial068:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-a 2026-02-24T18:00:41.157 DEBUG:teuthology.orchestra.run.trial068:> rm -- /home/ubuntu/cephtest/ceph.monmap 2026-02-24T18:00:41.200 INFO:tasks.ceph:Starting mon daemons in cluster ceph... 2026-02-24T18:00:41.200 INFO:tasks.ceph.mon.a:Restarting daemon 2026-02-24T18:00:41.200 DEBUG:teuthology.orchestra.run.trial068:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2026-02-24T18:00:41.242 INFO:tasks.ceph.mon.a:Started 2026-02-24T18:00:41.242 DEBUG:teuthology.orchestra.run.trial068:> sudo ceph config set mgr mgr_pool false --force 2026-02-24T18:00:41.376 INFO:tasks.ceph:Starting mgr daemons in cluster ceph... 2026-02-24T18:00:41.376 INFO:tasks.ceph.mgr.x:Restarting daemon 2026-02-24T18:00:41.377 DEBUG:teuthology.orchestra.run.trial068:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i x 2026-02-24T18:00:41.378 INFO:tasks.ceph.mgr.x:Started 2026-02-24T18:00:41.378 DEBUG:tasks.ceph:set 0 configs 2026-02-24T18:00:41.378 DEBUG:teuthology.orchestra.run.trial068:> sudo ceph --cluster ceph config dump 2026-02-24T18:00:41.471 INFO:teuthology.orchestra.run.trial068.stdout:WHO MASK LEVEL OPTION VALUE RO 2026-02-24T18:00:41.471 INFO:teuthology.orchestra.run.trial068.stdout:mgr dev mgr_pool false * 2026-02-24T18:00:41.480 INFO:tasks.ceph:Setting crush tunables to default 2026-02-24T18:00:41.480 DEBUG:teuthology.orchestra.run.trial068:> sudo ceph --cluster ceph osd crush tunables default 2026-02-24T18:00:41.583 INFO:teuthology.orchestra.run.trial068.stderr:adjusted tunables profile to default 2026-02-24T18:00:41.592 INFO:tasks.ceph:check_enable_crimson: False 2026-02-24T18:00:41.592 INFO:tasks.ceph:Starting osd daemons in cluster ceph... 2026-02-24T18:00:41.592 DEBUG:teuthology.orchestra.run.trial068:> set -ex 2026-02-24T18:00:41.592 DEBUG:teuthology.orchestra.run.trial068:> sudo dd if=/var/lib/ceph/osd/ceph-0/fsid of=/dev/stdout 2026-02-24T18:00:41.600 DEBUG:teuthology.orchestra.run.trial068:> set -ex 2026-02-24T18:00:41.600 DEBUG:teuthology.orchestra.run.trial068:> sudo dd if=/var/lib/ceph/osd/ceph-1/fsid of=/dev/stdout 2026-02-24T18:00:41.650 DEBUG:teuthology.orchestra.run.trial068:> set -ex 2026-02-24T18:00:41.650 DEBUG:teuthology.orchestra.run.trial068:> sudo dd if=/var/lib/ceph/osd/ceph-2/fsid of=/dev/stdout 2026-02-24T18:00:41.700 DEBUG:teuthology.orchestra.run.trial068:> set -ex 2026-02-24T18:00:41.701 DEBUG:teuthology.orchestra.run.trial068:> sudo dd if=/var/lib/ceph/osd/ceph-3/fsid of=/dev/stdout 2026-02-24T18:00:41.748 DEBUG:teuthology.orchestra.run.trial068:> set -ex 2026-02-24T18:00:41.749 DEBUG:teuthology.orchestra.run.trial068:> sudo dd if=/var/lib/ceph/osd/ceph-4/fsid of=/dev/stdout 2026-02-24T18:00:41.796 DEBUG:teuthology.orchestra.run.trial068:> set -ex 2026-02-24T18:00:41.796 DEBUG:teuthology.orchestra.run.trial068:> sudo dd if=/var/lib/ceph/osd/ceph-5/fsid of=/dev/stdout 2026-02-24T18:00:41.844 DEBUG:teuthology.orchestra.run.trial068:> set -ex 2026-02-24T18:00:41.844 DEBUG:teuthology.orchestra.run.trial068:> sudo dd if=/var/lib/ceph/osd/ceph-6/fsid of=/dev/stdout 2026-02-24T18:00:41.892 DEBUG:teuthology.orchestra.run.trial068:> set -ex 2026-02-24T18:00:41.892 DEBUG:teuthology.orchestra.run.trial068:> sudo dd if=/var/lib/ceph/osd/ceph-7/fsid of=/dev/stdout 2026-02-24T18:00:41.940 DEBUG:teuthology.orchestra.run.trial068:> set -ex 2026-02-24T18:00:41.940 DEBUG:teuthology.orchestra.run.trial068:> sudo dd if=/var/lib/ceph/osd/ceph-8/fsid of=/dev/stdout 2026-02-24T18:00:41.988 DEBUG:teuthology.orchestra.run.trial068:> set -ex 2026-02-24T18:00:41.988 DEBUG:teuthology.orchestra.run.trial068:> sudo dd if=/var/lib/ceph/osd/ceph-9/fsid of=/dev/stdout 2026-02-24T18:00:42.036 DEBUG:teuthology.orchestra.run.trial068:> sudo ceph --cluster ceph osd new 0685fb8e-c8df-4155-aa67-77c8085c54fa 0 2026-02-24T18:00:42.176 INFO:teuthology.orchestra.run.trial068.stdout:0 2026-02-24T18:00:42.186 DEBUG:teuthology.orchestra.run.trial068:> sudo ceph --cluster ceph osd new e6c8aa5f-faf0-46d2-8185-92c1037749e2 1 2026-02-24T18:00:42.288 INFO:teuthology.orchestra.run.trial068.stdout:1 2026-02-24T18:00:42.296 DEBUG:teuthology.orchestra.run.trial068:> sudo ceph --cluster ceph osd new 9de05692-da99-43ea-8eeb-7388526b5abc 2 2026-02-24T18:00:42.396 INFO:teuthology.orchestra.run.trial068.stdout:2 2026-02-24T18:00:42.407 DEBUG:teuthology.orchestra.run.trial068:> sudo ceph --cluster ceph osd new e221ca44-382e-4924-ae30-d1add04b6f0d 3 2026-02-24T18:00:42.510 INFO:teuthology.orchestra.run.trial068.stdout:3 2026-02-24T18:00:42.519 DEBUG:teuthology.orchestra.run.trial068:> sudo ceph --cluster ceph osd new 59a23325-5017-4345-ad4b-6cde76f1f74d 4 2026-02-24T18:00:42.619 INFO:teuthology.orchestra.run.trial068.stdout:4 2026-02-24T18:00:42.629 DEBUG:teuthology.orchestra.run.trial068:> sudo ceph --cluster ceph osd new 0d906a72-c6d6-4072-b5fb-383a18f376e4 5 2026-02-24T18:00:42.728 INFO:teuthology.orchestra.run.trial068.stdout:5 2026-02-24T18:00:42.737 DEBUG:teuthology.orchestra.run.trial068:> sudo ceph --cluster ceph osd new 89445f92-371d-492c-bf3f-814aa6533ad0 6 2026-02-24T18:00:42.831 INFO:tasks.ceph.mgr.x.trial068.stderr:ERROR:root:Module 'jmespath' is not installed. 2026-02-24T18:00:42.837 INFO:teuthology.orchestra.run.trial068.stdout:6 2026-02-24T18:00:42.845 DEBUG:teuthology.orchestra.run.trial068:> sudo ceph --cluster ceph osd new 3a2e9d71-2ddc-4b3b-8417-e70a1794c3c2 7 2026-02-24T18:00:42.884 INFO:tasks.ceph.mgr.x.trial068.stderr:ERROR:dashboard.controllers.nvmeof:Failed to import NVMeoFClient and related components: cannot import name 'NVMeoFClient' from 'dashboard.services.nvmeof_client' (/usr/share/ceph/mgr/dashboard/services/nvmeof_client.py) 2026-02-24T18:00:42.889 INFO:tasks.ceph.mgr.x.trial068.stderr:ERROR:sso:Module 'jmespath' is not installed. 2026-02-24T18:00:42.940 INFO:tasks.ceph.mgr.x.trial068.stderr:ERROR:root:Module 'xmltodict' is not installed. 2026-02-24T18:00:42.946 INFO:teuthology.orchestra.run.trial068.stdout:7 2026-02-24T18:00:42.956 DEBUG:teuthology.orchestra.run.trial068:> sudo ceph --cluster ceph osd new 98acdd38-bad4-49c7-a681-772ff7e34dcc 8 2026-02-24T18:00:43.104 INFO:teuthology.orchestra.run.trial068.stdout:8 2026-02-24T18:00:43.113 DEBUG:teuthology.orchestra.run.trial068:> sudo ceph --cluster ceph osd new e2c6da1d-87c9-4ebf-8234-4356adbb2bb5 9 2026-02-24T18:00:43.211 INFO:teuthology.orchestra.run.trial068.stdout:9 2026-02-24T18:00:43.220 INFO:tasks.ceph.osd.0:Restarting daemon 2026-02-24T18:00:43.221 DEBUG:teuthology.orchestra.run.trial068:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 0 2026-02-24T18:00:43.222 INFO:tasks.ceph.osd.0:Started 2026-02-24T18:00:43.222 INFO:tasks.ceph.osd.1:Restarting daemon 2026-02-24T18:00:43.222 DEBUG:teuthology.orchestra.run.trial068:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 1 2026-02-24T18:00:43.223 INFO:tasks.ceph.osd.1:Started 2026-02-24T18:00:43.223 INFO:tasks.ceph.osd.2:Restarting daemon 2026-02-24T18:00:43.223 DEBUG:teuthology.orchestra.run.trial068:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 2 2026-02-24T18:00:43.224 INFO:tasks.ceph.osd.2:Started 2026-02-24T18:00:43.224 INFO:tasks.ceph.osd.3:Restarting daemon 2026-02-24T18:00:43.224 DEBUG:teuthology.orchestra.run.trial068:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 3 2026-02-24T18:00:43.225 INFO:tasks.ceph.osd.3:Started 2026-02-24T18:00:43.225 INFO:tasks.ceph.osd.4:Restarting daemon 2026-02-24T18:00:43.225 DEBUG:teuthology.orchestra.run.trial068:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 4 2026-02-24T18:00:43.226 INFO:tasks.ceph.osd.4:Started 2026-02-24T18:00:43.226 INFO:tasks.ceph.osd.5:Restarting daemon 2026-02-24T18:00:43.226 DEBUG:teuthology.orchestra.run.trial068:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 5 2026-02-24T18:00:43.227 INFO:tasks.ceph.osd.5:Started 2026-02-24T18:00:43.227 INFO:tasks.ceph.osd.6:Restarting daemon 2026-02-24T18:00:43.227 DEBUG:teuthology.orchestra.run.trial068:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 6 2026-02-24T18:00:43.228 INFO:tasks.ceph.osd.6:Started 2026-02-24T18:00:43.228 INFO:tasks.ceph.osd.7:Restarting daemon 2026-02-24T18:00:43.228 DEBUG:teuthology.orchestra.run.trial068:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 7 2026-02-24T18:00:43.230 INFO:tasks.ceph.osd.7:Started 2026-02-24T18:00:43.230 INFO:tasks.ceph.osd.8:Restarting daemon 2026-02-24T18:00:43.230 DEBUG:teuthology.orchestra.run.trial068:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 8 2026-02-24T18:00:43.231 INFO:tasks.ceph.osd.8:Started 2026-02-24T18:00:43.231 INFO:tasks.ceph.osd.9:Restarting daemon 2026-02-24T18:00:43.231 DEBUG:teuthology.orchestra.run.trial068:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 9 2026-02-24T18:00:43.232 INFO:tasks.ceph.osd.9:Started 2026-02-24T18:00:43.232 DEBUG:teuthology.orchestra.run.trial068:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2026-02-24T18:00:43.354 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-24T18:00:43.355 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":12,"fsid":"85c6d86c-aad0-4a30-9c50-f24175e8fcf5","created":"2026-02-24T18:00:41.320093+0000","modified":"2026-02-24T18:00:43.208336+0000","last_up_change":"0.000000","last_in_change":"2026-02-24T18:00:43.208336+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":10,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"0685fb8e-c8df-4155-aa67-77c8085c54fa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"e6c8aa5f-faf0-46d2-8185-92c1037749e2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"9de05692-da99-43ea-8eeb-7388526b5abc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"e221ca44-382e-4924-ae30-d1add04b6f0d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"59a23325-5017-4345-ad4b-6cde76f1f74d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"0d906a72-c6d6-4072-b5fb-383a18f376e4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"89445f92-371d-492c-bf3f-814aa6533ad0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"3a2e9d71-2ddc-4b3b-8417-e70a1794c3c2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":8,"uuid":"98acdd38-bad4-49c7-a681-772ff7e34dcc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":9,"uuid":"e2c6da1d-87c9-4ebf-8234-4356adbb2bb5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-24T18:00:43.374 INFO:tasks.ceph.ceph_manager.ceph:[] 2026-02-24T18:00:43.374 INFO:tasks.ceph:Waiting for OSDs to come up 2026-02-24T18:00:43.393 INFO:tasks.ceph.mgr.x.trial068.stderr:/usr/lib/python3/dist-packages/scipy/__init__.py:67: UserWarning: NumPy was imported from a Python sub-interpreter but NumPy does not properly support sub-interpreters. This will likely work for most users but might cause hard to track down issues or subtle bugs. A common user of the rare sub-interpreter feature is wsgi which also allows single-interpreter mode. 2026-02-24T18:00:43.393 INFO:tasks.ceph.mgr.x.trial068.stderr:Improvements in the case of bugs are welcome, but is not on the NumPy roadmap, and full support may require significant effort to achieve. 2026-02-24T18:00:43.393 INFO:tasks.ceph.mgr.x.trial068.stderr: from numpy import show_config as show_numpy_config 2026-02-24T18:00:43.687 INFO:tasks.ceph.osd.8.trial068.stderr:2026-02-24T18:00:43.681+0000 7f75cab7da40 -1 Falling back to public interface 2026-02-24T18:00:43.723 INFO:tasks.ceph.osd.1.trial068.stderr:2026-02-24T18:00:43.717+0000 7f54b2b22a40 -1 Falling back to public interface 2026-02-24T18:00:43.731 INFO:tasks.ceph.osd.9.trial068.stderr:2026-02-24T18:00:43.725+0000 7f067a7a9a40 -1 Falling back to public interface 2026-02-24T18:00:43.735 INFO:tasks.ceph.osd.0.trial068.stderr:2026-02-24T18:00:43.729+0000 7f75bc5baa40 -1 Falling back to public interface 2026-02-24T18:00:43.735 INFO:tasks.ceph.osd.5.trial068.stderr:2026-02-24T18:00:43.729+0000 7f1724265a40 -1 Falling back to public interface 2026-02-24T18:00:43.755 INFO:tasks.ceph.osd.4.trial068.stderr:2026-02-24T18:00:43.749+0000 7fcc5ae51a40 -1 Falling back to public interface 2026-02-24T18:00:43.766 INFO:tasks.ceph.osd.6.trial068.stderr:2026-02-24T18:00:43.761+0000 7fef31a3ca40 -1 Falling back to public interface 2026-02-24T18:00:43.783 INFO:tasks.ceph.osd.2.trial068.stderr:2026-02-24T18:00:43.777+0000 7fc3c0063a40 -1 Falling back to public interface 2026-02-24T18:00:43.790 INFO:tasks.ceph.osd.3.trial068.stderr:2026-02-24T18:00:43.785+0000 7fd408058a40 -1 Falling back to public interface 2026-02-24T18:00:43.822 INFO:tasks.ceph.osd.7.trial068.stderr:2026-02-24T18:00:43.817+0000 7fb80bf1fa40 -1 Falling back to public interface 2026-02-24T18:00:44.380 DEBUG:teuthology.orchestra.run.trial068:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2026-02-24T18:00:44.464 INFO:tasks.ceph.mgr.x.trial068.stderr:2026-02-24T18:00:44.457+0000 7f0d1ce09640 -1 mgr.server handle_report got status from non-daemon mon.a 2026-02-24T18:00:44.472 INFO:teuthology.misc.health.trial068.stdout: 2026-02-24T18:00:44.472 INFO:teuthology.misc.health.trial068.stdout:{"epoch":12,"fsid":"85c6d86c-aad0-4a30-9c50-f24175e8fcf5","created":"2026-02-24T18:00:41.320093+0000","modified":"2026-02-24T18:00:43.208336+0000","last_up_change":"0.000000","last_in_change":"2026-02-24T18:00:43.208336+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":10,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"0685fb8e-c8df-4155-aa67-77c8085c54fa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"e6c8aa5f-faf0-46d2-8185-92c1037749e2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"9de05692-da99-43ea-8eeb-7388526b5abc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"e221ca44-382e-4924-ae30-d1add04b6f0d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"59a23325-5017-4345-ad4b-6cde76f1f74d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"0d906a72-c6d6-4072-b5fb-383a18f376e4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"89445f92-371d-492c-bf3f-814aa6533ad0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"3a2e9d71-2ddc-4b3b-8417-e70a1794c3c2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":8,"uuid":"98acdd38-bad4-49c7-a681-772ff7e34dcc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":9,"uuid":"e2c6da1d-87c9-4ebf-8234-4356adbb2bb5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-24T18:00:44.480 DEBUG:teuthology.misc:0 of 10 OSDs are up 2026-02-24T18:00:44.534 INFO:tasks.ceph.osd.0.trial068.stderr:2026-02-24T18:00:44.529+0000 7f75bc5baa40 -1 osd.0 0 log_to_monitors true 2026-02-24T18:00:44.546 INFO:tasks.ceph.osd.1.trial068.stderr:2026-02-24T18:00:44.541+0000 7f54b2b22a40 -1 osd.1 0 log_to_monitors true 2026-02-24T18:00:44.562 INFO:tasks.ceph.osd.8.trial068.stderr:2026-02-24T18:00:44.557+0000 7f75cab7da40 -1 osd.8 0 log_to_monitors true 2026-02-24T18:00:44.569 INFO:tasks.ceph.osd.9.trial068.stderr:2026-02-24T18:00:44.561+0000 7f067a7a9a40 -1 osd.9 0 log_to_monitors true 2026-02-24T18:00:44.571 INFO:tasks.ceph.osd.5.trial068.stderr:2026-02-24T18:00:44.565+0000 7f1724265a40 -1 osd.5 0 log_to_monitors true 2026-02-24T18:00:44.575 INFO:tasks.ceph.osd.3.trial068.stderr:2026-02-24T18:00:44.569+0000 7fd408058a40 -1 osd.3 0 log_to_monitors true 2026-02-24T18:00:44.581 INFO:tasks.ceph.osd.2.trial068.stderr:2026-02-24T18:00:44.577+0000 7fc3c0063a40 -1 osd.2 0 log_to_monitors true 2026-02-24T18:00:44.582 INFO:tasks.ceph.osd.6.trial068.stderr:2026-02-24T18:00:44.577+0000 7fef31a3ca40 -1 osd.6 0 log_to_monitors true 2026-02-24T18:00:44.588 INFO:tasks.ceph.osd.4.trial068.stderr:2026-02-24T18:00:44.581+0000 7fcc5ae51a40 -1 osd.4 0 log_to_monitors true 2026-02-24T18:00:44.664 INFO:tasks.ceph.osd.7.trial068.stderr:2026-02-24T18:00:44.657+0000 7fb80bf1fa40 -1 osd.7 0 log_to_monitors true 2026-02-24T18:00:46.473 INFO:tasks.ceph.osd.0.trial068.stderr:2026-02-24T18:00:46.465+0000 7f75b8564640 -1 osd.0 0 waiting for initial osdmap 2026-02-24T18:00:46.473 INFO:tasks.ceph.osd.1.trial068.stderr:2026-02-24T18:00:46.469+0000 7f54aeacc640 -1 osd.1 0 waiting for initial osdmap 2026-02-24T18:00:46.474 INFO:tasks.ceph.osd.8.trial068.stderr:2026-02-24T18:00:46.469+0000 7f75c6b27640 -1 osd.8 0 waiting for initial osdmap 2026-02-24T18:00:46.474 INFO:tasks.ceph.osd.9.trial068.stderr:2026-02-24T18:00:46.469+0000 7f0676753640 -1 osd.9 0 waiting for initial osdmap 2026-02-24T18:00:46.474 INFO:tasks.ceph.osd.5.trial068.stderr:2026-02-24T18:00:46.469+0000 7f172020f640 -1 osd.5 0 waiting for initial osdmap 2026-02-24T18:00:46.474 INFO:tasks.ceph.osd.3.trial068.stderr:2026-02-24T18:00:46.469+0000 7fd404002640 -1 osd.3 0 waiting for initial osdmap 2026-02-24T18:00:46.475 INFO:tasks.ceph.osd.6.trial068.stderr:2026-02-24T18:00:46.469+0000 7fef2d9e6640 -1 osd.6 0 waiting for initial osdmap 2026-02-24T18:00:46.475 INFO:tasks.ceph.osd.2.trial068.stderr:2026-02-24T18:00:46.469+0000 7fc3bc00d640 -1 osd.2 0 waiting for initial osdmap 2026-02-24T18:00:46.475 INFO:tasks.ceph.osd.4.trial068.stderr:2026-02-24T18:00:46.469+0000 7fcc56dfb640 -1 osd.4 0 waiting for initial osdmap 2026-02-24T18:00:46.475 INFO:tasks.ceph.osd.7.trial068.stderr:2026-02-24T18:00:46.469+0000 7fb807ec9640 -1 osd.7 0 waiting for initial osdmap 2026-02-24T18:00:46.476 INFO:tasks.ceph.osd.0.trial068.stderr:2026-02-24T18:00:46.469+0000 7f75afc33640 -1 osd.0 14 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2026-02-24T18:00:46.477 INFO:tasks.ceph.osd.1.trial068.stderr:2026-02-24T18:00:46.469+0000 7f54aa1a3640 -1 osd.1 14 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2026-02-24T18:00:46.477 INFO:tasks.ceph.osd.8.trial068.stderr:2026-02-24T18:00:46.473+0000 7f75be1f6640 -1 osd.8 14 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2026-02-24T18:00:46.478 INFO:tasks.ceph.osd.3.trial068.stderr:2026-02-24T18:00:46.473+0000 7fd3fb6d1640 -1 osd.3 14 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2026-02-24T18:00:46.478 INFO:tasks.ceph.osd.9.trial068.stderr:2026-02-24T18:00:46.473+0000 7f0671e2a640 -1 osd.9 14 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2026-02-24T18:00:46.478 INFO:tasks.ceph.osd.2.trial068.stderr:2026-02-24T18:00:46.473+0000 7fc3b76e4640 -1 osd.2 14 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2026-02-24T18:00:46.478 INFO:tasks.ceph.osd.5.trial068.stderr:2026-02-24T18:00:46.473+0000 7f17178de640 -1 osd.5 14 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2026-02-24T18:00:46.478 INFO:tasks.ceph.osd.6.trial068.stderr:2026-02-24T18:00:46.473+0000 7fef250b5640 -1 osd.6 14 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2026-02-24T18:00:46.479 INFO:tasks.ceph.osd.4.trial068.stderr:2026-02-24T18:00:46.473+0000 7fcc4e4ca640 -1 osd.4 14 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2026-02-24T18:00:46.479 INFO:tasks.ceph.osd.7.trial068.stderr:2026-02-24T18:00:46.473+0000 7fb8035a0640 -1 osd.7 14 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2026-02-24T18:00:51.486 DEBUG:teuthology.orchestra.run.trial068:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2026-02-24T18:00:51.634 INFO:teuthology.misc.health.trial068.stdout: 2026-02-24T18:00:51.634 INFO:teuthology.misc.health.trial068.stdout:{"epoch":16,"fsid":"85c6d86c-aad0-4a30-9c50-f24175e8fcf5","created":"2026-02-24T18:00:41.320093+0000","modified":"2026-02-24T18:00:48.469777+0000","last_up_change":"2026-02-24T18:00:47.465901+0000","last_in_change":"2026-02-24T18:00:43.208336+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":10,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"0685fb8e-c8df-4155-aa67-77c8085c54fa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6826","nonce":4252813754},{"type":"v1","addr":"10.20.193.68:6828","nonce":4252813754}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6831","nonce":4252813754},{"type":"v1","addr":"10.20.193.68:6833","nonce":4252813754}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6839","nonce":4252813754},{"type":"v1","addr":"10.20.193.68:6841","nonce":4252813754}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6835","nonce":4252813754},{"type":"v1","addr":"10.20.193.68:6837","nonce":4252813754}]},"public_addr":"10.20.193.68:6828/4252813754","cluster_addr":"10.20.193.68:6833/4252813754","heartbeat_back_addr":"10.20.193.68:6841/4252813754","heartbeat_front_addr":"10.20.193.68:6837/4252813754","state":["exists","up"]},{"osd":1,"uuid":"e6c8aa5f-faf0-46d2-8185-92c1037749e2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":1878455324},{"type":"v1","addr":"10.20.193.68:6811","nonce":1878455324}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":1878455324},{"type":"v1","addr":"10.20.193.68:6813","nonce":1878455324}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":1878455324},{"type":"v1","addr":"10.20.193.68:6817","nonce":1878455324}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":1878455324},{"type":"v1","addr":"10.20.193.68:6815","nonce":1878455324}]},"public_addr":"10.20.193.68:6811/1878455324","cluster_addr":"10.20.193.68:6813/1878455324","heartbeat_back_addr":"10.20.193.68:6817/1878455324","heartbeat_front_addr":"10.20.193.68:6815/1878455324","state":["exists","up"]},{"osd":2,"uuid":"9de05692-da99-43ea-8eeb-7388526b5abc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6858","nonce":3653189349},{"type":"v1","addr":"10.20.193.68:6859","nonce":3653189349}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6860","nonce":3653189349},{"type":"v1","addr":"10.20.193.68:6861","nonce":3653189349}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6864","nonce":3653189349},{"type":"v1","addr":"10.20.193.68:6866","nonce":3653189349}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6862","nonce":3653189349},{"type":"v1","addr":"10.20.193.68:6863","nonce":3653189349}]},"public_addr":"10.20.193.68:6859/3653189349","cluster_addr":"10.20.193.68:6861/3653189349","heartbeat_back_addr":"10.20.193.68:6866/3653189349","heartbeat_front_addr":"10.20.193.68:6863/3653189349","state":["exists","up"]},{"osd":3,"uuid":"e221ca44-382e-4924-ae30-d1add04b6f0d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6865","nonce":1980192188},{"type":"v1","addr":"10.20.193.68:6867","nonce":1980192188}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6868","nonce":1980192188},{"type":"v1","addr":"10.20.193.68:6869","nonce":1980192188}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6872","nonce":1980192188},{"type":"v1","addr":"10.20.193.68:6873","nonce":1980192188}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6870","nonce":1980192188},{"type":"v1","addr":"10.20.193.68:6871","nonce":1980192188}]},"public_addr":"10.20.193.68:6867/1980192188","cluster_addr":"10.20.193.68:6869/1980192188","heartbeat_back_addr":"10.20.193.68:6873/1980192188","heartbeat_front_addr":"10.20.193.68:6871/1980192188","state":["exists","up"]},{"osd":4,"uuid":"59a23325-5017-4345-ad4b-6cde76f1f74d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6842","nonce":1162412721},{"type":"v1","addr":"10.20.193.68:6843","nonce":1162412721}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6844","nonce":1162412721},{"type":"v1","addr":"10.20.193.68:6845","nonce":1162412721}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6848","nonce":1162412721},{"type":"v1","addr":"10.20.193.68:6849","nonce":1162412721}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6846","nonce":1162412721},{"type":"v1","addr":"10.20.193.68:6847","nonce":1162412721}]},"public_addr":"10.20.193.68:6843/1162412721","cluster_addr":"10.20.193.68:6845/1162412721","heartbeat_back_addr":"10.20.193.68:6849/1162412721","heartbeat_front_addr":"10.20.193.68:6847/1162412721","state":["exists","up"]},{"osd":5,"uuid":"0d906a72-c6d6-4072-b5fb-383a18f376e4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6827","nonce":2140276670},{"type":"v1","addr":"10.20.193.68:6829","nonce":2140276670}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6830","nonce":2140276670},{"type":"v1","addr":"10.20.193.68:6832","nonce":2140276670}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6838","nonce":2140276670},{"type":"v1","addr":"10.20.193.68:6840","nonce":2140276670}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6834","nonce":2140276670},{"type":"v1","addr":"10.20.193.68:6836","nonce":2140276670}]},"public_addr":"10.20.193.68:6829/2140276670","cluster_addr":"10.20.193.68:6832/2140276670","heartbeat_back_addr":"10.20.193.68:6840/2140276670","heartbeat_front_addr":"10.20.193.68:6836/2140276670","state":["exists","up"]},{"osd":6,"uuid":"89445f92-371d-492c-bf3f-814aa6533ad0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6850","nonce":2652457072},{"type":"v1","addr":"10.20.193.68:6851","nonce":2652457072}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6852","nonce":2652457072},{"type":"v1","addr":"10.20.193.68:6853","nonce":2652457072}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6856","nonce":2652457072},{"type":"v1","addr":"10.20.193.68:6857","nonce":2652457072}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6854","nonce":2652457072},{"type":"v1","addr":"10.20.193.68:6855","nonce":2652457072}]},"public_addr":"10.20.193.68:6851/2652457072","cluster_addr":"10.20.193.68:6853/2652457072","heartbeat_back_addr":"10.20.193.68:6857/2652457072","heartbeat_front_addr":"10.20.193.68:6855/2652457072","state":["exists","up"]},{"osd":7,"uuid":"3a2e9d71-2ddc-4b3b-8417-e70a1794c3c2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6874","nonce":1182331931},{"type":"v1","addr":"10.20.193.68:6875","nonce":1182331931}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6876","nonce":1182331931},{"type":"v1","addr":"10.20.193.68:6877","nonce":1182331931}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6880","nonce":1182331931},{"type":"v1","addr":"10.20.193.68:6881","nonce":1182331931}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6878","nonce":1182331931},{"type":"v1","addr":"10.20.193.68:6879","nonce":1182331931}]},"public_addr":"10.20.193.68:6875/1182331931","cluster_addr":"10.20.193.68:6877/1182331931","heartbeat_back_addr":"10.20.193.68:6881/1182331931","heartbeat_front_addr":"10.20.193.68:6879/1182331931","state":["exists","up"]},{"osd":8,"uuid":"98acdd38-bad4-49c7-a681-772ff7e34dcc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3946693881},{"type":"v1","addr":"10.20.193.68:6803","nonce":3946693881}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3946693881},{"type":"v1","addr":"10.20.193.68:6805","nonce":3946693881}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3946693881},{"type":"v1","addr":"10.20.193.68:6809","nonce":3946693881}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3946693881},{"type":"v1","addr":"10.20.193.68:6807","nonce":3946693881}]},"public_addr":"10.20.193.68:6803/3946693881","cluster_addr":"10.20.193.68:6805/3946693881","heartbeat_back_addr":"10.20.193.68:6809/3946693881","heartbeat_front_addr":"10.20.193.68:6807/3946693881","state":["exists","up"]},{"osd":9,"uuid":"e2c6da1d-87c9-4ebf-8234-4356adbb2bb5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6818","nonce":3148430907},{"type":"v1","addr":"10.20.193.68:6819","nonce":3148430907}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6820","nonce":3148430907},{"type":"v1","addr":"10.20.193.68:6821","nonce":3148430907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6824","nonce":3148430907},{"type":"v1","addr":"10.20.193.68:6825","nonce":3148430907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6822","nonce":3148430907},{"type":"v1","addr":"10.20.193.68:6823","nonce":3148430907}]},"public_addr":"10.20.193.68:6819/3148430907","cluster_addr":"10.20.193.68:6821/3148430907","heartbeat_back_addr":"10.20.193.68:6825/3148430907","heartbeat_front_addr":"10.20.193.68:6823/3148430907","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-24T18:00:45.570554+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-24T18:00:45.565655+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-24T18:00:45.572517+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-24T18:00:45.573166+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-24T18:00:45.595183+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-24T18:00:45.560129+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-24T18:00:45.591572+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-24T18:00:45.639584+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-24T18:00:45.530957+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-24T18:00:45.534329+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-24T18:00:51.643 DEBUG:teuthology.misc:10 of 10 OSDs are up 2026-02-24T18:00:51.643 INFO:tasks.ceph:Creating RBD pool 2026-02-24T18:00:51.643 DEBUG:teuthology.orchestra.run.trial068:> sudo ceph --cluster ceph osd pool create rbd 8 2026-02-24T18:00:52.480 INFO:teuthology.orchestra.run.trial068.stderr:pool 'rbd' created 2026-02-24T18:00:52.494 DEBUG:teuthology.orchestra.run.trial068:> rbd --cluster ceph pool init rbd 2026-02-24T18:00:55.495 INFO:tasks.ceph:Starting mds daemons in cluster ceph... 2026-02-24T18:00:55.495 DEBUG:teuthology.orchestra.run.trial068:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config log 1 --format=json 2026-02-24T18:00:55.495 INFO:tasks.daemonwatchdog.daemon_watchdog:watchdog starting 2026-02-24T18:00:55.660 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-24T18:00:55.671 INFO:teuthology.orchestra.run.trial068.stdout:[{"version":2,"timestamp":"2026-02-24T18:00:41.361087+0000","name":"","changes":[{"name":"mgr/mgr_pool","new_value":"false"}]}] 2026-02-24T18:00:55.671 INFO:tasks.ceph_manager:config epoch is 2 2026-02-24T18:00:55.671 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2026-02-24T18:00:55.671 INFO:tasks.ceph.ceph_manager.ceph:waiting for mgr available 2026-02-24T18:00:55.671 DEBUG:teuthology.orchestra.run.trial068:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr dump --format=json 2026-02-24T18:00:55.857 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-24T18:00:55.867 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":4,"flags":0,"active_gid":4101,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6800","nonce":226635827},{"type":"v1","addr":"10.20.193.68:6801","nonce":226635827}]},"active_addr":"10.20.193.68:6801/226635827","active_change":"2026-02-24T18:00:43.455707+0000","active_mgr_features":4544132024016699391,"available":true,"standbys":[],"modules":["iostat","nfs"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to, use commas to separate multiple","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"update_pg_upmap_activity":{"name":"update_pg_upmap_activity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Updates pg_upmap activity stats to be used in `balancer status detail`","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often the agent on each host will gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"The first TCP port the agent will try to bind to. Up to 1000 subsequent ports will be attempted if this port cannot be bound","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this option can facilitate debugging daemons that encounter runtime problems.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"How frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"Fraction of total system memory to divide among autotuned daemons. Converged systems hosting both Ceph and compute should set a lower value; systems with more than 10 GiB of memory per OSD may benefit from a slightly larger value so that the abundance of RAM can be fully utilized","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command persistent logging","long_desc":"","tags":[],"see_also":[]},"certificate_automated_rotation_enabled":{"name":"certificate_automated_rotation_enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"This flag controls whether cephadm automatically rotates certificates upon expiration.","long_desc":"","tags":[],"see_also":[]},"certificate_check_debug_mode":{"name":"certificate_check_debug_mode","type":"bool","level":"dev","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"FOR TESTING ONLY: This flag forces the certificate check instead of waiting for certificate_check_period.","long_desc":"","tags":[],"see_also":[]},"certificate_check_period":{"name":"certificate_check_period","type":"int","level":"advanced","flags":0,"default_value":"1","min":"0","max":"30","enum_allowed":[],"desc":"Specifies how often (in days) the certificate should be checked for validity.","long_desc":"","tags":[],"see_also":[]},"certificate_duration_days":{"name":"certificate_duration_days","type":"int","level":"advanced","flags":0,"default_value":"1095","min":"90","max":"3650","enum_allowed":[],"desc":"Specifies the duration of self-signed certificates generated and signed by the cephadm root CA","long_desc":"","tags":[],"see_also":[]},"certificate_renewal_threshold_days":{"name":"certificate_renewal_threshold_days","type":"int","level":"advanced","flags":0,"default_value":"30","min":"10","max":"90","enum_allowed":[],"desc":"Specifies the lead time in days to initiate certificate renewal before expiration.","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently Podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.28.1","min":"","max":"","enum_allowed":[],"desc":"Alertmanager container image","long_desc":"","tags":[],"see_also":[]},"container_image_alloy":{"name":"container_image_alloy","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/alloy:latest","min":"","max":"","enum_allowed":[],"desc":"Alloy container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"Elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:12.3.1","min":"","max":"","enum_allowed":[],"desc":"Grafana container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"Haproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_nginx":{"name":"container_image_nginx","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nginx:sclorg-nginx-126","min":"","max":"","enum_allowed":[],"desc":"Nginx container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.9.1","min":"","max":"","enum_allowed":[],"desc":"Node exporter container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.6","min":"","max":"","enum_allowed":[],"desc":"Nvmeof container image","long_desc":"","tags":[],"see_also":[]},"container_image_oauth2_proxy":{"name":"container_image_oauth2_proxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/oauth2-proxy/oauth2-proxy:v7.6.0","min":"","max":"","enum_allowed":[],"desc":"Oauth2 proxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v3.6.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba":{"name":"container_image_samba","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-server:devbuilds-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba_metrics":{"name":"container_image_samba_metrics","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-metrics:devbuilds-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba metrics container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"Snmp gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"Seconds for which to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"quay.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"Seconds for which to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"Seconds for which to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"Location of dashboards to include in Grafana","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"How frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether the hardware monitoring daemon be deployed on every host?","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at the debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"Max number of daemons per service per host: guard against mistakes and runaways","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"Max number of OSDs that will be drained simultaneously when OSDs are removed","long_desc":"","tags":[],"see_also":[]},"max_parallel_osd_upgrades":{"name":"max_parallel_osd_upgrades","type":"int","level":"advanced","flags":0,"default_value":"16","min":"","max":"","enum_allowed":[],"desc":"Maximum number of OSD daemons upgraded in parallel.","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"Mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default IP address for RedFish API (OOB management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"Location of alerts to include in Prometheus Alertmanager","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry URL for login purposes. This is not the default registry.","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"Cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Customized SSH config file to distribute to managed hosts","long_desc":"","tags":[],"see_also":[]},"ssh_keepalive_count_max":{"name":"ssh_keepalive_count_max","type":"int","level":"advanced","flags":0,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"How many sequential SSH connection liveness check failures before the host is marked offline","long_desc":"","tags":[],"see_also":[]},"ssh_keepalive_interval":{"name":"ssh_keepalive_interval","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"How often SSH connections are checked for liveness","long_desc":"","tags":[],"see_also":[]},"stray_daemon_check_interval":{"name":"stray_daemon_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"How frequently cephadm should check for the presence of stray daemons","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use the cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest to ensure that all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MANAGED_BY_CLUSTERS":{"name":"MANAGED_BY_CLUSTERS","type":"str","level":"advanced","flags":0,"default_value":"[]","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MULTICLUSTER_CONFIG":{"name":"MULTICLUSTER_CONFIG","type":"str","level":"advanced","flags":0,"default_value":"{}","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROM_ALERT_CREDENTIAL_CACHE_TTL":{"name":"PROM_ALERT_CREDENTIAL_CACHE_TTL","type":"int","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_HOSTNAME_PER_DAEMON":{"name":"RGW_HOSTNAME_PER_DAEMON","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crypto_caller":{"name":"crypto_caller","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sso_oauth2":{"name":"sso_oauth2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"healthcheck_history_max_entries":{"name":"healthcheck_history_max_entries","type":"int","level":"advanced","flags":1,"default_value":"1000","min":"","max":"","enum_allowed":[],"desc":"Maximum number of health check history entries to keep","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"secondary_zone_period_retry_limit":{"name":"secondary_zone_period_retry_limit","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"RGW module period update retry limit for secondary site","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"smb","can_run":true,"error_string":"","module_options":{"internal_store_backend":{"name":"internal_store_backend","type":"str","level":"dev","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"set internal store backend. for develoment and testing only","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"update_orchestration":{"name":"update_orchestration","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically update orchestration when smb resources are changed","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"pause_cloning":{"name":"pause_cloning","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Pause asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"pause_purging":{"name":"pause_purging","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Pause asynchronous subvolume purge threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"squid":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"tentacle":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"force_disabled_modules":{},"last_failure_osd_epoch":0,"active_clients":[{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"10.20.193.68:0","nonce":3972216823}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"10.20.193.68:0","nonce":1773723832}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"10.20.193.68:0","nonce":3271014110}]}]} 2026-02-24T18:00:55.868 INFO:tasks.ceph.ceph_manager.ceph:mgr available! 2026-02-24T18:00:55.868 INFO:tasks.ceph.ceph_manager.ceph:waiting for all up 2026-02-24T18:00:55.868 DEBUG:teuthology.orchestra.run.trial068:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2026-02-24T18:00:56.033 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-24T18:00:56.033 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":20,"fsid":"85c6d86c-aad0-4a30-9c50-f24175e8fcf5","created":"2026-02-24T18:00:41.320093+0000","modified":"2026-02-24T18:00:55.481181+0000","last_up_change":"2026-02-24T18:00:47.465901+0000","last_in_change":"2026-02-24T18:00:43.208336+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":10,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":false,"pools":[{"pool":1,"pool_name":"rbd","create_time":"2026-02-24T18:00:51.802289+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":2,"snap_epoch":20,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.5,"score_stable":2.5,"optimal_score":0.89999997615814209,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1.0000001192092896,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"0685fb8e-c8df-4155-aa67-77c8085c54fa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":17,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6826","nonce":4252813754},{"type":"v1","addr":"10.20.193.68:6828","nonce":4252813754}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6831","nonce":4252813754},{"type":"v1","addr":"10.20.193.68:6833","nonce":4252813754}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6839","nonce":4252813754},{"type":"v1","addr":"10.20.193.68:6841","nonce":4252813754}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6835","nonce":4252813754},{"type":"v1","addr":"10.20.193.68:6837","nonce":4252813754}]},"public_addr":"10.20.193.68:6828/4252813754","cluster_addr":"10.20.193.68:6833/4252813754","heartbeat_back_addr":"10.20.193.68:6841/4252813754","heartbeat_front_addr":"10.20.193.68:6837/4252813754","state":["exists","up"]},{"osd":1,"uuid":"e6c8aa5f-faf0-46d2-8185-92c1037749e2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":1878455324},{"type":"v1","addr":"10.20.193.68:6811","nonce":1878455324}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":1878455324},{"type":"v1","addr":"10.20.193.68:6813","nonce":1878455324}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":1878455324},{"type":"v1","addr":"10.20.193.68:6817","nonce":1878455324}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":1878455324},{"type":"v1","addr":"10.20.193.68:6815","nonce":1878455324}]},"public_addr":"10.20.193.68:6811/1878455324","cluster_addr":"10.20.193.68:6813/1878455324","heartbeat_back_addr":"10.20.193.68:6817/1878455324","heartbeat_front_addr":"10.20.193.68:6815/1878455324","state":["exists","up"]},{"osd":2,"uuid":"9de05692-da99-43ea-8eeb-7388526b5abc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6858","nonce":3653189349},{"type":"v1","addr":"10.20.193.68:6859","nonce":3653189349}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6860","nonce":3653189349},{"type":"v1","addr":"10.20.193.68:6861","nonce":3653189349}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6864","nonce":3653189349},{"type":"v1","addr":"10.20.193.68:6866","nonce":3653189349}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6862","nonce":3653189349},{"type":"v1","addr":"10.20.193.68:6863","nonce":3653189349}]},"public_addr":"10.20.193.68:6859/3653189349","cluster_addr":"10.20.193.68:6861/3653189349","heartbeat_back_addr":"10.20.193.68:6866/3653189349","heartbeat_front_addr":"10.20.193.68:6863/3653189349","state":["exists","up"]},{"osd":3,"uuid":"e221ca44-382e-4924-ae30-d1add04b6f0d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":17,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6865","nonce":1980192188},{"type":"v1","addr":"10.20.193.68:6867","nonce":1980192188}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6868","nonce":1980192188},{"type":"v1","addr":"10.20.193.68:6869","nonce":1980192188}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6872","nonce":1980192188},{"type":"v1","addr":"10.20.193.68:6873","nonce":1980192188}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6870","nonce":1980192188},{"type":"v1","addr":"10.20.193.68:6871","nonce":1980192188}]},"public_addr":"10.20.193.68:6867/1980192188","cluster_addr":"10.20.193.68:6869/1980192188","heartbeat_back_addr":"10.20.193.68:6873/1980192188","heartbeat_front_addr":"10.20.193.68:6871/1980192188","state":["exists","up"]},{"osd":4,"uuid":"59a23325-5017-4345-ad4b-6cde76f1f74d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":17,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6842","nonce":1162412721},{"type":"v1","addr":"10.20.193.68:6843","nonce":1162412721}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6844","nonce":1162412721},{"type":"v1","addr":"10.20.193.68:6845","nonce":1162412721}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6848","nonce":1162412721},{"type":"v1","addr":"10.20.193.68:6849","nonce":1162412721}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6846","nonce":1162412721},{"type":"v1","addr":"10.20.193.68:6847","nonce":1162412721}]},"public_addr":"10.20.193.68:6843/1162412721","cluster_addr":"10.20.193.68:6845/1162412721","heartbeat_back_addr":"10.20.193.68:6849/1162412721","heartbeat_front_addr":"10.20.193.68:6847/1162412721","state":["exists","up"]},{"osd":5,"uuid":"0d906a72-c6d6-4072-b5fb-383a18f376e4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":17,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6827","nonce":2140276670},{"type":"v1","addr":"10.20.193.68:6829","nonce":2140276670}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6830","nonce":2140276670},{"type":"v1","addr":"10.20.193.68:6832","nonce":2140276670}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6838","nonce":2140276670},{"type":"v1","addr":"10.20.193.68:6840","nonce":2140276670}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6834","nonce":2140276670},{"type":"v1","addr":"10.20.193.68:6836","nonce":2140276670}]},"public_addr":"10.20.193.68:6829/2140276670","cluster_addr":"10.20.193.68:6832/2140276670","heartbeat_back_addr":"10.20.193.68:6840/2140276670","heartbeat_front_addr":"10.20.193.68:6836/2140276670","state":["exists","up"]},{"osd":6,"uuid":"89445f92-371d-492c-bf3f-814aa6533ad0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":17,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6850","nonce":2652457072},{"type":"v1","addr":"10.20.193.68:6851","nonce":2652457072}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6852","nonce":2652457072},{"type":"v1","addr":"10.20.193.68:6853","nonce":2652457072}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6856","nonce":2652457072},{"type":"v1","addr":"10.20.193.68:6857","nonce":2652457072}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6854","nonce":2652457072},{"type":"v1","addr":"10.20.193.68:6855","nonce":2652457072}]},"public_addr":"10.20.193.68:6851/2652457072","cluster_addr":"10.20.193.68:6853/2652457072","heartbeat_back_addr":"10.20.193.68:6857/2652457072","heartbeat_front_addr":"10.20.193.68:6855/2652457072","state":["exists","up"]},{"osd":7,"uuid":"3a2e9d71-2ddc-4b3b-8417-e70a1794c3c2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6874","nonce":1182331931},{"type":"v1","addr":"10.20.193.68:6875","nonce":1182331931}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6876","nonce":1182331931},{"type":"v1","addr":"10.20.193.68:6877","nonce":1182331931}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6880","nonce":1182331931},{"type":"v1","addr":"10.20.193.68:6881","nonce":1182331931}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6878","nonce":1182331931},{"type":"v1","addr":"10.20.193.68:6879","nonce":1182331931}]},"public_addr":"10.20.193.68:6875/1182331931","cluster_addr":"10.20.193.68:6877/1182331931","heartbeat_back_addr":"10.20.193.68:6881/1182331931","heartbeat_front_addr":"10.20.193.68:6879/1182331931","state":["exists","up"]},{"osd":8,"uuid":"98acdd38-bad4-49c7-a681-772ff7e34dcc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":17,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3946693881},{"type":"v1","addr":"10.20.193.68:6803","nonce":3946693881}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3946693881},{"type":"v1","addr":"10.20.193.68:6805","nonce":3946693881}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3946693881},{"type":"v1","addr":"10.20.193.68:6809","nonce":3946693881}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3946693881},{"type":"v1","addr":"10.20.193.68:6807","nonce":3946693881}]},"public_addr":"10.20.193.68:6803/3946693881","cluster_addr":"10.20.193.68:6805/3946693881","heartbeat_back_addr":"10.20.193.68:6809/3946693881","heartbeat_front_addr":"10.20.193.68:6807/3946693881","state":["exists","up"]},{"osd":9,"uuid":"e2c6da1d-87c9-4ebf-8234-4356adbb2bb5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6818","nonce":3148430907},{"type":"v1","addr":"10.20.193.68:6819","nonce":3148430907}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6820","nonce":3148430907},{"type":"v1","addr":"10.20.193.68:6821","nonce":3148430907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6824","nonce":3148430907},{"type":"v1","addr":"10.20.193.68:6825","nonce":3148430907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6822","nonce":3148430907},{"type":"v1","addr":"10.20.193.68:6823","nonce":3148430907}]},"public_addr":"10.20.193.68:6819/3148430907","cluster_addr":"10.20.193.68:6821/3148430907","heartbeat_back_addr":"10.20.193.68:6825/3148430907","heartbeat_front_addr":"10.20.193.68:6823/3148430907","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-24T18:00:45.570554+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-24T18:00:45.565655+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-24T18:00:45.572517+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-24T18:00:45.573166+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-24T18:00:45.595183+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-24T18:00:45.560129+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-24T18:00:45.591572+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-24T18:00:45.639584+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-24T18:00:45.530957+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-24T18:00:45.534329+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":1,"snaps":[{"begin":2,"length":1}]}],"new_removed_snaps":[{"pool":1,"snaps":[{"begin":2,"length":1}]}],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-24T18:00:56.044 INFO:tasks.ceph.ceph_manager.ceph:all up! 2026-02-24T18:00:56.044 DEBUG:teuthology.orchestra.run.trial068:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2026-02-24T18:00:56.204 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-24T18:00:56.204 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":20,"fsid":"85c6d86c-aad0-4a30-9c50-f24175e8fcf5","created":"2026-02-24T18:00:41.320093+0000","modified":"2026-02-24T18:00:55.481181+0000","last_up_change":"2026-02-24T18:00:47.465901+0000","last_in_change":"2026-02-24T18:00:43.208336+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":10,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":false,"pools":[{"pool":1,"pool_name":"rbd","create_time":"2026-02-24T18:00:51.802289+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":2,"snap_epoch":20,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.5,"score_stable":2.5,"optimal_score":0.89999997615814209,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1.0000001192092896,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"0685fb8e-c8df-4155-aa67-77c8085c54fa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":17,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6826","nonce":4252813754},{"type":"v1","addr":"10.20.193.68:6828","nonce":4252813754}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6831","nonce":4252813754},{"type":"v1","addr":"10.20.193.68:6833","nonce":4252813754}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6839","nonce":4252813754},{"type":"v1","addr":"10.20.193.68:6841","nonce":4252813754}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6835","nonce":4252813754},{"type":"v1","addr":"10.20.193.68:6837","nonce":4252813754}]},"public_addr":"10.20.193.68:6828/4252813754","cluster_addr":"10.20.193.68:6833/4252813754","heartbeat_back_addr":"10.20.193.68:6841/4252813754","heartbeat_front_addr":"10.20.193.68:6837/4252813754","state":["exists","up"]},{"osd":1,"uuid":"e6c8aa5f-faf0-46d2-8185-92c1037749e2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":1878455324},{"type":"v1","addr":"10.20.193.68:6811","nonce":1878455324}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":1878455324},{"type":"v1","addr":"10.20.193.68:6813","nonce":1878455324}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":1878455324},{"type":"v1","addr":"10.20.193.68:6817","nonce":1878455324}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":1878455324},{"type":"v1","addr":"10.20.193.68:6815","nonce":1878455324}]},"public_addr":"10.20.193.68:6811/1878455324","cluster_addr":"10.20.193.68:6813/1878455324","heartbeat_back_addr":"10.20.193.68:6817/1878455324","heartbeat_front_addr":"10.20.193.68:6815/1878455324","state":["exists","up"]},{"osd":2,"uuid":"9de05692-da99-43ea-8eeb-7388526b5abc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6858","nonce":3653189349},{"type":"v1","addr":"10.20.193.68:6859","nonce":3653189349}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6860","nonce":3653189349},{"type":"v1","addr":"10.20.193.68:6861","nonce":3653189349}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6864","nonce":3653189349},{"type":"v1","addr":"10.20.193.68:6866","nonce":3653189349}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6862","nonce":3653189349},{"type":"v1","addr":"10.20.193.68:6863","nonce":3653189349}]},"public_addr":"10.20.193.68:6859/3653189349","cluster_addr":"10.20.193.68:6861/3653189349","heartbeat_back_addr":"10.20.193.68:6866/3653189349","heartbeat_front_addr":"10.20.193.68:6863/3653189349","state":["exists","up"]},{"osd":3,"uuid":"e221ca44-382e-4924-ae30-d1add04b6f0d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":17,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6865","nonce":1980192188},{"type":"v1","addr":"10.20.193.68:6867","nonce":1980192188}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6868","nonce":1980192188},{"type":"v1","addr":"10.20.193.68:6869","nonce":1980192188}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6872","nonce":1980192188},{"type":"v1","addr":"10.20.193.68:6873","nonce":1980192188}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6870","nonce":1980192188},{"type":"v1","addr":"10.20.193.68:6871","nonce":1980192188}]},"public_addr":"10.20.193.68:6867/1980192188","cluster_addr":"10.20.193.68:6869/1980192188","heartbeat_back_addr":"10.20.193.68:6873/1980192188","heartbeat_front_addr":"10.20.193.68:6871/1980192188","state":["exists","up"]},{"osd":4,"uuid":"59a23325-5017-4345-ad4b-6cde76f1f74d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":17,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6842","nonce":1162412721},{"type":"v1","addr":"10.20.193.68:6843","nonce":1162412721}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6844","nonce":1162412721},{"type":"v1","addr":"10.20.193.68:6845","nonce":1162412721}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6848","nonce":1162412721},{"type":"v1","addr":"10.20.193.68:6849","nonce":1162412721}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6846","nonce":1162412721},{"type":"v1","addr":"10.20.193.68:6847","nonce":1162412721}]},"public_addr":"10.20.193.68:6843/1162412721","cluster_addr":"10.20.193.68:6845/1162412721","heartbeat_back_addr":"10.20.193.68:6849/1162412721","heartbeat_front_addr":"10.20.193.68:6847/1162412721","state":["exists","up"]},{"osd":5,"uuid":"0d906a72-c6d6-4072-b5fb-383a18f376e4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":17,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6827","nonce":2140276670},{"type":"v1","addr":"10.20.193.68:6829","nonce":2140276670}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6830","nonce":2140276670},{"type":"v1","addr":"10.20.193.68:6832","nonce":2140276670}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6838","nonce":2140276670},{"type":"v1","addr":"10.20.193.68:6840","nonce":2140276670}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6834","nonce":2140276670},{"type":"v1","addr":"10.20.193.68:6836","nonce":2140276670}]},"public_addr":"10.20.193.68:6829/2140276670","cluster_addr":"10.20.193.68:6832/2140276670","heartbeat_back_addr":"10.20.193.68:6840/2140276670","heartbeat_front_addr":"10.20.193.68:6836/2140276670","state":["exists","up"]},{"osd":6,"uuid":"89445f92-371d-492c-bf3f-814aa6533ad0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":17,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6850","nonce":2652457072},{"type":"v1","addr":"10.20.193.68:6851","nonce":2652457072}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6852","nonce":2652457072},{"type":"v1","addr":"10.20.193.68:6853","nonce":2652457072}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6856","nonce":2652457072},{"type":"v1","addr":"10.20.193.68:6857","nonce":2652457072}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6854","nonce":2652457072},{"type":"v1","addr":"10.20.193.68:6855","nonce":2652457072}]},"public_addr":"10.20.193.68:6851/2652457072","cluster_addr":"10.20.193.68:6853/2652457072","heartbeat_back_addr":"10.20.193.68:6857/2652457072","heartbeat_front_addr":"10.20.193.68:6855/2652457072","state":["exists","up"]},{"osd":7,"uuid":"3a2e9d71-2ddc-4b3b-8417-e70a1794c3c2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6874","nonce":1182331931},{"type":"v1","addr":"10.20.193.68:6875","nonce":1182331931}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6876","nonce":1182331931},{"type":"v1","addr":"10.20.193.68:6877","nonce":1182331931}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6880","nonce":1182331931},{"type":"v1","addr":"10.20.193.68:6881","nonce":1182331931}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6878","nonce":1182331931},{"type":"v1","addr":"10.20.193.68:6879","nonce":1182331931}]},"public_addr":"10.20.193.68:6875/1182331931","cluster_addr":"10.20.193.68:6877/1182331931","heartbeat_back_addr":"10.20.193.68:6881/1182331931","heartbeat_front_addr":"10.20.193.68:6879/1182331931","state":["exists","up"]},{"osd":8,"uuid":"98acdd38-bad4-49c7-a681-772ff7e34dcc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":17,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3946693881},{"type":"v1","addr":"10.20.193.68:6803","nonce":3946693881}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3946693881},{"type":"v1","addr":"10.20.193.68:6805","nonce":3946693881}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3946693881},{"type":"v1","addr":"10.20.193.68:6809","nonce":3946693881}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3946693881},{"type":"v1","addr":"10.20.193.68:6807","nonce":3946693881}]},"public_addr":"10.20.193.68:6803/3946693881","cluster_addr":"10.20.193.68:6805/3946693881","heartbeat_back_addr":"10.20.193.68:6809/3946693881","heartbeat_front_addr":"10.20.193.68:6807/3946693881","state":["exists","up"]},{"osd":9,"uuid":"e2c6da1d-87c9-4ebf-8234-4356adbb2bb5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6818","nonce":3148430907},{"type":"v1","addr":"10.20.193.68:6819","nonce":3148430907}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6820","nonce":3148430907},{"type":"v1","addr":"10.20.193.68:6821","nonce":3148430907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6824","nonce":3148430907},{"type":"v1","addr":"10.20.193.68:6825","nonce":3148430907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6822","nonce":3148430907},{"type":"v1","addr":"10.20.193.68:6823","nonce":3148430907}]},"public_addr":"10.20.193.68:6819/3148430907","cluster_addr":"10.20.193.68:6821/3148430907","heartbeat_back_addr":"10.20.193.68:6825/3148430907","heartbeat_front_addr":"10.20.193.68:6823/3148430907","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-24T18:00:45.570554+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-24T18:00:45.565655+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-24T18:00:45.572517+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-24T18:00:45.573166+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-24T18:00:45.595183+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-24T18:00:45.560129+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-24T18:00:45.591572+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-24T18:00:45.639584+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-24T18:00:45.530957+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-24T18:00:45.534329+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":1,"snaps":[{"begin":2,"length":1}]}],"new_removed_snaps":[{"pool":1,"snaps":[{"begin":2,"length":1}]}],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-24T18:00:56.215 DEBUG:teuthology.orchestra.run.trial068:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2026-02-24T18:00:56.216 DEBUG:teuthology.orchestra.run.trial068:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2026-02-24T18:00:56.216 DEBUG:teuthology.orchestra.run.trial068:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2026-02-24T18:00:56.216 DEBUG:teuthology.orchestra.run.trial068:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 flush_pg_stats 2026-02-24T18:00:56.216 DEBUG:teuthology.orchestra.run.trial068:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.4 flush_pg_stats 2026-02-24T18:00:56.217 DEBUG:teuthology.orchestra.run.trial068:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.5 flush_pg_stats 2026-02-24T18:00:56.217 DEBUG:teuthology.orchestra.run.trial068:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.6 flush_pg_stats 2026-02-24T18:00:56.217 DEBUG:teuthology.orchestra.run.trial068:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.7 flush_pg_stats 2026-02-24T18:00:56.217 DEBUG:teuthology.orchestra.run.trial068:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.8 flush_pg_stats 2026-02-24T18:00:56.217 DEBUG:teuthology.orchestra.run.trial068:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.9 flush_pg_stats 2026-02-24T18:00:56.313 INFO:teuthology.orchestra.run.trial068.stdout:64424509443 2026-02-24T18:00:56.314 DEBUG:teuthology.orchestra.run.trial068:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2026-02-24T18:00:56.315 INFO:teuthology.orchestra.run.trial068.stdout:64424509443 2026-02-24T18:00:56.316 DEBUG:teuthology.orchestra.run.trial068:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.6 2026-02-24T18:00:56.317 INFO:teuthology.orchestra.run.trial068.stdout:64424509443 2026-02-24T18:00:56.317 DEBUG:teuthology.orchestra.run.trial068:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.9 2026-02-24T18:00:56.318 INFO:teuthology.orchestra.run.trial068.stdout:64424509444 2026-02-24T18:00:56.319 DEBUG:teuthology.orchestra.run.trial068:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2026-02-24T18:00:56.320 INFO:teuthology.orchestra.run.trial068.stdout:64424509443 2026-02-24T18:00:56.320 DEBUG:teuthology.orchestra.run.trial068:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.4 2026-02-24T18:00:56.321 INFO:teuthology.orchestra.run.trial068.stdout:64424509443 2026-02-24T18:00:56.322 DEBUG:teuthology.orchestra.run.trial068:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.8 2026-02-24T18:00:56.323 INFO:teuthology.orchestra.run.trial068.stdout:64424509443 2026-02-24T18:00:56.324 DEBUG:teuthology.orchestra.run.trial068:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.7 2026-02-24T18:00:56.325 INFO:teuthology.orchestra.run.trial068.stdout:64424509443 2026-02-24T18:00:56.326 DEBUG:teuthology.orchestra.run.trial068:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2026-02-24T18:00:56.327 INFO:teuthology.orchestra.run.trial068.stdout:64424509443 2026-02-24T18:00:56.327 DEBUG:teuthology.orchestra.run.trial068:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.5 2026-02-24T18:00:56.329 INFO:teuthology.orchestra.run.trial068.stdout:64424509443 2026-02-24T18:00:56.330 DEBUG:teuthology.orchestra.run.trial068:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2026-02-24T18:00:56.504 INFO:teuthology.orchestra.run.trial068.stdout:64424509442 2026-02-24T18:00:56.511 INFO:teuthology.orchestra.run.trial068.stdout:64424509442 2026-02-24T18:00:56.518 INFO:tasks.ceph.ceph_manager.ceph:need seq 64424509443 got 64424509442 for osd.6 2026-02-24T18:00:56.525 INFO:tasks.ceph.ceph_manager.ceph:need seq 64424509443 got 64424509442 for osd.1 2026-02-24T18:00:56.551 INFO:teuthology.orchestra.run.trial068.stdout:64424509442 2026-02-24T18:00:56.555 INFO:teuthology.orchestra.run.trial068.stdout:64424509442 2026-02-24T18:00:56.556 INFO:teuthology.orchestra.run.trial068.stdout:64424509442 2026-02-24T18:00:56.556 INFO:teuthology.orchestra.run.trial068.stdout:64424509443 2026-02-24T18:00:56.564 INFO:teuthology.orchestra.run.trial068.stdout:64424509442 2026-02-24T18:00:56.564 INFO:tasks.ceph.ceph_manager.ceph:need seq 64424509443 got 64424509442 for osd.5 2026-02-24T18:00:56.565 INFO:teuthology.orchestra.run.trial068.stdout:64424509442 2026-02-24T18:00:56.568 INFO:tasks.ceph.ceph_manager.ceph:need seq 64424509443 got 64424509442 for osd.4 2026-02-24T18:00:56.569 INFO:tasks.ceph.ceph_manager.ceph:need seq 64424509444 got 64424509443 for osd.0 2026-02-24T18:00:56.570 INFO:tasks.ceph.ceph_manager.ceph:need seq 64424509443 got 64424509442 for osd.9 2026-02-24T18:00:56.576 INFO:teuthology.orchestra.run.trial068.stdout:64424509442 2026-02-24T18:00:56.577 INFO:tasks.ceph.ceph_manager.ceph:need seq 64424509443 got 64424509442 for osd.2 2026-02-24T18:00:56.578 INFO:tasks.ceph.ceph_manager.ceph:need seq 64424509443 got 64424509442 for osd.7 2026-02-24T18:00:56.582 INFO:teuthology.orchestra.run.trial068.stdout:64424509442 2026-02-24T18:00:56.589 INFO:tasks.ceph.ceph_manager.ceph:need seq 64424509443 got 64424509442 for osd.8 2026-02-24T18:00:56.594 INFO:tasks.ceph.ceph_manager.ceph:need seq 64424509443 got 64424509442 for osd.3 2026-02-24T18:00:57.519 DEBUG:teuthology.orchestra.run.trial068:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.6 2026-02-24T18:00:57.525 DEBUG:teuthology.orchestra.run.trial068:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2026-02-24T18:00:57.565 DEBUG:teuthology.orchestra.run.trial068:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.5 2026-02-24T18:00:57.568 DEBUG:teuthology.orchestra.run.trial068:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.4 2026-02-24T18:00:57.570 DEBUG:teuthology.orchestra.run.trial068:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2026-02-24T18:00:57.570 DEBUG:teuthology.orchestra.run.trial068:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.9 2026-02-24T18:00:57.578 DEBUG:teuthology.orchestra.run.trial068:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2026-02-24T18:00:57.579 DEBUG:teuthology.orchestra.run.trial068:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.7 2026-02-24T18:00:57.590 DEBUG:teuthology.orchestra.run.trial068:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.8 2026-02-24T18:00:57.596 DEBUG:teuthology.orchestra.run.trial068:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2026-02-24T18:00:57.688 INFO:teuthology.orchestra.run.trial068.stdout:64424509443 2026-02-24T18:00:57.702 INFO:tasks.ceph.ceph_manager.ceph:need seq 64424509443 got 64424509443 for osd.6 2026-02-24T18:00:57.702 DEBUG:teuthology.parallel:result is None 2026-02-24T18:00:57.728 INFO:teuthology.orchestra.run.trial068.stdout:64424509443 2026-02-24T18:00:57.741 INFO:tasks.ceph.ceph_manager.ceph:need seq 64424509443 got 64424509443 for osd.1 2026-02-24T18:00:57.741 DEBUG:teuthology.parallel:result is None 2026-02-24T18:00:57.773 INFO:teuthology.orchestra.run.trial068.stdout:64424509443 2026-02-24T18:00:57.773 INFO:teuthology.orchestra.run.trial068.stdout:64424509443 2026-02-24T18:00:57.776 INFO:teuthology.orchestra.run.trial068.stdout:64424509443 2026-02-24T18:00:57.781 INFO:teuthology.orchestra.run.trial068.stdout:64424509443 2026-02-24T18:00:57.783 INFO:teuthology.orchestra.run.trial068.stdout:64424509443 2026-02-24T18:00:57.786 INFO:tasks.ceph.ceph_manager.ceph:need seq 64424509443 got 64424509443 for osd.5 2026-02-24T18:00:57.786 DEBUG:teuthology.parallel:result is None 2026-02-24T18:00:57.787 INFO:teuthology.orchestra.run.trial068.stdout:64424509443 2026-02-24T18:00:57.787 INFO:tasks.ceph.ceph_manager.ceph:need seq 64424509443 got 64424509443 for osd.7 2026-02-24T18:00:57.787 DEBUG:teuthology.parallel:result is None 2026-02-24T18:00:57.789 INFO:tasks.ceph.ceph_manager.ceph:need seq 64424509443 got 64424509443 for osd.2 2026-02-24T18:00:57.789 DEBUG:teuthology.parallel:result is None 2026-02-24T18:00:57.791 INFO:teuthology.orchestra.run.trial068.stdout:64424509444 2026-02-24T18:00:57.793 INFO:teuthology.orchestra.run.trial068.stdout:64424509443 2026-02-24T18:00:57.794 INFO:tasks.ceph.ceph_manager.ceph:need seq 64424509443 got 64424509443 for osd.4 2026-02-24T18:00:57.795 DEBUG:teuthology.parallel:result is None 2026-02-24T18:00:57.796 INFO:tasks.ceph.ceph_manager.ceph:need seq 64424509443 got 64424509443 for osd.3 2026-02-24T18:00:57.796 DEBUG:teuthology.parallel:result is None 2026-02-24T18:00:57.799 INFO:tasks.ceph.ceph_manager.ceph:need seq 64424509443 got 64424509443 for osd.9 2026-02-24T18:00:57.799 DEBUG:teuthology.parallel:result is None 2026-02-24T18:00:57.804 INFO:tasks.ceph.ceph_manager.ceph:need seq 64424509444 got 64424509444 for osd.0 2026-02-24T18:00:57.804 DEBUG:teuthology.parallel:result is None 2026-02-24T18:00:57.806 INFO:tasks.ceph.ceph_manager.ceph:need seq 64424509443 got 64424509443 for osd.8 2026-02-24T18:00:57.806 DEBUG:teuthology.parallel:result is None 2026-02-24T18:00:57.806 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2026-02-24T18:00:57.806 DEBUG:teuthology.orchestra.run.trial068:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-24T18:00:58.008 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-24T18:00:58.008 INFO:teuthology.orchestra.run.trial068.stderr:dumped all 2026-02-24T18:00:58.018 INFO:teuthology.orchestra.run.trial068.stdout:{"pg_ready":true,"pg_map":{"version":16,"stamp":"2026-02-24T18:00:57.460567+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":10,"num_per_pool_osds":10,"num_per_pool_omap_osds":9,"kb":1048576000,"kb_used":271408,"kb_used_data":1808,"kb_used_omap":64,"kb_used_meta":269375,"kb_avail":1048304592,"statfs":{"total":1073741824000,"available":1073463902208,"internally_reserved":0,"allocated":1851392,"data_stored":594204,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":65618,"internal_metadata":275840942},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"0.000000"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":18,"reported_epoch":20,"state":"active+clean","last_fresh":"2026-02-24T18:00:55.662553+0000","last_change":"2026-02-24T18:00:55.662553+0000","last_active":"2026-02-24T18:00:55.662553+0000","last_peered":"2026-02-24T18:00:55.662553+0000","last_clean":"2026-02-24T18:00:55.662553+0000","last_became_active":"2026-02-24T18:00:53.843991+0000","last_became_peered":"2026-02-24T18:00:53.843991+0000","last_unstale":"2026-02-24T18:00:55.662553+0000","last_undegraded":"2026-02-24T18:00:55.662553+0000","last_fullsized":"2026-02-24T18:00:55.662553+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-24T18:00:52.474019+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-24T18:00:52.474019+0000","last_clean_scrub_stamp":"2026-02-24T18:00:52.474019+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-26T03:15:45.978864+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000255505,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"1.6","version":"0'0","reported_seq":18,"reported_epoch":20,"state":"active+clean","last_fresh":"2026-02-24T18:00:55.662592+0000","last_change":"2026-02-24T18:00:55.662592+0000","last_active":"2026-02-24T18:00:55.662592+0000","last_peered":"2026-02-24T18:00:55.662592+0000","last_clean":"2026-02-24T18:00:55.662592+0000","last_became_active":"2026-02-24T18:00:53.524271+0000","last_became_peered":"2026-02-24T18:00:53.524271+0000","last_unstale":"2026-02-24T18:00:55.662592+0000","last_undegraded":"2026-02-24T18:00:55.662592+0000","last_fullsized":"2026-02-24T18:00:55.662592+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-24T18:00:52.474019+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-24T18:00:52.474019+0000","last_clean_scrub_stamp":"2026-02-24T18:00:52.474019+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-26T04:35:58.161616+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00029952599999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,9],"acting":[6,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"1.5","version":"0'0","reported_seq":20,"reported_epoch":20,"state":"active+clean","last_fresh":"2026-02-24T18:00:55.662189+0000","last_change":"2026-02-24T18:00:55.662189+0000","last_active":"2026-02-24T18:00:55.662189+0000","last_peered":"2026-02-24T18:00:55.662189+0000","last_clean":"2026-02-24T18:00:55.662189+0000","last_became_active":"2026-02-24T18:00:53.482837+0000","last_became_peered":"2026-02-24T18:00:53.482837+0000","last_unstale":"2026-02-24T18:00:55.662189+0000","last_undegraded":"2026-02-24T18:00:55.662189+0000","last_fullsized":"2026-02-24T18:00:55.662189+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-24T18:00:52.474019+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-24T18:00:52.474019+0000","last_clean_scrub_stamp":"2026-02-24T18:00:52.474019+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-25T22:42:14.088556+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00028598099999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8],"acting":[4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"1.4","version":"0'0","reported_seq":20,"reported_epoch":20,"state":"active+clean","last_fresh":"2026-02-24T18:00:55.662412+0000","last_change":"2026-02-24T18:00:55.662412+0000","last_active":"2026-02-24T18:00:55.662412+0000","last_peered":"2026-02-24T18:00:55.662412+0000","last_clean":"2026-02-24T18:00:55.662412+0000","last_became_active":"2026-02-24T18:00:53.482545+0000","last_became_peered":"2026-02-24T18:00:53.482545+0000","last_unstale":"2026-02-24T18:00:55.662412+0000","last_undegraded":"2026-02-24T18:00:55.662412+0000","last_fullsized":"2026-02-24T18:00:55.662412+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-24T18:00:52.474019+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-24T18:00:52.474019+0000","last_clean_scrub_stamp":"2026-02-24T18:00:52.474019+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-26T00:33:54.968970+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00030417500000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"1.3","version":"18'1","reported_seq":21,"reported_epoch":20,"state":"active+clean","last_fresh":"2026-02-24T18:00:55.662213+0000","last_change":"2026-02-24T18:00:55.662213+0000","last_active":"2026-02-24T18:00:55.662213+0000","last_peered":"2026-02-24T18:00:55.662213+0000","last_clean":"2026-02-24T18:00:55.662213+0000","last_became_active":"2026-02-24T18:00:53.871921+0000","last_became_peered":"2026-02-24T18:00:53.871921+0000","last_unstale":"2026-02-24T18:00:55.662213+0000","last_undegraded":"2026-02-24T18:00:55.662213+0000","last_fullsized":"2026-02-24T18:00:55.662213+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-24T18:00:52.474019+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-24T18:00:52.474019+0000","last_clean_scrub_stamp":"2026-02-24T18:00:52.474019+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-26T02:06:43.191798+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00028932799999999999,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"1.2","version":"20'2","reported_seq":22,"reported_epoch":20,"state":"active+clean","last_fresh":"2026-02-24T18:00:55.486009+0000","last_change":"2026-02-24T18:00:55.486009+0000","last_active":"2026-02-24T18:00:55.486009+0000","last_peered":"2026-02-24T18:00:55.486009+0000","last_clean":"2026-02-24T18:00:55.486009+0000","last_became_active":"2026-02-24T18:00:53.844026+0000","last_became_peered":"2026-02-24T18:00:53.844026+0000","last_unstale":"2026-02-24T18:00:55.486009+0000","last_undegraded":"2026-02-24T18:00:55.486009+0000","last_fullsized":"2026-02-24T18:00:55.486009+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-24T18:00:52.474019+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-24T18:00:52.474019+0000","last_clean_scrub_stamp":"2026-02-24T18:00:52.474019+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-26T01:38:08.256400+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00073412399999999995,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"1.1","version":"0'0","reported_seq":18,"reported_epoch":20,"state":"active+clean","last_fresh":"2026-02-24T18:00:55.484015+0000","last_change":"2026-02-24T18:00:55.484015+0000","last_active":"2026-02-24T18:00:55.484015+0000","last_peered":"2026-02-24T18:00:55.484015+0000","last_clean":"2026-02-24T18:00:55.484015+0000","last_became_active":"2026-02-24T18:00:53.482477+0000","last_became_peered":"2026-02-24T18:00:53.482477+0000","last_unstale":"2026-02-24T18:00:55.484015+0000","last_undegraded":"2026-02-24T18:00:55.484015+0000","last_fullsized":"2026-02-24T18:00:55.484015+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-24T18:00:52.474019+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-24T18:00:52.474019+0000","last_clean_scrub_stamp":"2026-02-24T18:00:52.474019+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-25T20:04:21.256629+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00026963099999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8],"acting":[5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"1.0","version":"0'0","reported_seq":18,"reported_epoch":20,"state":"active+clean","last_fresh":"2026-02-24T18:00:55.513621+0000","last_change":"2026-02-24T18:00:55.513621+0000","last_active":"2026-02-24T18:00:55.513621+0000","last_peered":"2026-02-24T18:00:55.513621+0000","last_clean":"2026-02-24T18:00:55.513621+0000","last_became_active":"2026-02-24T18:00:53.482856+0000","last_became_peered":"2026-02-24T18:00:53.482856+0000","last_unstale":"2026-02-24T18:00:55.513621+0000","last_undegraded":"2026-02-24T18:00:55.513621+0000","last_fullsized":"2026-02-24T18:00:55.513621+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-24T18:00:52.474019+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-24T18:00:52.474019+0000","last_clean_scrub_stamp":"2026-02-24T18:00:52.474019+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-26T03:33:42.575439+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00021002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0],"acting":[8,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[{"start":"2","length":"1"}]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":9}],"osd_stats":[{"osd":9,"up_from":15,"seq":64424509443,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":27152,"kb_used_data":192,"kb_used_omap":6,"kb_used_meta":26937,"kb_avail":104830448,"statfs":{"total":107374182400,"available":107346378752,"internally_reserved":0,"allocated":196608,"data_stored":65582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6830,"internal_metadata":27583826},"hb_peers":[0,1,2,3,4,5,6,7,8],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":15,"seq":64424509443,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":27128,"kb_used_data":168,"kb_used_omap":6,"kb_used_meta":26937,"kb_avail":104830472,"statfs":{"total":107374182400,"available":107346403328,"internally_reserved":0,"allocated":172032,"data_stored":53268,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6168,"internal_metadata":27584488},"hb_peers":[0,1,2,3,4,5,6,7,9],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":15,"seq":64424509443,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":27156,"kb_used_data":196,"kb_used_omap":7,"kb_used_meta":26936,"kb_avail":104830444,"statfs":{"total":107374182400,"available":107346374656,"internally_reserved":0,"allocated":200704,"data_stored":65601,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7470,"internal_metadata":27583186},"hb_peers":[0,1,2,3,4,5,6,8,9],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":15,"seq":64424509443,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":27128,"kb_used_data":168,"kb_used_omap":6,"kb_used_meta":26937,"kb_avail":104830472,"statfs":{"total":107374182400,"available":107346403328,"internally_reserved":0,"allocated":172032,"data_stored":53268,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6167,"internal_metadata":27584489},"hb_peers":[0,1,2,3,4,5,7,8,9],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":15,"seq":64424509443,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":27152,"kb_used_data":192,"kb_used_omap":6,"kb_used_meta":26937,"kb_avail":104830448,"statfs":{"total":107374182400,"available":107346378752,"internally_reserved":0,"allocated":196608,"data_stored":65582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6825,"internal_metadata":27583831},"hb_peers":[0,1,2,3,4,6,7,8,9],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":15,"seq":64424509443,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":27140,"kb_used_data":180,"kb_used_omap":6,"kb_used_meta":26937,"kb_avail":104830460,"statfs":{"total":107374182400,"available":107346391040,"internally_reserved":0,"allocated":184320,"data_stored":59383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6825,"internal_metadata":27583831},"hb_peers":[0,1,2,3,5,6,7,8,9],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":15,"seq":64424509443,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":27140,"kb_used_data":180,"kb_used_omap":6,"kb_used_meta":26937,"kb_avail":104830460,"statfs":{"total":107374182400,"available":107346391040,"internally_reserved":0,"allocated":184320,"data_stored":59383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6822,"internal_metadata":27583834},"hb_peers":[0,1,2,4,5,6,7,8,9],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":15,"seq":64424509443,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":104857600,"kb_used":27128,"kb_used_data":168,"kb_used_omap":6,"kb_used_meta":26937,"kb_avail":104830472,"statfs":{"total":107374182400,"available":107346403328,"internally_reserved":0,"allocated":172032,"data_stored":53268,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6168,"internal_metadata":27584488},"hb_peers":[0,1,3,4,5,6,7,8,9],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":15,"seq":64424509443,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":27128,"kb_used_data":168,"kb_used_omap":6,"kb_used_meta":26937,"kb_avail":104830472,"statfs":{"total":107374182400,"available":107346403328,"internally_reserved":0,"allocated":172032,"data_stored":53268,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6168,"internal_metadata":27584488},"hb_peers":[0,2,3,4,5,6,7,8,9],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":15,"seq":64424509444,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":27156,"kb_used_data":196,"kb_used_omap":6,"kb_used_meta":26937,"kb_avail":104830444,"statfs":{"total":107374182400,"available":107346374656,"internally_reserved":0,"allocated":200704,"data_stored":65601,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6175,"internal_metadata":27584481},"hb_peers":[1,2,3,4,5,6,7,8,9],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-24T18:00:58.019 DEBUG:teuthology.orchestra.run.trial068:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-24T18:00:58.178 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-24T18:00:58.178 INFO:teuthology.orchestra.run.trial068.stderr:dumped all 2026-02-24T18:00:58.190 INFO:teuthology.orchestra.run.trial068.stdout:{"pg_ready":true,"pg_map":{"version":16,"stamp":"2026-02-24T18:00:57.460567+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":10,"num_per_pool_osds":10,"num_per_pool_omap_osds":9,"kb":1048576000,"kb_used":271408,"kb_used_data":1808,"kb_used_omap":64,"kb_used_meta":269375,"kb_avail":1048304592,"statfs":{"total":1073741824000,"available":1073463902208,"internally_reserved":0,"allocated":1851392,"data_stored":594204,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":65618,"internal_metadata":275840942},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"0.000000"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":18,"reported_epoch":20,"state":"active+clean","last_fresh":"2026-02-24T18:00:55.662553+0000","last_change":"2026-02-24T18:00:55.662553+0000","last_active":"2026-02-24T18:00:55.662553+0000","last_peered":"2026-02-24T18:00:55.662553+0000","last_clean":"2026-02-24T18:00:55.662553+0000","last_became_active":"2026-02-24T18:00:53.843991+0000","last_became_peered":"2026-02-24T18:00:53.843991+0000","last_unstale":"2026-02-24T18:00:55.662553+0000","last_undegraded":"2026-02-24T18:00:55.662553+0000","last_fullsized":"2026-02-24T18:00:55.662553+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-24T18:00:52.474019+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-24T18:00:52.474019+0000","last_clean_scrub_stamp":"2026-02-24T18:00:52.474019+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-26T03:15:45.978864+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000255505,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"1.6","version":"0'0","reported_seq":18,"reported_epoch":20,"state":"active+clean","last_fresh":"2026-02-24T18:00:55.662592+0000","last_change":"2026-02-24T18:00:55.662592+0000","last_active":"2026-02-24T18:00:55.662592+0000","last_peered":"2026-02-24T18:00:55.662592+0000","last_clean":"2026-02-24T18:00:55.662592+0000","last_became_active":"2026-02-24T18:00:53.524271+0000","last_became_peered":"2026-02-24T18:00:53.524271+0000","last_unstale":"2026-02-24T18:00:55.662592+0000","last_undegraded":"2026-02-24T18:00:55.662592+0000","last_fullsized":"2026-02-24T18:00:55.662592+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-24T18:00:52.474019+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-24T18:00:52.474019+0000","last_clean_scrub_stamp":"2026-02-24T18:00:52.474019+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-26T04:35:58.161616+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00029952599999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,9],"acting":[6,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"1.5","version":"0'0","reported_seq":20,"reported_epoch":20,"state":"active+clean","last_fresh":"2026-02-24T18:00:55.662189+0000","last_change":"2026-02-24T18:00:55.662189+0000","last_active":"2026-02-24T18:00:55.662189+0000","last_peered":"2026-02-24T18:00:55.662189+0000","last_clean":"2026-02-24T18:00:55.662189+0000","last_became_active":"2026-02-24T18:00:53.482837+0000","last_became_peered":"2026-02-24T18:00:53.482837+0000","last_unstale":"2026-02-24T18:00:55.662189+0000","last_undegraded":"2026-02-24T18:00:55.662189+0000","last_fullsized":"2026-02-24T18:00:55.662189+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-24T18:00:52.474019+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-24T18:00:52.474019+0000","last_clean_scrub_stamp":"2026-02-24T18:00:52.474019+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-25T22:42:14.088556+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00028598099999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8],"acting":[4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"1.4","version":"0'0","reported_seq":20,"reported_epoch":20,"state":"active+clean","last_fresh":"2026-02-24T18:00:55.662412+0000","last_change":"2026-02-24T18:00:55.662412+0000","last_active":"2026-02-24T18:00:55.662412+0000","last_peered":"2026-02-24T18:00:55.662412+0000","last_clean":"2026-02-24T18:00:55.662412+0000","last_became_active":"2026-02-24T18:00:53.482545+0000","last_became_peered":"2026-02-24T18:00:53.482545+0000","last_unstale":"2026-02-24T18:00:55.662412+0000","last_undegraded":"2026-02-24T18:00:55.662412+0000","last_fullsized":"2026-02-24T18:00:55.662412+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-24T18:00:52.474019+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-24T18:00:52.474019+0000","last_clean_scrub_stamp":"2026-02-24T18:00:52.474019+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-26T00:33:54.968970+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00030417500000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"1.3","version":"18'1","reported_seq":21,"reported_epoch":20,"state":"active+clean","last_fresh":"2026-02-24T18:00:55.662213+0000","last_change":"2026-02-24T18:00:55.662213+0000","last_active":"2026-02-24T18:00:55.662213+0000","last_peered":"2026-02-24T18:00:55.662213+0000","last_clean":"2026-02-24T18:00:55.662213+0000","last_became_active":"2026-02-24T18:00:53.871921+0000","last_became_peered":"2026-02-24T18:00:53.871921+0000","last_unstale":"2026-02-24T18:00:55.662213+0000","last_undegraded":"2026-02-24T18:00:55.662213+0000","last_fullsized":"2026-02-24T18:00:55.662213+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-24T18:00:52.474019+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-24T18:00:52.474019+0000","last_clean_scrub_stamp":"2026-02-24T18:00:52.474019+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-26T02:06:43.191798+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00028932799999999999,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"1.2","version":"20'2","reported_seq":22,"reported_epoch":20,"state":"active+clean","last_fresh":"2026-02-24T18:00:55.486009+0000","last_change":"2026-02-24T18:00:55.486009+0000","last_active":"2026-02-24T18:00:55.486009+0000","last_peered":"2026-02-24T18:00:55.486009+0000","last_clean":"2026-02-24T18:00:55.486009+0000","last_became_active":"2026-02-24T18:00:53.844026+0000","last_became_peered":"2026-02-24T18:00:53.844026+0000","last_unstale":"2026-02-24T18:00:55.486009+0000","last_undegraded":"2026-02-24T18:00:55.486009+0000","last_fullsized":"2026-02-24T18:00:55.486009+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-24T18:00:52.474019+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-24T18:00:52.474019+0000","last_clean_scrub_stamp":"2026-02-24T18:00:52.474019+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-26T01:38:08.256400+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00073412399999999995,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"1.1","version":"0'0","reported_seq":18,"reported_epoch":20,"state":"active+clean","last_fresh":"2026-02-24T18:00:55.484015+0000","last_change":"2026-02-24T18:00:55.484015+0000","last_active":"2026-02-24T18:00:55.484015+0000","last_peered":"2026-02-24T18:00:55.484015+0000","last_clean":"2026-02-24T18:00:55.484015+0000","last_became_active":"2026-02-24T18:00:53.482477+0000","last_became_peered":"2026-02-24T18:00:53.482477+0000","last_unstale":"2026-02-24T18:00:55.484015+0000","last_undegraded":"2026-02-24T18:00:55.484015+0000","last_fullsized":"2026-02-24T18:00:55.484015+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-24T18:00:52.474019+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-24T18:00:52.474019+0000","last_clean_scrub_stamp":"2026-02-24T18:00:52.474019+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-25T20:04:21.256629+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00026963099999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8],"acting":[5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"1.0","version":"0'0","reported_seq":18,"reported_epoch":20,"state":"active+clean","last_fresh":"2026-02-24T18:00:55.513621+0000","last_change":"2026-02-24T18:00:55.513621+0000","last_active":"2026-02-24T18:00:55.513621+0000","last_peered":"2026-02-24T18:00:55.513621+0000","last_clean":"2026-02-24T18:00:55.513621+0000","last_became_active":"2026-02-24T18:00:53.482856+0000","last_became_peered":"2026-02-24T18:00:53.482856+0000","last_unstale":"2026-02-24T18:00:55.513621+0000","last_undegraded":"2026-02-24T18:00:55.513621+0000","last_fullsized":"2026-02-24T18:00:55.513621+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-24T18:00:52.474019+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-24T18:00:52.474019+0000","last_clean_scrub_stamp":"2026-02-24T18:00:52.474019+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-26T03:33:42.575439+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00021002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0],"acting":[8,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[{"start":"2","length":"1"}]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":9}],"osd_stats":[{"osd":9,"up_from":15,"seq":64424509443,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":27152,"kb_used_data":192,"kb_used_omap":6,"kb_used_meta":26937,"kb_avail":104830448,"statfs":{"total":107374182400,"available":107346378752,"internally_reserved":0,"allocated":196608,"data_stored":65582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6830,"internal_metadata":27583826},"hb_peers":[0,1,2,3,4,5,6,7,8],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":15,"seq":64424509443,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":27128,"kb_used_data":168,"kb_used_omap":6,"kb_used_meta":26937,"kb_avail":104830472,"statfs":{"total":107374182400,"available":107346403328,"internally_reserved":0,"allocated":172032,"data_stored":53268,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6168,"internal_metadata":27584488},"hb_peers":[0,1,2,3,4,5,6,7,9],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":15,"seq":64424509443,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":27156,"kb_used_data":196,"kb_used_omap":7,"kb_used_meta":26936,"kb_avail":104830444,"statfs":{"total":107374182400,"available":107346374656,"internally_reserved":0,"allocated":200704,"data_stored":65601,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7470,"internal_metadata":27583186},"hb_peers":[0,1,2,3,4,5,6,8,9],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":15,"seq":64424509443,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":27128,"kb_used_data":168,"kb_used_omap":6,"kb_used_meta":26937,"kb_avail":104830472,"statfs":{"total":107374182400,"available":107346403328,"internally_reserved":0,"allocated":172032,"data_stored":53268,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6167,"internal_metadata":27584489},"hb_peers":[0,1,2,3,4,5,7,8,9],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":15,"seq":64424509443,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":27152,"kb_used_data":192,"kb_used_omap":6,"kb_used_meta":26937,"kb_avail":104830448,"statfs":{"total":107374182400,"available":107346378752,"internally_reserved":0,"allocated":196608,"data_stored":65582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6825,"internal_metadata":27583831},"hb_peers":[0,1,2,3,4,6,7,8,9],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":15,"seq":64424509443,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":27140,"kb_used_data":180,"kb_used_omap":6,"kb_used_meta":26937,"kb_avail":104830460,"statfs":{"total":107374182400,"available":107346391040,"internally_reserved":0,"allocated":184320,"data_stored":59383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6825,"internal_metadata":27583831},"hb_peers":[0,1,2,3,5,6,7,8,9],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":15,"seq":64424509443,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":27140,"kb_used_data":180,"kb_used_omap":6,"kb_used_meta":26937,"kb_avail":104830460,"statfs":{"total":107374182400,"available":107346391040,"internally_reserved":0,"allocated":184320,"data_stored":59383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6822,"internal_metadata":27583834},"hb_peers":[0,1,2,4,5,6,7,8,9],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":15,"seq":64424509443,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":104857600,"kb_used":27128,"kb_used_data":168,"kb_used_omap":6,"kb_used_meta":26937,"kb_avail":104830472,"statfs":{"total":107374182400,"available":107346403328,"internally_reserved":0,"allocated":172032,"data_stored":53268,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6168,"internal_metadata":27584488},"hb_peers":[0,1,3,4,5,6,7,8,9],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":15,"seq":64424509443,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":27128,"kb_used_data":168,"kb_used_omap":6,"kb_used_meta":26937,"kb_avail":104830472,"statfs":{"total":107374182400,"available":107346403328,"internally_reserved":0,"allocated":172032,"data_stored":53268,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6168,"internal_metadata":27584488},"hb_peers":[0,2,3,4,5,6,7,8,9],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":15,"seq":64424509444,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":27156,"kb_used_data":196,"kb_used_omap":6,"kb_used_meta":26937,"kb_avail":104830444,"statfs":{"total":107374182400,"available":107346374656,"internally_reserved":0,"allocated":200704,"data_stored":65601,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6175,"internal_metadata":27584481},"hb_peers":[1,2,3,4,5,6,7,8,9],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-24T18:00:58.190 INFO:tasks.ceph.ceph_manager.ceph:clean! 2026-02-24T18:00:58.190 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2026-02-24T18:00:58.190 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy 2026-02-24T18:00:58.191 DEBUG:teuthology.orchestra.run.trial068:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2026-02-24T18:00:58.361 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-24T18:00:58.361 INFO:teuthology.orchestra.run.trial068.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2026-02-24T18:00:58.371 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy done 2026-02-24T18:00:58.371 INFO:teuthology.run_tasks:Running task workunit... 2026-02-24T18:00:58.386 INFO:tasks.workunit:Pulling workunits from ref 618c318a7a0b287ab5aeff207b7bf1d73ef45044 2026-02-24T18:00:58.386 INFO:tasks.workunit:Making a separate scratch dir for every client... 2026-02-24T18:00:58.386 INFO:tasks.workunit:timeout=3h 2026-02-24T18:00:58.386 INFO:tasks.workunit:cleanup=True 2026-02-24T18:00:58.386 DEBUG:teuthology.orchestra.run.trial068:> stat -- /home/ubuntu/cephtest/mnt.0 2026-02-24T18:00:58.389 DEBUG:teuthology.orchestra.run:got remote process result: 1 2026-02-24T18:00:58.389 INFO:teuthology.orchestra.run.trial068.stderr:stat: cannot statx '/home/ubuntu/cephtest/mnt.0': No such file or directory 2026-02-24T18:00:58.390 DEBUG:teuthology.orchestra.run.trial068:> mkdir -- /home/ubuntu/cephtest/mnt.0 2026-02-24T18:00:58.437 INFO:tasks.workunit:Created dir /home/ubuntu/cephtest/mnt.0 2026-02-24T18:00:58.437 DEBUG:teuthology.orchestra.run.trial068:> cd -- /home/ubuntu/cephtest/mnt.0 && mkdir -- client.0 2026-02-24T18:00:58.482 DEBUG:teuthology.orchestra.run.trial068:> rm -rf /home/ubuntu/cephtest/clone.client.0 && git clone https://github.com/ceph/ceph-ci.git /home/ubuntu/cephtest/clone.client.0 && cd /home/ubuntu/cephtest/clone.client.0 && git checkout 618c318a7a0b287ab5aeff207b7bf1d73ef45044 2026-02-24T18:00:58.536 INFO:tasks.workunit.client.0.trial068.stderr:Cloning into '/home/ubuntu/cephtest/clone.client.0'... 2026-02-24T18:01:20.942 INFO:tasks.workunit.client.0.trial068.stderr:Note: switching to '618c318a7a0b287ab5aeff207b7bf1d73ef45044'. 2026-02-24T18:01:20.942 INFO:tasks.workunit.client.0.trial068.stderr: 2026-02-24T18:01:20.942 INFO:tasks.workunit.client.0.trial068.stderr:You are in 'detached HEAD' state. You can look around, make experimental 2026-02-24T18:01:20.942 INFO:tasks.workunit.client.0.trial068.stderr:changes and commit them, and you can discard any commits you make in this 2026-02-24T18:01:20.942 INFO:tasks.workunit.client.0.trial068.stderr:state without impacting any branches by switching back to a branch. 2026-02-24T18:01:20.942 INFO:tasks.workunit.client.0.trial068.stderr: 2026-02-24T18:01:20.942 INFO:tasks.workunit.client.0.trial068.stderr:If you want to create a new branch to retain commits you create, you may 2026-02-24T18:01:20.942 INFO:tasks.workunit.client.0.trial068.stderr:do so (now or later) by using -c with the switch command. Example: 2026-02-24T18:01:20.942 INFO:tasks.workunit.client.0.trial068.stderr: 2026-02-24T18:01:20.943 INFO:tasks.workunit.client.0.trial068.stderr: git switch -c 2026-02-24T18:01:20.943 INFO:tasks.workunit.client.0.trial068.stderr: 2026-02-24T18:01:20.943 INFO:tasks.workunit.client.0.trial068.stderr:Or undo this operation with: 2026-02-24T18:01:20.943 INFO:tasks.workunit.client.0.trial068.stderr: 2026-02-24T18:01:20.943 INFO:tasks.workunit.client.0.trial068.stderr: git switch - 2026-02-24T18:01:20.943 INFO:tasks.workunit.client.0.trial068.stderr: 2026-02-24T18:01:20.943 INFO:tasks.workunit.client.0.trial068.stderr:Turn off this advice by setting config variable advice.detachedHead to false 2026-02-24T18:01:20.943 INFO:tasks.workunit.client.0.trial068.stderr: 2026-02-24T18:01:20.943 INFO:tasks.workunit.client.0.trial068.stderr:HEAD is now at 618c318a7a0 Merge branch 'stretched-cluster' of https://github.com/leonidc/ceph into wip-yuri8-testing-2026-02-23-2128 2026-02-24T18:01:20.952 DEBUG:teuthology.orchestra.run.trial068:> cd -- /home/ubuntu/cephtest/clone.client.0/qa/workunits && if test -e Makefile ; then make ; fi && find -executable -type f -printf '%P\0' >/home/ubuntu/cephtest/workunits.list.client.0 2026-02-24T18:01:20.999 INFO:tasks.workunit.client.0.trial068.stdout:for d in direct_io fs ; do ( cd $d ; make all ) ; done 2026-02-24T18:01:21.000 INFO:tasks.workunit.client.0.trial068.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2026-02-24T18:01:21.001 INFO:tasks.workunit.client.0.trial068.stdout:cc -Wall -Wextra -D_GNU_SOURCE direct_io_test.c -o direct_io_test 2026-02-24T18:01:21.095 INFO:tasks.workunit.client.0.trial068.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_sync_io.c -o test_sync_io 2026-02-24T18:01:21.133 INFO:tasks.workunit.client.0.trial068.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_short_dio_read.c -o test_short_dio_read 2026-02-24T18:01:21.159 INFO:tasks.workunit.client.0.trial068.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2026-02-24T18:01:21.160 INFO:tasks.workunit.client.0.trial068.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2026-02-24T18:01:21.160 INFO:tasks.workunit.client.0.trial068.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_o_trunc.c -o test_o_trunc 2026-02-24T18:01:21.185 INFO:tasks.workunit.client.0.trial068.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2026-02-24T18:01:21.188 DEBUG:teuthology.orchestra.run.trial068:> set -ex 2026-02-24T18:01:21.188 DEBUG:teuthology.orchestra.run.trial068:> dd if=/home/ubuntu/cephtest/workunits.list.client.0 of=/dev/stdout 2026-02-24T18:01:21.234 INFO:tasks.workunit:Running workunits matching rados/test_health_warnings.sh on client.0... 2026-02-24T18:01:21.235 INFO:tasks.workunit:Running workunit rados/test_health_warnings.sh... 2026-02-24T18:01:21.235 DEBUG:teuthology.orchestra.run.trial068:workunit test rados/test_health_warnings.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=618c318a7a0b287ab5aeff207b7bf1d73ef45044 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/rados/test_health_warnings.sh 2026-02-24T18:01:21.282 INFO:tasks.workunit.client.0.trial068.stderr:+ crushtool -o crushmap --build --num_osds 10 host straw 2 rack straw 2 row straw 2 root straw 0 2026-02-24T18:01:21.291 INFO:tasks.workunit.client.0.trial068.stderr:+ ceph osd setcrushmap -i crushmap 2026-02-24T18:01:23.303 INFO:tasks.workunit.client.0.trial068.stderr:6 2026-02-24T18:01:23.316 INFO:tasks.workunit.client.0.trial068.stderr:+ ceph osd tree 2026-02-24T18:01:23.516 INFO:tasks.workunit.client.0.trial068.stdout:ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF 2026-02-24T18:01:23.516 INFO:tasks.workunit.client.0.trial068.stdout:-11 10.00000 root root 2026-02-24T18:01:23.516 INFO:tasks.workunit.client.0.trial068.stdout: -9 8.00000 row row0 2026-02-24T18:01:23.516 INFO:tasks.workunit.client.0.trial068.stdout: -6 4.00000 rack rack0 2026-02-24T18:01:23.516 INFO:tasks.workunit.client.0.trial068.stdout: -1 2.00000 host host0 2026-02-24T18:01:23.516 INFO:tasks.workunit.client.0.trial068.stdout: 0 1.00000 osd.0 up 1.00000 1.00000 2026-02-24T18:01:23.516 INFO:tasks.workunit.client.0.trial068.stdout: 1 1.00000 osd.1 up 1.00000 1.00000 2026-02-24T18:01:23.516 INFO:tasks.workunit.client.0.trial068.stdout: -2 2.00000 host host1 2026-02-24T18:01:23.516 INFO:tasks.workunit.client.0.trial068.stdout: 2 1.00000 osd.2 up 1.00000 1.00000 2026-02-24T18:01:23.516 INFO:tasks.workunit.client.0.trial068.stdout: 3 1.00000 osd.3 up 1.00000 1.00000 2026-02-24T18:01:23.516 INFO:tasks.workunit.client.0.trial068.stdout: -7 4.00000 rack rack1 2026-02-24T18:01:23.516 INFO:tasks.workunit.client.0.trial068.stdout: -3 2.00000 host host2 2026-02-24T18:01:23.516 INFO:tasks.workunit.client.0.trial068.stdout: 4 1.00000 osd.4 up 1.00000 1.00000 2026-02-24T18:01:23.516 INFO:tasks.workunit.client.0.trial068.stdout: 5 1.00000 osd.5 up 1.00000 1.00000 2026-02-24T18:01:23.516 INFO:tasks.workunit.client.0.trial068.stdout: -4 2.00000 host host3 2026-02-24T18:01:23.516 INFO:tasks.workunit.client.0.trial068.stdout: 6 1.00000 osd.6 up 1.00000 1.00000 2026-02-24T18:01:23.516 INFO:tasks.workunit.client.0.trial068.stdout: 7 1.00000 osd.7 up 1.00000 1.00000 2026-02-24T18:01:23.516 INFO:tasks.workunit.client.0.trial068.stdout:-10 2.00000 row row1 2026-02-24T18:01:23.516 INFO:tasks.workunit.client.0.trial068.stdout: -8 2.00000 rack rack2 2026-02-24T18:01:23.516 INFO:tasks.workunit.client.0.trial068.stdout: -5 2.00000 host host4 2026-02-24T18:01:23.516 INFO:tasks.workunit.client.0.trial068.stdout: 8 1.00000 osd.8 up 1.00000 1.00000 2026-02-24T18:01:23.516 INFO:tasks.workunit.client.0.trial068.stdout: 9 1.00000 osd.9 up 1.00000 1.00000 2026-02-24T18:01:23.525 INFO:tasks.workunit.client.0.trial068.stderr:+ ceph tell 'osd.*' injectargs --osd_max_markdown_count 1024 --osd_max_markdown_period 1 2026-02-24T18:01:23.618 INFO:tasks.workunit.client.0.trial068.stdout:osd.0: {} 2026-02-24T18:01:23.635 INFO:tasks.workunit.client.0.trial068.stdout:osd.1: {} 2026-02-24T18:01:23.651 INFO:tasks.workunit.client.0.trial068.stdout:osd.2: {} 2026-02-24T18:01:23.669 INFO:tasks.workunit.client.0.trial068.stdout:osd.3: {} 2026-02-24T18:01:23.686 INFO:tasks.workunit.client.0.trial068.stdout:osd.4: {} 2026-02-24T18:01:23.703 INFO:tasks.workunit.client.0.trial068.stdout:osd.5: {} 2026-02-24T18:01:23.721 INFO:tasks.workunit.client.0.trial068.stdout:osd.6: {} 2026-02-24T18:01:23.738 INFO:tasks.workunit.client.0.trial068.stdout:osd.7: {} 2026-02-24T18:01:23.754 INFO:tasks.workunit.client.0.trial068.stdout:osd.8: {} 2026-02-24T18:01:23.771 INFO:tasks.workunit.client.0.trial068.stdout:osd.9: {} 2026-02-24T18:01:23.783 INFO:tasks.workunit.client.0.trial068.stderr:+ ceph osd set noout 2026-02-24T18:01:25.306 INFO:tasks.workunit.client.0.trial068.stderr:noout is set 2026-02-24T18:01:25.318 INFO:tasks.workunit.client.0.trial068.stderr:+ test_mark_two_osds_same_host_down 2026-02-24T18:01:25.318 INFO:tasks.workunit.client.0.trial068.stderr:+ ceph osd set noup 2026-02-24T18:01:27.309 INFO:tasks.workunit.client.0.trial068.stderr:noup is set 2026-02-24T18:01:27.322 INFO:tasks.workunit.client.0.trial068.stderr:+ ceph osd down osd.0 osd.1 2026-02-24T18:01:27.475 INFO:tasks.ceph.mon.a.trial068.stderr:2026-02-24T18:01:27.468+0000 7fa3e05d4640 -1 mon.a@0(leader).osd e27 definitely_dead 0 2026-02-24T18:01:28.365 INFO:tasks.ceph.mon.a.trial068.stderr:2026-02-24T18:01:28.356+0000 7fa3e05d4640 -1 mon.a@0(leader).osd e28 definitely_dead 0 2026-02-24T18:01:28.365 INFO:tasks.workunit.client.0.trial068.stderr:osd.0 is already down. osd.1 is already down. 2026-02-24T18:01:28.375 INFO:tasks.workunit.client.0.trial068.stderr:+ ceph health detail 2026-02-24T18:01:28.652 INFO:tasks.workunit.client.0.trial068.stdout:HEALTH_WARN noup,noout flag(s) set; 2 osds down; 1 host (2 osds) down 2026-02-24T18:01:28.652 INFO:tasks.workunit.client.0.trial068.stdout:[WRN] OSDMAP_FLAGS: noup,noout flag(s) set 2026-02-24T18:01:28.652 INFO:tasks.workunit.client.0.trial068.stdout:[WRN] OSD_DOWN: 2 osds down 2026-02-24T18:01:28.652 INFO:tasks.workunit.client.0.trial068.stdout: osd.0 (root=root,row=row0,rack=rack0,host=host0) is down 2026-02-24T18:01:28.652 INFO:tasks.workunit.client.0.trial068.stdout: osd.1 (root=root,row=row0,rack=rack0,host=host0) is down 2026-02-24T18:01:28.652 INFO:tasks.workunit.client.0.trial068.stdout:[WRN] OSD_HOST_DOWN: 1 host (2 osds) down 2026-02-24T18:01:28.652 INFO:tasks.workunit.client.0.trial068.stdout: host host0 (root=root,row=row0,rack=rack0) (2 osds) is down 2026-02-24T18:01:28.663 INFO:tasks.workunit.client.0.trial068.stderr:+ ceph health 2026-02-24T18:01:28.663 INFO:tasks.workunit.client.0.trial068.stderr:+ grep '1 host' 2026-02-24T18:01:28.913 INFO:tasks.workunit.client.0.trial068.stdout:HEALTH_WARN noup,noout flag(s) set; 2 osds down; 1 host (2 osds) down 2026-02-24T18:01:28.914 INFO:tasks.workunit.client.0.trial068.stderr:+ ceph health 2026-02-24T18:01:28.914 INFO:tasks.workunit.client.0.trial068.stderr:+ grep '2 osds' 2026-02-24T18:01:29.154 INFO:tasks.workunit.client.0.trial068.stdout:HEALTH_WARN noup,noout flag(s) set; 2 osds down; 1 host (2 osds) down 2026-02-24T18:01:29.154 INFO:tasks.workunit.client.0.trial068.stderr:+ ceph health detail 2026-02-24T18:01:29.154 INFO:tasks.workunit.client.0.trial068.stderr:+ grep osd.0 2026-02-24T18:01:29.435 INFO:tasks.workunit.client.0.trial068.stdout: osd.0 (root=root,row=row0,rack=rack0,host=host0) is down 2026-02-24T18:01:29.436 INFO:tasks.workunit.client.0.trial068.stderr:+ ceph health detail 2026-02-24T18:01:29.436 INFO:tasks.workunit.client.0.trial068.stderr:+ grep osd.1 2026-02-24T18:01:29.706 INFO:tasks.workunit.client.0.trial068.stdout: osd.1 (root=root,row=row0,rack=rack0,host=host0) is down 2026-02-24T18:01:29.706 INFO:tasks.workunit.client.0.trial068.stderr:+ ceph osd unset noup 2026-02-24T18:01:30.421 INFO:tasks.ceph.osd.1.trial068.stderr:2026-02-24T18:01:30.412+0000 7f54aa1a3640 -1 osd.1 30 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2026-02-24T18:01:31.321 INFO:tasks.workunit.client.0.trial068.stderr:noup is unset 2026-02-24T18:01:31.335 INFO:tasks.workunit.client.0.trial068.stderr:+ wait_for_healthy 2026-02-24T18:01:31.335 INFO:tasks.workunit.client.0.trial068.stderr:+ ceph health 2026-02-24T18:01:31.336 INFO:tasks.workunit.client.0.trial068.stderr:+ grep down 2026-02-24T18:01:31.579 INFO:tasks.workunit.client.0.trial068.stdout:HEALTH_WARN noout flag(s) set; 1 osds down 2026-02-24T18:01:31.579 INFO:tasks.workunit.client.0.trial068.stderr:+ sleep 1 2026-02-24T18:01:32.580 INFO:tasks.workunit.client.0.trial068.stderr:+ ceph health 2026-02-24T18:01:32.580 INFO:tasks.workunit.client.0.trial068.stderr:+ grep down 2026-02-24T18:01:32.833 INFO:tasks.workunit.client.0.trial068.stdout:HEALTH_WARN noout flag(s) set; 1 osds down 2026-02-24T18:01:32.834 INFO:tasks.workunit.client.0.trial068.stderr:+ sleep 1 2026-02-24T18:01:33.835 INFO:tasks.workunit.client.0.trial068.stderr:+ ceph health 2026-02-24T18:01:33.835 INFO:tasks.workunit.client.0.trial068.stderr:+ grep down 2026-02-24T18:01:34.076 INFO:tasks.workunit.client.0.trial068.stdout:HEALTH_WARN noout flag(s) set; 1 osds down 2026-02-24T18:01:34.077 INFO:tasks.workunit.client.0.trial068.stderr:+ sleep 1 2026-02-24T18:01:34.212 INFO:tasks.ceph.osd.0.trial068.stderr:2026-02-24T18:01:34.204+0000 7f75afc33640 -1 osd.0 32 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2026-02-24T18:01:35.078 INFO:tasks.workunit.client.0.trial068.stderr:+ ceph health 2026-02-24T18:01:35.078 INFO:tasks.workunit.client.0.trial068.stderr:+ grep down 2026-02-24T18:01:35.317 INFO:tasks.workunit.client.0.trial068.stderr:+ test_mark_two_osds_same_rack_down 2026-02-24T18:01:35.317 INFO:tasks.workunit.client.0.trial068.stderr:+ ceph osd set noup 2026-02-24T18:01:37.333 INFO:tasks.workunit.client.0.trial068.stderr:noup is set 2026-02-24T18:01:37.344 INFO:tasks.workunit.client.0.trial068.stderr:+ ceph osd down osd.8 osd.9 2026-02-24T18:01:37.498 INFO:tasks.ceph.mon.a.trial068.stderr:2026-02-24T18:01:37.488+0000 7fa3e05d4640 -1 mon.a@0(leader).osd e36 definitely_dead 0 2026-02-24T18:01:38.391 INFO:tasks.ceph.mon.a.trial068.stderr:2026-02-24T18:01:38.384+0000 7fa3e05d4640 -1 mon.a@0(leader).osd e37 definitely_dead 0 2026-02-24T18:01:38.391 INFO:tasks.workunit.client.0.trial068.stderr:osd.8 is already down. osd.9 is already down. 2026-02-24T18:01:38.402 INFO:tasks.workunit.client.0.trial068.stderr:+ ceph health detail 2026-02-24T18:01:38.424 INFO:tasks.ceph.osd.8.trial068.stderr:2026-02-24T18:01:38.416+0000 7f75c7328640 -1 osd.8 37 osdmap NOUP flag is set, waiting for it to clear 2026-02-24T18:01:38.671 INFO:tasks.workunit.client.0.trial068.stdout:HEALTH_WARN noup,noout flag(s) set; 2 osds down; 1 host (2 osds) down; 1 rack (2 osds) down; 1 row (2 osds) down 2026-02-24T18:01:38.671 INFO:tasks.workunit.client.0.trial068.stdout:[WRN] OSDMAP_FLAGS: noup,noout flag(s) set 2026-02-24T18:01:38.671 INFO:tasks.workunit.client.0.trial068.stdout:[WRN] OSD_DOWN: 2 osds down 2026-02-24T18:01:38.671 INFO:tasks.workunit.client.0.trial068.stdout: osd.8 (root=root,row=row1,rack=rack2,host=host4) is down 2026-02-24T18:01:38.671 INFO:tasks.workunit.client.0.trial068.stdout: osd.9 (root=root,row=row1,rack=rack2,host=host4) is down 2026-02-24T18:01:38.671 INFO:tasks.workunit.client.0.trial068.stdout:[WRN] OSD_HOST_DOWN: 1 host (2 osds) down 2026-02-24T18:01:38.672 INFO:tasks.workunit.client.0.trial068.stdout: host host4 (root=root,row=row1,rack=rack2) (2 osds) is down 2026-02-24T18:01:38.672 INFO:tasks.workunit.client.0.trial068.stdout:[WRN] OSD_RACK_DOWN: 1 rack (2 osds) down 2026-02-24T18:01:38.672 INFO:tasks.workunit.client.0.trial068.stdout: rack rack2 (root=root,row=row1) (2 osds) is down 2026-02-24T18:01:38.672 INFO:tasks.workunit.client.0.trial068.stdout:[WRN] OSD_ROW_DOWN: 1 row (2 osds) down 2026-02-24T18:01:38.672 INFO:tasks.workunit.client.0.trial068.stdout: row row1 (root=root) (2 osds) is down 2026-02-24T18:01:38.681 INFO:tasks.workunit.client.0.trial068.stderr:+ ceph health 2026-02-24T18:01:38.681 INFO:tasks.workunit.client.0.trial068.stderr:+ grep '1 host' 2026-02-24T18:01:38.936 INFO:tasks.workunit.client.0.trial068.stdout:HEALTH_WARN noup,noout flag(s) set; 2 osds down; 1 host (2 osds) down; 1 rack (2 osds) down; 1 row (2 osds) down 2026-02-24T18:01:38.936 INFO:tasks.workunit.client.0.trial068.stderr:+ ceph health 2026-02-24T18:01:38.936 INFO:tasks.workunit.client.0.trial068.stderr:+ grep '1 rack' 2026-02-24T18:01:39.183 INFO:tasks.workunit.client.0.trial068.stdout:HEALTH_WARN noup,noout flag(s) set; 2 osds down; 1 host (2 osds) down; 1 rack (2 osds) down; 1 row (2 osds) down 2026-02-24T18:01:39.183 INFO:tasks.workunit.client.0.trial068.stderr:+ ceph health 2026-02-24T18:01:39.183 INFO:tasks.workunit.client.0.trial068.stderr:+ grep '1 row' 2026-02-24T18:01:39.340 INFO:tasks.ceph.osd.8.trial068.stderr:2026-02-24T18:01:39.332+0000 7f75b5eac640 -1 osd.8 38 osdmap NOUP flag is set, waiting for it to clear 2026-02-24T18:01:39.420 INFO:tasks.workunit.client.0.trial068.stdout:HEALTH_WARN noup,noout flag(s) set; 2 osds down; 1 host (2 osds) down; 1 rack (2 osds) down; 1 row (2 osds) down 2026-02-24T18:01:39.421 INFO:tasks.workunit.client.0.trial068.stderr:+ ceph health 2026-02-24T18:01:39.421 INFO:tasks.workunit.client.0.trial068.stderr:+ grep '2 osds' 2026-02-24T18:01:39.669 INFO:tasks.workunit.client.0.trial068.stdout:HEALTH_WARN noup,noout flag(s) set; 2 osds down; 1 host (2 osds) down; 1 rack (2 osds) down; 1 row (2 osds) down 2026-02-24T18:01:39.669 INFO:tasks.workunit.client.0.trial068.stderr:+ ceph health detail 2026-02-24T18:01:39.669 INFO:tasks.workunit.client.0.trial068.stderr:+ grep osd.8 2026-02-24T18:01:39.937 INFO:tasks.workunit.client.0.trial068.stdout: osd.8 (root=root,row=row1,rack=rack2,host=host4) is down 2026-02-24T18:01:39.938 INFO:tasks.workunit.client.0.trial068.stderr:+ ceph health detail 2026-02-24T18:01:39.938 INFO:tasks.workunit.client.0.trial068.stderr:+ grep osd.9 2026-02-24T18:01:40.216 INFO:tasks.workunit.client.0.trial068.stdout: osd.9 (root=root,row=row1,rack=rack2,host=host4) is down 2026-02-24T18:01:40.216 INFO:tasks.workunit.client.0.trial068.stderr:+ ceph osd unset noup 2026-02-24T18:01:41.345 INFO:tasks.ceph.osd.8.trial068.stderr:2026-02-24T18:01:41.336+0000 7f75be1f6640 -1 osd.8 39 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2026-02-24T18:01:42.316 INFO:tasks.ceph.osd.9.trial068.stderr:2026-02-24T18:01:42.308+0000 7f0671e2a640 -1 osd.9 39 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2026-02-24T18:01:42.345 INFO:tasks.workunit.client.0.trial068.stderr:noup is unset 2026-02-24T18:01:42.357 INFO:tasks.workunit.client.0.trial068.stderr:+ wait_for_healthy 2026-02-24T18:01:42.358 INFO:tasks.workunit.client.0.trial068.stderr:+ ceph health 2026-02-24T18:01:42.358 INFO:tasks.workunit.client.0.trial068.stderr:+ grep down 2026-02-24T18:01:42.605 INFO:tasks.workunit.client.0.trial068.stderr:+ test_mark_all_but_last_osds_down 2026-02-24T18:01:42.605 INFO:tasks.workunit.client.0.trial068.stderr:+ ceph osd set noup 2026-02-24T18:01:44.348 INFO:tasks.workunit.client.0.trial068.stderr:noup is set 2026-02-24T18:01:44.362 INFO:tasks.workunit.client.0.trial068.stderr:++ ceph osd ls 2026-02-24T18:01:44.362 INFO:tasks.workunit.client.0.trial068.stderr:++ sed '$d' 2026-02-24T18:01:44.581 INFO:tasks.workunit.client.0.trial068.stderr:+ ceph osd down 0 1 2 3 4 5 6 7 8 2026-02-24T18:01:44.742 INFO:tasks.ceph.mon.a.trial068.stderr:2026-02-24T18:01:44.732+0000 7fa3e05d4640 -1 mon.a@0(leader).osd e42 definitely_dead 0 2026-02-24T18:01:45.403 INFO:tasks.ceph.mon.a.trial068.stderr:2026-02-24T18:01:45.396+0000 7fa3e05d4640 -1 mon.a@0(leader).osd e43 definitely_dead 0 2026-02-24T18:01:45.403 INFO:tasks.workunit.client.0.trial068.stderr:osd.0 is already down. osd.1 is already down. osd.2 is already down. osd.3 is already down. osd.4 is already down. osd.5 is already down. osd.6 is already down. osd.7 is already down. osd.8 is already down. 2026-02-24T18:01:45.414 INFO:tasks.workunit.client.0.trial068.stderr:+ ceph health detail 2026-02-24T18:01:45.450 INFO:tasks.ceph.osd.5.trial068.stderr:2026-02-24T18:01:45.440+0000 7f1720a10640 -1 osd.5 43 osdmap NOUP flag is set, waiting for it to clear 2026-02-24T18:01:45.672 INFO:tasks.workunit.client.0.trial068.stdout:HEALTH_WARN noup,noout flag(s) set; 9 osds down; 4 hosts (8 osds) down; 2 racks (8 osds) down; 1 row (8 osds) down 2026-02-24T18:01:45.672 INFO:tasks.workunit.client.0.trial068.stdout:[WRN] OSDMAP_FLAGS: noup,noout flag(s) set 2026-02-24T18:01:45.672 INFO:tasks.workunit.client.0.trial068.stdout:[WRN] OSD_DOWN: 9 osds down 2026-02-24T18:01:45.672 INFO:tasks.workunit.client.0.trial068.stdout: osd.0 (root=root,row=row0,rack=rack0,host=host0) is down 2026-02-24T18:01:45.672 INFO:tasks.workunit.client.0.trial068.stdout: osd.1 (root=root,row=row0,rack=rack0,host=host0) is down 2026-02-24T18:01:45.672 INFO:tasks.workunit.client.0.trial068.stdout: osd.2 (root=root,row=row0,rack=rack0,host=host1) is down 2026-02-24T18:01:45.673 INFO:tasks.workunit.client.0.trial068.stdout: osd.3 (root=root,row=row0,rack=rack0,host=host1) is down 2026-02-24T18:01:45.673 INFO:tasks.workunit.client.0.trial068.stdout: osd.4 (root=root,row=row0,rack=rack1,host=host2) is down 2026-02-24T18:01:45.673 INFO:tasks.workunit.client.0.trial068.stdout: osd.5 (root=root,row=row0,rack=rack1,host=host2) is down 2026-02-24T18:01:45.673 INFO:tasks.workunit.client.0.trial068.stdout: osd.6 (root=root,row=row0,rack=rack1,host=host3) is down 2026-02-24T18:01:45.673 INFO:tasks.workunit.client.0.trial068.stdout: osd.7 (root=root,row=row0,rack=rack1,host=host3) is down 2026-02-24T18:01:45.673 INFO:tasks.workunit.client.0.trial068.stdout: osd.8 (root=root,row=row1,rack=rack2,host=host4) is down 2026-02-24T18:01:45.673 INFO:tasks.workunit.client.0.trial068.stdout:[WRN] OSD_HOST_DOWN: 4 hosts (8 osds) down 2026-02-24T18:01:45.673 INFO:tasks.workunit.client.0.trial068.stdout: host host0 (root=root,row=row0,rack=rack0) (2 osds) is down 2026-02-24T18:01:45.673 INFO:tasks.workunit.client.0.trial068.stdout: host host1 (root=root,row=row0,rack=rack0) (2 osds) is down 2026-02-24T18:01:45.673 INFO:tasks.workunit.client.0.trial068.stdout: host host2 (root=root,row=row0,rack=rack1) (2 osds) is down 2026-02-24T18:01:45.673 INFO:tasks.workunit.client.0.trial068.stdout: host host3 (root=root,row=row0,rack=rack1) (2 osds) is down 2026-02-24T18:01:45.673 INFO:tasks.workunit.client.0.trial068.stdout:[WRN] OSD_RACK_DOWN: 2 racks (8 osds) down 2026-02-24T18:01:45.673 INFO:tasks.workunit.client.0.trial068.stdout: rack rack0 (root=root,row=row0) (4 osds) is down 2026-02-24T18:01:45.673 INFO:tasks.workunit.client.0.trial068.stdout: rack rack1 (root=root,row=row0) (4 osds) is down 2026-02-24T18:01:45.673 INFO:tasks.workunit.client.0.trial068.stdout:[WRN] OSD_ROW_DOWN: 1 row (8 osds) down 2026-02-24T18:01:45.673 INFO:tasks.workunit.client.0.trial068.stdout: row row0 (root=root) (8 osds) is down 2026-02-24T18:01:45.682 INFO:tasks.workunit.client.0.trial068.stderr:+ ceph health 2026-02-24T18:01:45.682 INFO:tasks.workunit.client.0.trial068.stderr:+ grep '1 row' 2026-02-24T18:01:45.938 INFO:tasks.workunit.client.0.trial068.stdout:HEALTH_WARN noup,noout flag(s) set; 9 osds down; 4 hosts (8 osds) down; 2 racks (8 osds) down; 1 row (8 osds) down 2026-02-24T18:01:45.939 INFO:tasks.workunit.client.0.trial068.stderr:+ ceph health 2026-02-24T18:01:45.939 INFO:tasks.workunit.client.0.trial068.stderr:+ grep '2 racks' 2026-02-24T18:01:46.185 INFO:tasks.workunit.client.0.trial068.stdout:HEALTH_WARN noup,noout flag(s) set; 9 osds down; 4 hosts (8 osds) down; 2 racks (8 osds) down; 1 row (8 osds) down 2026-02-24T18:01:46.186 INFO:tasks.workunit.client.0.trial068.stderr:+ ceph health 2026-02-24T18:01:46.186 INFO:tasks.workunit.client.0.trial068.stderr:+ grep '4 hosts' 2026-02-24T18:01:46.224 INFO:tasks.ceph.osd.1.trial068.stderr:2026-02-24T18:01:46.216+0000 7f54aeacc640 -1 osd.1 43 osdmap NOUP flag is set, waiting for it to clear 2026-02-24T18:01:46.285 INFO:tasks.ceph.osd.0.trial068.stderr:2026-02-24T18:01:46.276+0000 7f75b8564640 -1 osd.0 43 osdmap NOUP flag is set, waiting for it to clear 2026-02-24T18:01:46.351 INFO:tasks.ceph.osd.5.trial068.stderr:2026-02-24T18:01:46.344+0000 7f170f594640 -1 osd.5 44 osdmap NOUP flag is set, waiting for it to clear 2026-02-24T18:01:46.352 INFO:tasks.ceph.osd.1.trial068.stderr:2026-02-24T18:01:46.344+0000 7f54a1e59640 -1 osd.1 44 osdmap NOUP flag is set, waiting for it to clear 2026-02-24T18:01:46.352 INFO:tasks.ceph.osd.0.trial068.stderr:2026-02-24T18:01:46.344+0000 7f75a78e9640 -1 osd.0 44 osdmap NOUP flag is set, waiting for it to clear 2026-02-24T18:01:46.398 INFO:tasks.ceph.osd.8.trial068.stderr:2026-02-24T18:01:46.388+0000 7f75c6b27640 -1 osd.8 44 osdmap NOUP flag is set, waiting for it to clear 2026-02-24T18:01:46.430 INFO:tasks.workunit.client.0.trial068.stdout:HEALTH_WARN noup,noout flag(s) set; 9 osds down; 4 hosts (8 osds) down; 2 racks (8 osds) down; 1 row (8 osds) down 2026-02-24T18:01:46.430 INFO:tasks.workunit.client.0.trial068.stderr:+ ceph health 2026-02-24T18:01:46.431 INFO:tasks.workunit.client.0.trial068.stderr:+ grep '9 osds' 2026-02-24T18:01:46.559 INFO:tasks.ceph.osd.7.trial068.stderr:2026-02-24T18:01:46.552+0000 7fb8086ca640 -1 osd.7 44 osdmap NOUP flag is set, waiting for it to clear 2026-02-24T18:01:46.670 INFO:tasks.workunit.client.0.trial068.stdout:HEALTH_WARN noup,noout flag(s) set; 9 osds down; 4 hosts (8 osds) down; 2 racks (8 osds) down; 1 row (8 osds) down 2026-02-24T18:01:46.671 INFO:tasks.workunit.client.0.trial068.stderr:+ ceph osd unset noup 2026-02-24T18:01:46.739 INFO:tasks.ceph.osd.2.trial068.stderr:2026-02-24T18:01:46.732+0000 7fc3bc80e640 -1 osd.2 44 osdmap NOUP flag is set, waiting for it to clear 2026-02-24T18:01:46.880 INFO:tasks.ceph.osd.6.trial068.stderr:2026-02-24T18:01:46.872+0000 7fef2e1e7640 -1 osd.6 44 osdmap NOUP flag is set, waiting for it to clear 2026-02-24T18:01:47.010 INFO:tasks.ceph.osd.4.trial068.stderr:2026-02-24T18:01:47.004+0000 7fcc575fc640 -1 osd.4 44 osdmap NOUP flag is set, waiting for it to clear 2026-02-24T18:01:47.116 INFO:tasks.ceph.osd.3.trial068.stderr:2026-02-24T18:01:47.108+0000 7fd404803640 -1 osd.3 44 osdmap NOUP flag is set, waiting for it to clear 2026-02-24T18:01:47.360 INFO:tasks.ceph.osd.0.trial068.stderr:2026-02-24T18:01:47.352+0000 7f75afc33640 -1 osd.0 45 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2026-02-24T18:01:47.360 INFO:tasks.ceph.osd.2.trial068.stderr:2026-02-24T18:01:47.352+0000 7fc3b76e4640 -1 osd.2 45 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2026-02-24T18:01:47.361 INFO:tasks.ceph.osd.1.trial068.stderr:2026-02-24T18:01:47.352+0000 7f54aa1a3640 -1 osd.1 45 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2026-02-24T18:01:47.361 INFO:tasks.ceph.osd.3.trial068.stderr:2026-02-24T18:01:47.352+0000 7fd3fb6d1640 -1 osd.3 45 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2026-02-24T18:01:47.361 INFO:tasks.ceph.osd.4.trial068.stderr:2026-02-24T18:01:47.352+0000 7fcc4e4ca640 -1 osd.4 45 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2026-02-24T18:01:47.362 INFO:tasks.ceph.osd.5.trial068.stderr:2026-02-24T18:01:47.352+0000 7f17178de640 -1 osd.5 45 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2026-02-24T18:01:47.362 INFO:tasks.ceph.osd.6.trial068.stderr:2026-02-24T18:01:47.352+0000 7fef250b5640 -1 osd.6 45 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2026-02-24T18:01:47.362 INFO:tasks.ceph.osd.7.trial068.stderr:2026-02-24T18:01:47.352+0000 7fb8035a0640 -1 osd.7 45 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2026-02-24T18:01:47.363 INFO:tasks.ceph.osd.8.trial068.stderr:2026-02-24T18:01:47.352+0000 7f75be1f6640 -1 osd.8 45 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2026-02-24T18:01:48.359 INFO:tasks.workunit.client.0.trial068.stderr:noup is unset 2026-02-24T18:01:48.374 INFO:tasks.workunit.client.0.trial068.stderr:+ wait_for_healthy 2026-02-24T18:01:48.374 INFO:tasks.workunit.client.0.trial068.stderr:+ ceph health 2026-02-24T18:01:48.374 INFO:tasks.workunit.client.0.trial068.stderr:+ grep down 2026-02-24T18:01:48.613 INFO:tasks.workunit.client.0.trial068.stderr:+ test_mark_two_osds_same_host_down_with_classes 2026-02-24T18:01:48.614 INFO:tasks.workunit.client.0.trial068.stderr:+ ceph osd set noup 2026-02-24T18:01:50.361 INFO:tasks.workunit.client.0.trial068.stderr:noup is set 2026-02-24T18:01:50.374 INFO:tasks.workunit.client.0.trial068.stderr:+ ceph osd crush set-device-class ssd osd.0 osd.2 osd.4 osd.6 osd.8 2026-02-24T18:01:52.367 INFO:tasks.workunit.client.0.trial068.stderr:osd.0 already set to class ssd. set-device-class item id 0 name 'osd.0' device_class 'ssd': no change. osd.2 already set to class ssd. set-device-class item id 2 name 'osd.2' device_class 'ssd': no change. osd.4 already set to class ssd. set-device-class item id 4 name 'osd.4' device_class 'ssd': no change. osd.6 already set to class ssd. set-device-class item id 6 name 'osd.6' device_class 'ssd': no change. osd.8 already set to class ssd. set-device-class item id 8 name 'osd.8' device_class 'ssd': no change. set osd(s) to class 'ssd' 2026-02-24T18:01:52.379 INFO:tasks.workunit.client.0.trial068.stderr:+ ceph osd crush set-device-class hdd osd.1 osd.3 osd.5 osd.7 osd.9 2026-02-24T18:01:54.372 INFO:tasks.workunit.client.0.trial068.stderr:osd.1 already set to class hdd. set-device-class item id 1 name 'osd.1' device_class 'hdd': no change. osd.3 already set to class hdd. set-device-class item id 3 name 'osd.3' device_class 'hdd': no change. osd.5 already set to class hdd. set-device-class item id 5 name 'osd.5' device_class 'hdd': no change. osd.7 already set to class hdd. set-device-class item id 7 name 'osd.7' device_class 'hdd': no change. osd.9 already set to class hdd. set-device-class item id 9 name 'osd.9' device_class 'hdd': no change. set osd(s) to class 'hdd' 2026-02-24T18:01:54.385 INFO:tasks.workunit.client.0.trial068.stderr:+ ceph osd down osd.0 osd.1 2026-02-24T18:01:54.536 INFO:tasks.ceph.mon.a.trial068.stderr:2026-02-24T18:01:54.528+0000 7fa3e05d4640 -1 mon.a@0(leader).osd e52 definitely_dead 0 2026-02-24T18:01:55.427 INFO:tasks.ceph.mon.a.trial068.stderr:2026-02-24T18:01:55.420+0000 7fa3e05d4640 -1 mon.a@0(leader).osd e53 definitely_dead 0 2026-02-24T18:01:55.427 INFO:tasks.workunit.client.0.trial068.stderr:osd.0 is already down. osd.1 is already down. 2026-02-24T18:01:55.437 INFO:tasks.workunit.client.0.trial068.stderr:+ ceph health detail 2026-02-24T18:01:55.705 INFO:tasks.workunit.client.0.trial068.stdout:HEALTH_WARN noup,noout flag(s) set; 2 osds down; 1 host (2 osds) down 2026-02-24T18:01:55.706 INFO:tasks.workunit.client.0.trial068.stdout:[WRN] OSDMAP_FLAGS: noup,noout flag(s) set 2026-02-24T18:01:55.706 INFO:tasks.workunit.client.0.trial068.stdout:[WRN] OSD_DOWN: 2 osds down 2026-02-24T18:01:55.706 INFO:tasks.workunit.client.0.trial068.stdout: osd.0 (root=root,row=row0,rack=rack0,host=host0) is down 2026-02-24T18:01:55.706 INFO:tasks.workunit.client.0.trial068.stdout: osd.1 (root=root,row=row0,rack=rack0,host=host0) is down 2026-02-24T18:01:55.706 INFO:tasks.workunit.client.0.trial068.stdout:[WRN] OSD_HOST_DOWN: 1 host (2 osds) down 2026-02-24T18:01:55.706 INFO:tasks.workunit.client.0.trial068.stdout: host host0 (root=root,row=row0,rack=rack0) (2 osds) is down 2026-02-24T18:01:55.716 INFO:tasks.workunit.client.0.trial068.stderr:+ ceph health 2026-02-24T18:01:55.716 INFO:tasks.workunit.client.0.trial068.stderr:+ grep '1 host' 2026-02-24T18:01:55.969 INFO:tasks.workunit.client.0.trial068.stdout:HEALTH_WARN noup,noout flag(s) set; 2 osds down; 1 host (2 osds) down 2026-02-24T18:01:55.970 INFO:tasks.workunit.client.0.trial068.stderr:+ ceph health 2026-02-24T18:01:55.970 INFO:tasks.workunit.client.0.trial068.stderr:+ grep '2 osds' 2026-02-24T18:01:56.205 INFO:tasks.workunit.client.0.trial068.stdout:HEALTH_WARN noup,noout flag(s) set; 2 osds down; 1 host (2 osds) down 2026-02-24T18:01:56.206 INFO:tasks.workunit.client.0.trial068.stderr:+ ceph health detail 2026-02-24T18:01:56.206 INFO:tasks.workunit.client.0.trial068.stderr:+ grep osd.0 2026-02-24T18:01:56.478 INFO:tasks.workunit.client.0.trial068.stdout: osd.0 (root=root,row=row0,rack=rack0,host=host0) is down 2026-02-24T18:01:56.479 INFO:tasks.workunit.client.0.trial068.stderr:+ ceph health detail 2026-02-24T18:01:56.479 INFO:tasks.workunit.client.0.trial068.stderr:+ grep osd.1 2026-02-24T18:01:56.757 INFO:tasks.workunit.client.0.trial068.stdout: osd.1 (root=root,row=row0,rack=rack0,host=host0) is down 2026-02-24T18:01:56.757 INFO:tasks.workunit.client.0.trial068.stderr:+ ceph osd unset noup 2026-02-24T18:01:57.255 INFO:tasks.ceph.osd.0.trial068.stderr:2026-02-24T18:01:57.248+0000 7f75b8d65640 -1 osd.0 54 osdmap NOUP flag is set, waiting for it to clear 2026-02-24T18:01:57.380 INFO:tasks.ceph.osd.0.trial068.stderr:2026-02-24T18:01:57.372+0000 7f75afc33640 -1 osd.0 55 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2026-02-24T18:01:58.210 INFO:tasks.ceph.osd.1.trial068.stderr:2026-02-24T18:01:58.200+0000 7f54aa1a3640 -1 osd.1 55 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2026-02-24T18:01:58.380 INFO:tasks.workunit.client.0.trial068.stderr:noup is unset 2026-02-24T18:01:58.392 INFO:tasks.workunit.client.0.trial068.stderr:+ wait_for_healthy 2026-02-24T18:01:58.392 INFO:tasks.workunit.client.0.trial068.stderr:+ ceph health 2026-02-24T18:01:58.392 INFO:tasks.workunit.client.0.trial068.stderr:+ grep down 2026-02-24T18:01:58.638 INFO:tasks.workunit.client.0.trial068.stderr:+ exit 0 2026-02-24T18:01:58.638 INFO:teuthology.orchestra.run:Running command with timeout 3600 2026-02-24T18:01:58.638 DEBUG:teuthology.orchestra.run.trial068:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2026-02-24T18:01:58.694 INFO:tasks.workunit:Stopping ['rados/test_health_warnings.sh'] on client.0... 2026-02-24T18:01:58.694 DEBUG:teuthology.orchestra.run.trial068:> sudo rm -rf -- /home/ubuntu/cephtest/workunits.list.client.0 /home/ubuntu/cephtest/clone.client.0 2026-02-24T18:01:59.053 DEBUG:teuthology.parallel:result is None 2026-02-24T18:01:59.053 DEBUG:teuthology.orchestra.run.trial068:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0 2026-02-24T18:01:59.061 INFO:tasks.workunit:Deleted dir /home/ubuntu/cephtest/mnt.0/client.0 2026-02-24T18:01:59.061 DEBUG:teuthology.orchestra.run.trial068:> rmdir -- /home/ubuntu/cephtest/mnt.0 2026-02-24T18:01:59.110 INFO:tasks.workunit:Deleted artificial mount point /home/ubuntu/cephtest/mnt.0/client.0 2026-02-24T18:01:59.110 DEBUG:teuthology.run_tasks:Unwinding manager ceph 2026-02-24T18:01:59.125 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2026-02-24T18:01:59.125 DEBUG:teuthology.orchestra.run.trial068:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-24T18:01:59.307 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-24T18:01:59.307 INFO:teuthology.orchestra.run.trial068.stderr:dumped all 2026-02-24T18:01:59.317 INFO:teuthology.orchestra.run.trial068.stdout:{"pg_ready":true,"pg_map":{"version":82,"stamp":"2026-02-24T18:01:58.376403+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":10,"num_per_pool_osds":10,"num_per_pool_omap_osds":8,"kb":1048576000,"kb_used":275488,"kb_used_data":5888,"kb_used_omap":246,"kb_used_meta":269193,"kb_avail":1048300512,"statfs":{"total":1073741824000,"available":1073459724288,"internally_reserved":0,"allocated":6029312,"data_stored":2819844,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":252027,"internal_metadata":275654533},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.007608"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":69,"reported_epoch":54,"state":"active+clean","last_fresh":"2026-02-24T18:01:56.374147+0000","last_change":"2026-02-24T18:01:49.359327+0000","last_active":"2026-02-24T18:01:56.374147+0000","last_peered":"2026-02-24T18:01:56.374147+0000","last_clean":"2026-02-24T18:01:56.374147+0000","last_became_active":"2026-02-24T18:01:49.359085+0000","last_became_peered":"2026-02-24T18:01:49.359085+0000","last_unstale":"2026-02-24T18:01:56.374147+0000","last_undegraded":"2026-02-24T18:01:56.374147+0000","last_fullsized":"2026-02-24T18:01:56.374147+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-24T18:00:52.474019+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-24T18:00:52.474019+0000","last_clean_scrub_stamp":"2026-02-24T18:00:52.474019+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-25T23:52:48.720387+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000255505,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":88,"reported_epoch":54,"state":"active+clean","last_fresh":"2026-02-24T18:01:56.374068+0000","last_change":"2026-02-24T18:01:49.359276+0000","last_active":"2026-02-24T18:01:56.374068+0000","last_peered":"2026-02-24T18:01:56.374068+0000","last_clean":"2026-02-24T18:01:56.374068+0000","last_became_active":"2026-02-24T18:01:49.359064+0000","last_became_peered":"2026-02-24T18:01:49.359064+0000","last_unstale":"2026-02-24T18:01:56.374068+0000","last_undegraded":"2026-02-24T18:01:56.374068+0000","last_fullsized":"2026-02-24T18:01:56.374068+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-24T18:00:52.474019+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-24T18:00:52.474019+0000","last_clean_scrub_stamp":"2026-02-24T18:00:52.474019+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-26T05:08:30.898058+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00029952599999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":96,"reported_epoch":55,"state":"active+undersized","last_fresh":"2026-02-24T18:01:57.376232+0000","last_change":"2026-02-24T18:01:57.376232+0000","last_active":"2026-02-24T18:01:57.376232+0000","last_peered":"2026-02-24T18:01:57.376232+0000","last_clean":"2026-02-24T18:01:54.618236+0000","last_became_active":"2026-02-24T18:01:56.374381+0000","last_became_peered":"2026-02-24T18:01:56.374381+0000","last_unstale":"2026-02-24T18:01:57.376232+0000","last_undegraded":"2026-02-24T18:01:57.376232+0000","last_fullsized":"2026-02-24T18:01:56.374083+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-24T18:00:52.474019+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-24T18:00:52.474019+0000","last_clean_scrub_stamp":"2026-02-24T18:00:52.474019+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-25T18:19:12.949262+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00028598099999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9],"acting":[9],"avail_no_missing":["9"],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":61,"reported_epoch":54,"state":"active+undersized+wait","last_fresh":"2026-02-24T18:01:56.374574+0000","last_change":"2026-02-24T18:01:56.374333+0000","last_active":"2026-02-24T18:01:56.374574+0000","last_peered":"2026-02-24T18:01:56.374574+0000","last_clean":"2026-02-24T18:01:46.156753+0000","last_became_active":"2026-02-24T18:01:56.374333+0000","last_became_peered":"2026-02-24T18:01:56.374333+0000","last_unstale":"2026-02-24T18:01:56.374574+0000","last_undegraded":"2026-02-24T18:01:56.374574+0000","last_fullsized":"2026-02-24T18:01:56.374090+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-24T18:00:52.474019+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-24T18:00:52.474019+0000","last_clean_scrub_stamp":"2026-02-24T18:00:52.474019+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-26T03:42:00.638077+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6],"acting":[6],"avail_no_missing":["6"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.3","version":"18'1","reported_seq":84,"reported_epoch":54,"state":"active+undersized+degraded+wait","last_fresh":"2026-02-24T18:01:56.374616+0000","last_change":"2026-02-24T18:01:56.374449+0000","last_active":"2026-02-24T18:01:56.374616+0000","last_peered":"2026-02-24T18:01:56.374616+0000","last_clean":"2026-02-24T18:01:54.618235+0000","last_became_active":"2026-02-24T18:01:56.374449+0000","last_became_peered":"2026-02-24T18:01:56.374449+0000","last_unstale":"2026-02-24T18:01:56.374616+0000","last_undegraded":"2026-02-24T18:01:56.374200+0000","last_fullsized":"2026-02-24T18:01:56.374158+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-24T18:00:52.474019+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-24T18:00:52.474019+0000","last_clean_scrub_stamp":"2026-02-24T18:00:52.474019+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-25T19:06:51.757172+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[{"shards":"3","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"20'2","reported_seq":92,"reported_epoch":54,"state":"active+undersized+degraded+wait","last_fresh":"2026-02-24T18:01:56.374391+0000","last_change":"2026-02-24T18:01:56.374282+0000","last_active":"2026-02-24T18:01:56.374391+0000","last_peered":"2026-02-24T18:01:56.374391+0000","last_clean":"2026-02-24T18:01:54.618334+0000","last_became_active":"2026-02-24T18:01:56.374282+0000","last_became_peered":"2026-02-24T18:01:56.374282+0000","last_unstale":"2026-02-24T18:01:56.374391+0000","last_undegraded":"2026-02-24T18:01:56.374112+0000","last_fullsized":"2026-02-24T18:01:56.374072+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-24T18:00:52.474019+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-24T18:00:52.474019+0000","last_clean_scrub_stamp":"2026-02-24T18:00:52.474019+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-26T04:49:05.797860+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00073412399999999995,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4],"acting":[4],"avail_no_missing":["4"],"object_location_counts":[{"shards":"4","objects":1}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":69,"reported_epoch":53,"state":"active+clean","last_fresh":"2026-02-24T18:01:55.618250+0000","last_change":"2026-02-24T18:01:49.359419+0000","last_active":"2026-02-24T18:01:55.618250+0000","last_peered":"2026-02-24T18:01:55.618250+0000","last_clean":"2026-02-24T18:01:55.618250+0000","last_became_active":"2026-02-24T18:01:49.359167+0000","last_became_peered":"2026-02-24T18:01:49.359167+0000","last_unstale":"2026-02-24T18:01:55.618250+0000","last_undegraded":"2026-02-24T18:01:55.618250+0000","last_fullsized":"2026-02-24T18:01:55.618250+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-24T18:00:52.474019+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-24T18:00:52.474019+0000","last_clean_scrub_stamp":"2026-02-24T18:00:52.474019+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-25T19:47:00.731011+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00026963099999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":80,"reported_epoch":53,"state":"active+clean","last_fresh":"2026-02-24T18:01:55.618229+0000","last_change":"2026-02-24T18:01:49.359091+0000","last_active":"2026-02-24T18:01:55.618229+0000","last_peered":"2026-02-24T18:01:55.618229+0000","last_clean":"2026-02-24T18:01:55.618229+0000","last_became_active":"2026-02-24T18:01:49.358904+0000","last_became_peered":"2026-02-24T18:01:49.358904+0000","last_unstale":"2026-02-24T18:01:55.618229+0000","last_undegraded":"2026-02-24T18:01:55.618229+0000","last_fullsized":"2026-02-24T18:01:55.618229+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-24T18:00:52.474019+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-24T18:00:52.474019+0000","last_clean_scrub_stamp":"2026-02-24T18:00:52.474019+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-26T04:25:16.057495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9],"acting":[5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":12,"acting":12,"num_store_stats":9}],"osd_stats":[{"osd":9,"up_from":40,"seq":171798691856,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":27560,"kb_used_data":600,"kb_used_omap":25,"kb_used_meta":26918,"kb_avail":104830040,"statfs":{"total":107374182400,"available":107345960960,"internally_reserved":0,"allocated":614400,"data_stored":290302,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":25794,"internal_metadata":27564862},"hb_peers":[0,1,2,3,4,5,6,7,8],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":46,"seq":197568495632,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":104857600,"kb_used":27548,"kb_used_data":588,"kb_used_omap":26,"kb_used_meta":26917,"kb_avail":104830052,"statfs":{"total":107374182400,"available":107345973248,"internally_reserved":0,"allocated":602112,"data_stored":282636,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":27091,"internal_metadata":27563565},"hb_peers":[2,3,4,5,6,7,9],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":46,"seq":197568495632,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":27548,"kb_used_data":588,"kb_used_omap":26,"kb_used_meta":26917,"kb_avail":104830052,"statfs":{"total":107374182400,"available":107345973248,"internally_reserved":0,"allocated":602112,"data_stored":282636,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":27086,"internal_metadata":27563570},"hb_peers":[2,3,4,5,6,8,9],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":46,"seq":197568495632,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":27560,"kb_used_data":600,"kb_used_omap":25,"kb_used_meta":26918,"kb_avail":104830040,"statfs":{"total":107374182400,"available":107345960960,"internally_reserved":0,"allocated":614400,"data_stored":290302,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":25788,"internal_metadata":27564868},"hb_peers":[0,1,2,3,4,5,7,8,9],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":46,"seq":197568495632,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":27548,"kb_used_data":588,"kb_used_omap":26,"kb_used_meta":26917,"kb_avail":104830052,"statfs":{"total":107374182400,"available":107345973248,"internally_reserved":0,"allocated":602112,"data_stored":282636,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":27091,"internal_metadata":27563565},"hb_peers":[0,1,2,3,4,6,7,8,9],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":46,"seq":197568495632,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":27564,"kb_used_data":604,"kb_used_omap":28,"kb_used_meta":26915,"kb_avail":104830036,"statfs":{"total":107374182400,"available":107345956864,"internally_reserved":0,"allocated":618496,"data_stored":290321,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":29044,"internal_metadata":27561612},"hb_peers":[0,1,2,3,5,6,7,8,9],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":46,"seq":197568495632,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":27560,"kb_used_data":600,"kb_used_omap":25,"kb_used_meta":26918,"kb_avail":104830040,"statfs":{"total":107374182400,"available":107345960960,"internally_reserved":0,"allocated":614400,"data_stored":290302,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":26437,"internal_metadata":27564219},"hb_peers":[0,1,2,4,5,6,7,8,9],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":46,"seq":197568495632,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":104857600,"kb_used":27560,"kb_used_data":600,"kb_used_omap":22,"kb_used_meta":26921,"kb_avail":104830040,"statfs":{"total":107374182400,"available":107345960960,"internally_reserved":0,"allocated":614400,"data_stored":290302,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":22534,"internal_metadata":27568122},"hb_peers":[3,4,5,6,7,8,9],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":16,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":27540,"kb_used_data":580,"kb_used_omap":26,"kb_used_meta":26917,"kb_avail":104830060,"statfs":{"total":107374182400,"available":107345981440,"internally_reserved":0,"allocated":593920,"data_stored":274921,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":27091,"internal_metadata":27563565},"hb_peers":[2,3,4,5,6,7,8,9],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":46,"seq":197568495632,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":27500,"kb_used_data":540,"kb_used_omap":13,"kb_used_meta":26930,"kb_avail":104830100,"statfs":{"total":107374182400,"available":107346022400,"internally_reserved":0,"allocated":552960,"data_stored":245486,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14071,"internal_metadata":27576585},"hb_peers":[1,2,3,4,5,6,7,8,9],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-24T18:01:59.319 DEBUG:teuthology.orchestra.run.trial068:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-24T18:01:59.476 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-24T18:01:59.476 INFO:teuthology.orchestra.run.trial068.stderr:dumped all 2026-02-24T18:01:59.488 INFO:teuthology.orchestra.run.trial068.stdout:{"pg_ready":true,"pg_map":{"version":84,"stamp":"2026-02-24T18:01:59.468418+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":10,"num_per_pool_osds":10,"num_per_pool_omap_osds":8,"kb":1048576000,"kb_used":275560,"kb_used_data":5960,"kb_used_omap":246,"kb_used_meta":269193,"kb_avail":1048300440,"statfs":{"total":1073741824000,"available":1073459650560,"internally_reserved":0,"allocated":6103040,"data_stored":2872394,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":252027,"internal_metadata":275654533},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.000430"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":69,"reported_epoch":54,"state":"active+clean","last_fresh":"2026-02-24T18:01:56.374147+0000","last_change":"2026-02-24T18:01:49.359327+0000","last_active":"2026-02-24T18:01:56.374147+0000","last_peered":"2026-02-24T18:01:56.374147+0000","last_clean":"2026-02-24T18:01:56.374147+0000","last_became_active":"2026-02-24T18:01:49.359085+0000","last_became_peered":"2026-02-24T18:01:49.359085+0000","last_unstale":"2026-02-24T18:01:56.374147+0000","last_undegraded":"2026-02-24T18:01:56.374147+0000","last_fullsized":"2026-02-24T18:01:56.374147+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-24T18:00:52.474019+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-24T18:00:52.474019+0000","last_clean_scrub_stamp":"2026-02-24T18:00:52.474019+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-25T23:52:48.720387+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000255505,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":88,"reported_epoch":54,"state":"active+clean","last_fresh":"2026-02-24T18:01:56.374068+0000","last_change":"2026-02-24T18:01:49.359276+0000","last_active":"2026-02-24T18:01:56.374068+0000","last_peered":"2026-02-24T18:01:56.374068+0000","last_clean":"2026-02-24T18:01:56.374068+0000","last_became_active":"2026-02-24T18:01:49.359064+0000","last_became_peered":"2026-02-24T18:01:49.359064+0000","last_unstale":"2026-02-24T18:01:56.374068+0000","last_undegraded":"2026-02-24T18:01:56.374068+0000","last_fullsized":"2026-02-24T18:01:56.374068+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-24T18:00:52.474019+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-24T18:00:52.474019+0000","last_clean_scrub_stamp":"2026-02-24T18:00:52.474019+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-26T05:08:30.898058+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00029952599999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":96,"reported_epoch":55,"state":"active+undersized","last_fresh":"2026-02-24T18:01:57.376232+0000","last_change":"2026-02-24T18:01:57.376232+0000","last_active":"2026-02-24T18:01:57.376232+0000","last_peered":"2026-02-24T18:01:57.376232+0000","last_clean":"2026-02-24T18:01:54.618236+0000","last_became_active":"2026-02-24T18:01:56.374381+0000","last_became_peered":"2026-02-24T18:01:56.374381+0000","last_unstale":"2026-02-24T18:01:57.376232+0000","last_undegraded":"2026-02-24T18:01:57.376232+0000","last_fullsized":"2026-02-24T18:01:56.374083+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-24T18:00:52.474019+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-24T18:00:52.474019+0000","last_clean_scrub_stamp":"2026-02-24T18:00:52.474019+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-25T18:19:12.949262+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00028598099999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9],"acting":[9],"avail_no_missing":["9"],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":61,"reported_epoch":54,"state":"active+undersized+wait","last_fresh":"2026-02-24T18:01:56.374574+0000","last_change":"2026-02-24T18:01:56.374333+0000","last_active":"2026-02-24T18:01:56.374574+0000","last_peered":"2026-02-24T18:01:56.374574+0000","last_clean":"2026-02-24T18:01:46.156753+0000","last_became_active":"2026-02-24T18:01:56.374333+0000","last_became_peered":"2026-02-24T18:01:56.374333+0000","last_unstale":"2026-02-24T18:01:56.374574+0000","last_undegraded":"2026-02-24T18:01:56.374574+0000","last_fullsized":"2026-02-24T18:01:56.374090+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-24T18:00:52.474019+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-24T18:00:52.474019+0000","last_clean_scrub_stamp":"2026-02-24T18:00:52.474019+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-26T03:42:00.638077+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6],"acting":[6],"avail_no_missing":["6"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.3","version":"18'1","reported_seq":84,"reported_epoch":54,"state":"active+undersized+degraded+wait","last_fresh":"2026-02-24T18:01:56.374616+0000","last_change":"2026-02-24T18:01:56.374449+0000","last_active":"2026-02-24T18:01:56.374616+0000","last_peered":"2026-02-24T18:01:56.374616+0000","last_clean":"2026-02-24T18:01:54.618235+0000","last_became_active":"2026-02-24T18:01:56.374449+0000","last_became_peered":"2026-02-24T18:01:56.374449+0000","last_unstale":"2026-02-24T18:01:56.374616+0000","last_undegraded":"2026-02-24T18:01:56.374200+0000","last_fullsized":"2026-02-24T18:01:56.374158+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-24T18:00:52.474019+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-24T18:00:52.474019+0000","last_clean_scrub_stamp":"2026-02-24T18:00:52.474019+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-25T19:06:51.757172+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[{"shards":"3","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"20'2","reported_seq":92,"reported_epoch":54,"state":"active+undersized+degraded+wait","last_fresh":"2026-02-24T18:01:56.374391+0000","last_change":"2026-02-24T18:01:56.374282+0000","last_active":"2026-02-24T18:01:56.374391+0000","last_peered":"2026-02-24T18:01:56.374391+0000","last_clean":"2026-02-24T18:01:54.618334+0000","last_became_active":"2026-02-24T18:01:56.374282+0000","last_became_peered":"2026-02-24T18:01:56.374282+0000","last_unstale":"2026-02-24T18:01:56.374391+0000","last_undegraded":"2026-02-24T18:01:56.374112+0000","last_fullsized":"2026-02-24T18:01:56.374072+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-24T18:00:52.474019+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-24T18:00:52.474019+0000","last_clean_scrub_stamp":"2026-02-24T18:00:52.474019+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-26T04:49:05.797860+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00073412399999999995,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4],"acting":[4],"avail_no_missing":["4"],"object_location_counts":[{"shards":"4","objects":1}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":69,"reported_epoch":53,"state":"active+clean","last_fresh":"2026-02-24T18:01:55.618250+0000","last_change":"2026-02-24T18:01:49.359419+0000","last_active":"2026-02-24T18:01:55.618250+0000","last_peered":"2026-02-24T18:01:55.618250+0000","last_clean":"2026-02-24T18:01:55.618250+0000","last_became_active":"2026-02-24T18:01:49.359167+0000","last_became_peered":"2026-02-24T18:01:49.359167+0000","last_unstale":"2026-02-24T18:01:55.618250+0000","last_undegraded":"2026-02-24T18:01:55.618250+0000","last_fullsized":"2026-02-24T18:01:55.618250+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-24T18:00:52.474019+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-24T18:00:52.474019+0000","last_clean_scrub_stamp":"2026-02-24T18:00:52.474019+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-25T19:47:00.731011+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00026963099999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":80,"reported_epoch":53,"state":"active+clean","last_fresh":"2026-02-24T18:01:55.618229+0000","last_change":"2026-02-24T18:01:49.359091+0000","last_active":"2026-02-24T18:01:55.618229+0000","last_peered":"2026-02-24T18:01:55.618229+0000","last_clean":"2026-02-24T18:01:55.618229+0000","last_became_active":"2026-02-24T18:01:49.358904+0000","last_became_peered":"2026-02-24T18:01:49.358904+0000","last_unstale":"2026-02-24T18:01:55.618229+0000","last_undegraded":"2026-02-24T18:01:55.618229+0000","last_fullsized":"2026-02-24T18:01:55.618229+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-24T18:00:52.474019+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-24T18:00:52.474019+0000","last_clean_scrub_stamp":"2026-02-24T18:00:52.474019+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-26T04:25:16.057495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9],"acting":[5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":12,"acting":12,"num_store_stats":9}],"osd_stats":[{"osd":9,"up_from":40,"seq":171798691856,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":27560,"kb_used_data":600,"kb_used_omap":25,"kb_used_meta":26918,"kb_avail":104830040,"statfs":{"total":107374182400,"available":107345960960,"internally_reserved":0,"allocated":614400,"data_stored":290302,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":25794,"internal_metadata":27564862},"hb_peers":[0,1,2,3,4,5,6,7,8],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":46,"seq":197568495632,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":104857600,"kb_used":27548,"kb_used_data":588,"kb_used_omap":26,"kb_used_meta":26917,"kb_avail":104830052,"statfs":{"total":107374182400,"available":107345973248,"internally_reserved":0,"allocated":602112,"data_stored":282636,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":27091,"internal_metadata":27563565},"hb_peers":[2,3,4,5,6,7,9],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":46,"seq":197568495632,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":27548,"kb_used_data":588,"kb_used_omap":26,"kb_used_meta":26917,"kb_avail":104830052,"statfs":{"total":107374182400,"available":107345973248,"internally_reserved":0,"allocated":602112,"data_stored":282636,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":27086,"internal_metadata":27563570},"hb_peers":[2,3,4,5,6,8,9],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":46,"seq":197568495632,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":27560,"kb_used_data":600,"kb_used_omap":25,"kb_used_meta":26918,"kb_avail":104830040,"statfs":{"total":107374182400,"available":107345960960,"internally_reserved":0,"allocated":614400,"data_stored":290302,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":25788,"internal_metadata":27564868},"hb_peers":[0,1,2,3,4,5,7,8,9],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":46,"seq":197568495632,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":27548,"kb_used_data":588,"kb_used_omap":26,"kb_used_meta":26917,"kb_avail":104830052,"statfs":{"total":107374182400,"available":107345973248,"internally_reserved":0,"allocated":602112,"data_stored":282636,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":27091,"internal_metadata":27563565},"hb_peers":[0,1,2,3,4,6,7,8,9],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":46,"seq":197568495632,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":27564,"kb_used_data":604,"kb_used_omap":28,"kb_used_meta":26915,"kb_avail":104830036,"statfs":{"total":107374182400,"available":107345956864,"internally_reserved":0,"allocated":618496,"data_stored":290321,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":29044,"internal_metadata":27561612},"hb_peers":[0,1,2,3,5,6,7,8,9],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":46,"seq":197568495632,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":27560,"kb_used_data":600,"kb_used_omap":25,"kb_used_meta":26918,"kb_avail":104830040,"statfs":{"total":107374182400,"available":107345960960,"internally_reserved":0,"allocated":614400,"data_stored":290302,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":26437,"internal_metadata":27564219},"hb_peers":[0,1,2,4,5,6,7,8,9],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":46,"seq":197568495632,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":104857600,"kb_used":27560,"kb_used_data":600,"kb_used_omap":22,"kb_used_meta":26921,"kb_avail":104830040,"statfs":{"total":107374182400,"available":107345960960,"internally_reserved":0,"allocated":614400,"data_stored":290302,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":22534,"internal_metadata":27568122},"hb_peers":[3,4,5,6,7,8,9],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":16,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":27540,"kb_used_data":580,"kb_used_omap":26,"kb_used_meta":26917,"kb_avail":104830060,"statfs":{"total":107374182400,"available":107345981440,"internally_reserved":0,"allocated":593920,"data_stored":274921,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":27091,"internal_metadata":27563565},"hb_peers":[2,3,4,5,6,7,8,9],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":56,"seq":240518168593,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":27572,"kb_used_data":612,"kb_used_omap":13,"kb_used_meta":26930,"kb_avail":104830028,"statfs":{"total":107374182400,"available":107345948672,"internally_reserved":0,"allocated":626688,"data_stored":298036,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14071,"internal_metadata":27576585},"hb_peers":[2,3,4,5,6,7,8,9],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-24T18:01:59.489 DEBUG:teuthology.orchestra.run.trial068:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-24T18:01:59.655 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-24T18:01:59.656 INFO:teuthology.orchestra.run.trial068.stderr:dumped all 2026-02-24T18:01:59.666 INFO:teuthology.orchestra.run.trial068.stdout:{"pg_ready":true,"pg_map":{"version":84,"stamp":"2026-02-24T18:01:59.468418+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":10,"num_per_pool_osds":10,"num_per_pool_omap_osds":8,"kb":1048576000,"kb_used":275560,"kb_used_data":5960,"kb_used_omap":246,"kb_used_meta":269193,"kb_avail":1048300440,"statfs":{"total":1073741824000,"available":1073459650560,"internally_reserved":0,"allocated":6103040,"data_stored":2872394,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":252027,"internal_metadata":275654533},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.000430"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":69,"reported_epoch":54,"state":"active+clean","last_fresh":"2026-02-24T18:01:56.374147+0000","last_change":"2026-02-24T18:01:49.359327+0000","last_active":"2026-02-24T18:01:56.374147+0000","last_peered":"2026-02-24T18:01:56.374147+0000","last_clean":"2026-02-24T18:01:56.374147+0000","last_became_active":"2026-02-24T18:01:49.359085+0000","last_became_peered":"2026-02-24T18:01:49.359085+0000","last_unstale":"2026-02-24T18:01:56.374147+0000","last_undegraded":"2026-02-24T18:01:56.374147+0000","last_fullsized":"2026-02-24T18:01:56.374147+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-24T18:00:52.474019+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-24T18:00:52.474019+0000","last_clean_scrub_stamp":"2026-02-24T18:00:52.474019+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-25T23:52:48.720387+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000255505,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":88,"reported_epoch":54,"state":"active+clean","last_fresh":"2026-02-24T18:01:56.374068+0000","last_change":"2026-02-24T18:01:49.359276+0000","last_active":"2026-02-24T18:01:56.374068+0000","last_peered":"2026-02-24T18:01:56.374068+0000","last_clean":"2026-02-24T18:01:56.374068+0000","last_became_active":"2026-02-24T18:01:49.359064+0000","last_became_peered":"2026-02-24T18:01:49.359064+0000","last_unstale":"2026-02-24T18:01:56.374068+0000","last_undegraded":"2026-02-24T18:01:56.374068+0000","last_fullsized":"2026-02-24T18:01:56.374068+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-24T18:00:52.474019+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-24T18:00:52.474019+0000","last_clean_scrub_stamp":"2026-02-24T18:00:52.474019+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-26T05:08:30.898058+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00029952599999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":96,"reported_epoch":55,"state":"active+undersized","last_fresh":"2026-02-24T18:01:57.376232+0000","last_change":"2026-02-24T18:01:57.376232+0000","last_active":"2026-02-24T18:01:57.376232+0000","last_peered":"2026-02-24T18:01:57.376232+0000","last_clean":"2026-02-24T18:01:54.618236+0000","last_became_active":"2026-02-24T18:01:56.374381+0000","last_became_peered":"2026-02-24T18:01:56.374381+0000","last_unstale":"2026-02-24T18:01:57.376232+0000","last_undegraded":"2026-02-24T18:01:57.376232+0000","last_fullsized":"2026-02-24T18:01:56.374083+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-24T18:00:52.474019+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-24T18:00:52.474019+0000","last_clean_scrub_stamp":"2026-02-24T18:00:52.474019+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-25T18:19:12.949262+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00028598099999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9],"acting":[9],"avail_no_missing":["9"],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":61,"reported_epoch":54,"state":"active+undersized+wait","last_fresh":"2026-02-24T18:01:56.374574+0000","last_change":"2026-02-24T18:01:56.374333+0000","last_active":"2026-02-24T18:01:56.374574+0000","last_peered":"2026-02-24T18:01:56.374574+0000","last_clean":"2026-02-24T18:01:46.156753+0000","last_became_active":"2026-02-24T18:01:56.374333+0000","last_became_peered":"2026-02-24T18:01:56.374333+0000","last_unstale":"2026-02-24T18:01:56.374574+0000","last_undegraded":"2026-02-24T18:01:56.374574+0000","last_fullsized":"2026-02-24T18:01:56.374090+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-24T18:00:52.474019+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-24T18:00:52.474019+0000","last_clean_scrub_stamp":"2026-02-24T18:00:52.474019+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-26T03:42:00.638077+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6],"acting":[6],"avail_no_missing":["6"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.3","version":"18'1","reported_seq":84,"reported_epoch":54,"state":"active+undersized+degraded+wait","last_fresh":"2026-02-24T18:01:56.374616+0000","last_change":"2026-02-24T18:01:56.374449+0000","last_active":"2026-02-24T18:01:56.374616+0000","last_peered":"2026-02-24T18:01:56.374616+0000","last_clean":"2026-02-24T18:01:54.618235+0000","last_became_active":"2026-02-24T18:01:56.374449+0000","last_became_peered":"2026-02-24T18:01:56.374449+0000","last_unstale":"2026-02-24T18:01:56.374616+0000","last_undegraded":"2026-02-24T18:01:56.374200+0000","last_fullsized":"2026-02-24T18:01:56.374158+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-24T18:00:52.474019+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-24T18:00:52.474019+0000","last_clean_scrub_stamp":"2026-02-24T18:00:52.474019+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-25T19:06:51.757172+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[{"shards":"3","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"20'2","reported_seq":92,"reported_epoch":54,"state":"active+undersized+degraded+wait","last_fresh":"2026-02-24T18:01:56.374391+0000","last_change":"2026-02-24T18:01:56.374282+0000","last_active":"2026-02-24T18:01:56.374391+0000","last_peered":"2026-02-24T18:01:56.374391+0000","last_clean":"2026-02-24T18:01:54.618334+0000","last_became_active":"2026-02-24T18:01:56.374282+0000","last_became_peered":"2026-02-24T18:01:56.374282+0000","last_unstale":"2026-02-24T18:01:56.374391+0000","last_undegraded":"2026-02-24T18:01:56.374112+0000","last_fullsized":"2026-02-24T18:01:56.374072+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-24T18:00:52.474019+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-24T18:00:52.474019+0000","last_clean_scrub_stamp":"2026-02-24T18:00:52.474019+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-26T04:49:05.797860+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00073412399999999995,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4],"acting":[4],"avail_no_missing":["4"],"object_location_counts":[{"shards":"4","objects":1}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":69,"reported_epoch":53,"state":"active+clean","last_fresh":"2026-02-24T18:01:55.618250+0000","last_change":"2026-02-24T18:01:49.359419+0000","last_active":"2026-02-24T18:01:55.618250+0000","last_peered":"2026-02-24T18:01:55.618250+0000","last_clean":"2026-02-24T18:01:55.618250+0000","last_became_active":"2026-02-24T18:01:49.359167+0000","last_became_peered":"2026-02-24T18:01:49.359167+0000","last_unstale":"2026-02-24T18:01:55.618250+0000","last_undegraded":"2026-02-24T18:01:55.618250+0000","last_fullsized":"2026-02-24T18:01:55.618250+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-24T18:00:52.474019+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-24T18:00:52.474019+0000","last_clean_scrub_stamp":"2026-02-24T18:00:52.474019+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-25T19:47:00.731011+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00026963099999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":80,"reported_epoch":53,"state":"active+clean","last_fresh":"2026-02-24T18:01:55.618229+0000","last_change":"2026-02-24T18:01:49.359091+0000","last_active":"2026-02-24T18:01:55.618229+0000","last_peered":"2026-02-24T18:01:55.618229+0000","last_clean":"2026-02-24T18:01:55.618229+0000","last_became_active":"2026-02-24T18:01:49.358904+0000","last_became_peered":"2026-02-24T18:01:49.358904+0000","last_unstale":"2026-02-24T18:01:55.618229+0000","last_undegraded":"2026-02-24T18:01:55.618229+0000","last_fullsized":"2026-02-24T18:01:55.618229+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-24T18:00:52.474019+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-24T18:00:52.474019+0000","last_clean_scrub_stamp":"2026-02-24T18:00:52.474019+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-26T04:25:16.057495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9],"acting":[5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":12,"acting":12,"num_store_stats":9}],"osd_stats":[{"osd":9,"up_from":40,"seq":171798691856,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":27560,"kb_used_data":600,"kb_used_omap":25,"kb_used_meta":26918,"kb_avail":104830040,"statfs":{"total":107374182400,"available":107345960960,"internally_reserved":0,"allocated":614400,"data_stored":290302,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":25794,"internal_metadata":27564862},"hb_peers":[0,1,2,3,4,5,6,7,8],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":46,"seq":197568495632,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":104857600,"kb_used":27548,"kb_used_data":588,"kb_used_omap":26,"kb_used_meta":26917,"kb_avail":104830052,"statfs":{"total":107374182400,"available":107345973248,"internally_reserved":0,"allocated":602112,"data_stored":282636,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":27091,"internal_metadata":27563565},"hb_peers":[2,3,4,5,6,7,9],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":46,"seq":197568495632,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":27548,"kb_used_data":588,"kb_used_omap":26,"kb_used_meta":26917,"kb_avail":104830052,"statfs":{"total":107374182400,"available":107345973248,"internally_reserved":0,"allocated":602112,"data_stored":282636,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":27086,"internal_metadata":27563570},"hb_peers":[2,3,4,5,6,8,9],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":46,"seq":197568495632,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":27560,"kb_used_data":600,"kb_used_omap":25,"kb_used_meta":26918,"kb_avail":104830040,"statfs":{"total":107374182400,"available":107345960960,"internally_reserved":0,"allocated":614400,"data_stored":290302,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":25788,"internal_metadata":27564868},"hb_peers":[0,1,2,3,4,5,7,8,9],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":46,"seq":197568495632,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":27548,"kb_used_data":588,"kb_used_omap":26,"kb_used_meta":26917,"kb_avail":104830052,"statfs":{"total":107374182400,"available":107345973248,"internally_reserved":0,"allocated":602112,"data_stored":282636,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":27091,"internal_metadata":27563565},"hb_peers":[0,1,2,3,4,6,7,8,9],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":46,"seq":197568495632,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":27564,"kb_used_data":604,"kb_used_omap":28,"kb_used_meta":26915,"kb_avail":104830036,"statfs":{"total":107374182400,"available":107345956864,"internally_reserved":0,"allocated":618496,"data_stored":290321,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":29044,"internal_metadata":27561612},"hb_peers":[0,1,2,3,5,6,7,8,9],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":46,"seq":197568495632,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":27560,"kb_used_data":600,"kb_used_omap":25,"kb_used_meta":26918,"kb_avail":104830040,"statfs":{"total":107374182400,"available":107345960960,"internally_reserved":0,"allocated":614400,"data_stored":290302,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":26437,"internal_metadata":27564219},"hb_peers":[0,1,2,4,5,6,7,8,9],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":46,"seq":197568495632,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":104857600,"kb_used":27560,"kb_used_data":600,"kb_used_omap":22,"kb_used_meta":26921,"kb_avail":104830040,"statfs":{"total":107374182400,"available":107345960960,"internally_reserved":0,"allocated":614400,"data_stored":290302,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":22534,"internal_metadata":27568122},"hb_peers":[3,4,5,6,7,8,9],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":16,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":27540,"kb_used_data":580,"kb_used_omap":26,"kb_used_meta":26917,"kb_avail":104830060,"statfs":{"total":107374182400,"available":107345981440,"internally_reserved":0,"allocated":593920,"data_stored":274921,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":27091,"internal_metadata":27563565},"hb_peers":[2,3,4,5,6,7,8,9],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":56,"seq":240518168593,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":27572,"kb_used_data":612,"kb_used_omap":13,"kb_used_meta":26930,"kb_avail":104830028,"statfs":{"total":107374182400,"available":107345948672,"internally_reserved":0,"allocated":626688,"data_stored":298036,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14071,"internal_metadata":27576585},"hb_peers":[2,3,4,5,6,7,8,9],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-24T18:01:59.668 INFO:tasks.ceph.ceph_manager.ceph:PG 1.5 is not active+clean 2026-02-24T18:01:59.668 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '1.5', 'version': "0'0", 'reported_seq': 96, 'reported_epoch': 55, 'state': 'active+undersized', 'last_fresh': '2026-02-24T18:01:57.376232+0000', 'last_change': '2026-02-24T18:01:57.376232+0000', 'last_active': '2026-02-24T18:01:57.376232+0000', 'last_peered': '2026-02-24T18:01:57.376232+0000', 'last_clean': '2026-02-24T18:01:54.618236+0000', 'last_became_active': '2026-02-24T18:01:56.374381+0000', 'last_became_peered': '2026-02-24T18:01:56.374381+0000', 'last_unstale': '2026-02-24T18:01:57.376232+0000', 'last_undegraded': '2026-02-24T18:01:57.376232+0000', 'last_fullsized': '2026-02-24T18:01:56.374083+0000', 'mapping_epoch': 53, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 17, 'last_epoch_clean': 47, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2026-02-24T18:00:52.474019+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2026-02-24T18:00:52.474019+0000', 'last_clean_scrub_stamp': '2026-02-24T18:00:52.474019+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2026-02-25T18:19:12.949262+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.000285981, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [9], 'acting': [9], 'avail_no_missing': ['9'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 9, 'acting_primary': 9, 'purged_snaps': []} 2026-02-24T18:01:59.668 INFO:tasks.ceph.ceph_manager.ceph:PG 1.4 is not active+clean 2026-02-24T18:01:59.669 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '1.4', 'version': "0'0", 'reported_seq': 61, 'reported_epoch': 54, 'state': 'active+undersized+wait', 'last_fresh': '2026-02-24T18:01:56.374574+0000', 'last_change': '2026-02-24T18:01:56.374333+0000', 'last_active': '2026-02-24T18:01:56.374574+0000', 'last_peered': '2026-02-24T18:01:56.374574+0000', 'last_clean': '2026-02-24T18:01:46.156753+0000', 'last_became_active': '2026-02-24T18:01:56.374333+0000', 'last_became_peered': '2026-02-24T18:01:56.374333+0000', 'last_unstale': '2026-02-24T18:01:56.374574+0000', 'last_undegraded': '2026-02-24T18:01:56.374574+0000', 'last_fullsized': '2026-02-24T18:01:56.374090+0000', 'mapping_epoch': 53, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 17, 'last_epoch_clean': 47, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2026-02-24T18:00:52.474019+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2026-02-24T18:00:52.474019+0000', 'last_clean_scrub_stamp': '2026-02-24T18:00:52.474019+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2026-02-26T03:42:00.638077+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6], 'acting': [6], 'avail_no_missing': ['6'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2026-02-24T18:01:59.669 INFO:tasks.ceph.ceph_manager.ceph:PG 1.3 is not active+clean 2026-02-24T18:01:59.669 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '1.3', 'version': "18'1", 'reported_seq': 84, 'reported_epoch': 54, 'state': 'active+undersized+degraded+wait', 'last_fresh': '2026-02-24T18:01:56.374616+0000', 'last_change': '2026-02-24T18:01:56.374449+0000', 'last_active': '2026-02-24T18:01:56.374616+0000', 'last_peered': '2026-02-24T18:01:56.374616+0000', 'last_clean': '2026-02-24T18:01:54.618235+0000', 'last_became_active': '2026-02-24T18:01:56.374449+0000', 'last_became_peered': '2026-02-24T18:01:56.374449+0000', 'last_unstale': '2026-02-24T18:01:56.374616+0000', 'last_undegraded': '2026-02-24T18:01:56.374200+0000', 'last_fullsized': '2026-02-24T18:01:56.374158+0000', 'mapping_epoch': 53, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 17, 'last_epoch_clean': 47, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2026-02-24T18:00:52.474019+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2026-02-24T18:00:52.474019+0000', 'last_clean_scrub_stamp': '2026-02-24T18:00:52.474019+0000', 'objects_scrubbed': 0, 'log_size': 1, 'log_dups_size': 0, 'ondisk_log_size': 1, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2026-02-25T19:06:51.757172+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3], 'acting': [3], 'avail_no_missing': ['3'], 'object_location_counts': [{'shards': '3', 'objects': 1}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-02-24T18:01:59.669 INFO:tasks.ceph.ceph_manager.ceph:PG 1.2 is not active+clean 2026-02-24T18:01:59.669 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '1.2', 'version': "20'2", 'reported_seq': 92, 'reported_epoch': 54, 'state': 'active+undersized+degraded+wait', 'last_fresh': '2026-02-24T18:01:56.374391+0000', 'last_change': '2026-02-24T18:01:56.374282+0000', 'last_active': '2026-02-24T18:01:56.374391+0000', 'last_peered': '2026-02-24T18:01:56.374391+0000', 'last_clean': '2026-02-24T18:01:54.618334+0000', 'last_became_active': '2026-02-24T18:01:56.374282+0000', 'last_became_peered': '2026-02-24T18:01:56.374282+0000', 'last_unstale': '2026-02-24T18:01:56.374391+0000', 'last_undegraded': '2026-02-24T18:01:56.374112+0000', 'last_fullsized': '2026-02-24T18:01:56.374072+0000', 'mapping_epoch': 53, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 17, 'last_epoch_clean': 47, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2026-02-24T18:00:52.474019+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2026-02-24T18:00:52.474019+0000', 'last_clean_scrub_stamp': '2026-02-24T18:00:52.474019+0000', 'objects_scrubbed': 0, 'log_size': 2, 'log_dups_size': 0, 'ondisk_log_size': 2, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2026-02-26T04:49:05.797860+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.000734124, 'stat_sum': {'num_bytes': 19, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2, 'num_write_kb': 2, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 19, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4], 'acting': [4], 'avail_no_missing': ['4'], 'object_location_counts': [{'shards': '4', 'objects': 1}], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2026-02-24T18:01:59.670 DEBUG:teuthology.orchestra.run.trial068:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2026-02-24T18:01:59.858 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-24T18:01:59.858 INFO:teuthology.orchestra.run.trial068.stdout:{"fsid":"85c6d86c-aad0-4a30-9c50-f24175e8fcf5","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"noout flag(s) set","count":5},"muted":false}},"mutes":[]},"election_epoch":3,"quorum":[0],"quorum_names":["a"],"quorum_age":78,"monmap":{"epoch":1,"min_mon_release_name":"tentacle","num_mons":1},"osdmap":{"epoch":57,"num_osds":10,"num_up_osds":10,"osd_up_since":1771956118,"num_in_osds":10,"osd_in_since":1771956043,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":7},{"state_name":"stale+active+clean","count":1}],"num_pgs":8,"num_pools":1,"num_objects":2,"data_bytes":19,"bytes_used":281522176,"bytes_avail":1073460301824,"bytes_total":1073741824000},"fsmap":{"epoch":1,"btime":"2026-02-24T18:00:41:319879+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs"],"services":{}},"servicemap":{"epoch":3,"modified":"2026-02-24T18:01:49.467350+0000","services":{}},"progress_events":{}} 2026-02-24T18:01:59.871 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2026-02-24T18:01:59.872 DEBUG:teuthology.orchestra.run.trial068:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-24T18:02:00.032 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-24T18:02:00.033 INFO:teuthology.orchestra.run.trial068.stderr:dumped all 2026-02-24T18:02:00.043 INFO:teuthology.orchestra.run.trial068.stdout:{"pg_ready":true,"pg_map":{"version":84,"stamp":"2026-02-24T18:01:59.468418+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":10,"num_per_pool_osds":10,"num_per_pool_omap_osds":8,"kb":1048576000,"kb_used":275560,"kb_used_data":5960,"kb_used_omap":246,"kb_used_meta":269193,"kb_avail":1048300440,"statfs":{"total":1073741824000,"available":1073459650560,"internally_reserved":0,"allocated":6103040,"data_stored":2872394,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":252027,"internal_metadata":275654533},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.000430"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":69,"reported_epoch":54,"state":"active+clean","last_fresh":"2026-02-24T18:01:56.374147+0000","last_change":"2026-02-24T18:01:49.359327+0000","last_active":"2026-02-24T18:01:56.374147+0000","last_peered":"2026-02-24T18:01:56.374147+0000","last_clean":"2026-02-24T18:01:56.374147+0000","last_became_active":"2026-02-24T18:01:49.359085+0000","last_became_peered":"2026-02-24T18:01:49.359085+0000","last_unstale":"2026-02-24T18:01:56.374147+0000","last_undegraded":"2026-02-24T18:01:56.374147+0000","last_fullsized":"2026-02-24T18:01:56.374147+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-24T18:00:52.474019+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-24T18:00:52.474019+0000","last_clean_scrub_stamp":"2026-02-24T18:00:52.474019+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-25T23:52:48.720387+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000255505,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":88,"reported_epoch":54,"state":"active+clean","last_fresh":"2026-02-24T18:01:56.374068+0000","last_change":"2026-02-24T18:01:49.359276+0000","last_active":"2026-02-24T18:01:56.374068+0000","last_peered":"2026-02-24T18:01:56.374068+0000","last_clean":"2026-02-24T18:01:56.374068+0000","last_became_active":"2026-02-24T18:01:49.359064+0000","last_became_peered":"2026-02-24T18:01:49.359064+0000","last_unstale":"2026-02-24T18:01:56.374068+0000","last_undegraded":"2026-02-24T18:01:56.374068+0000","last_fullsized":"2026-02-24T18:01:56.374068+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-24T18:00:52.474019+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-24T18:00:52.474019+0000","last_clean_scrub_stamp":"2026-02-24T18:00:52.474019+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-26T05:08:30.898058+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00029952599999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":96,"reported_epoch":55,"state":"active+undersized","last_fresh":"2026-02-24T18:01:57.376232+0000","last_change":"2026-02-24T18:01:57.376232+0000","last_active":"2026-02-24T18:01:57.376232+0000","last_peered":"2026-02-24T18:01:57.376232+0000","last_clean":"2026-02-24T18:01:54.618236+0000","last_became_active":"2026-02-24T18:01:56.374381+0000","last_became_peered":"2026-02-24T18:01:56.374381+0000","last_unstale":"2026-02-24T18:01:57.376232+0000","last_undegraded":"2026-02-24T18:01:57.376232+0000","last_fullsized":"2026-02-24T18:01:56.374083+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-24T18:00:52.474019+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-24T18:00:52.474019+0000","last_clean_scrub_stamp":"2026-02-24T18:00:52.474019+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-25T18:19:12.949262+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00028598099999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9],"acting":[9],"avail_no_missing":["9"],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":61,"reported_epoch":54,"state":"active+undersized+wait","last_fresh":"2026-02-24T18:01:56.374574+0000","last_change":"2026-02-24T18:01:56.374333+0000","last_active":"2026-02-24T18:01:56.374574+0000","last_peered":"2026-02-24T18:01:56.374574+0000","last_clean":"2026-02-24T18:01:46.156753+0000","last_became_active":"2026-02-24T18:01:56.374333+0000","last_became_peered":"2026-02-24T18:01:56.374333+0000","last_unstale":"2026-02-24T18:01:56.374574+0000","last_undegraded":"2026-02-24T18:01:56.374574+0000","last_fullsized":"2026-02-24T18:01:56.374090+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-24T18:00:52.474019+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-24T18:00:52.474019+0000","last_clean_scrub_stamp":"2026-02-24T18:00:52.474019+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-26T03:42:00.638077+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6],"acting":[6],"avail_no_missing":["6"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.3","version":"18'1","reported_seq":84,"reported_epoch":54,"state":"active+undersized+degraded+wait","last_fresh":"2026-02-24T18:01:56.374616+0000","last_change":"2026-02-24T18:01:56.374449+0000","last_active":"2026-02-24T18:01:56.374616+0000","last_peered":"2026-02-24T18:01:56.374616+0000","last_clean":"2026-02-24T18:01:54.618235+0000","last_became_active":"2026-02-24T18:01:56.374449+0000","last_became_peered":"2026-02-24T18:01:56.374449+0000","last_unstale":"2026-02-24T18:01:56.374616+0000","last_undegraded":"2026-02-24T18:01:56.374200+0000","last_fullsized":"2026-02-24T18:01:56.374158+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-24T18:00:52.474019+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-24T18:00:52.474019+0000","last_clean_scrub_stamp":"2026-02-24T18:00:52.474019+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-25T19:06:51.757172+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[{"shards":"3","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"20'2","reported_seq":92,"reported_epoch":54,"state":"active+undersized+degraded+wait","last_fresh":"2026-02-24T18:01:56.374391+0000","last_change":"2026-02-24T18:01:56.374282+0000","last_active":"2026-02-24T18:01:56.374391+0000","last_peered":"2026-02-24T18:01:56.374391+0000","last_clean":"2026-02-24T18:01:54.618334+0000","last_became_active":"2026-02-24T18:01:56.374282+0000","last_became_peered":"2026-02-24T18:01:56.374282+0000","last_unstale":"2026-02-24T18:01:56.374391+0000","last_undegraded":"2026-02-24T18:01:56.374112+0000","last_fullsized":"2026-02-24T18:01:56.374072+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-24T18:00:52.474019+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-24T18:00:52.474019+0000","last_clean_scrub_stamp":"2026-02-24T18:00:52.474019+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-26T04:49:05.797860+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00073412399999999995,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4],"acting":[4],"avail_no_missing":["4"],"object_location_counts":[{"shards":"4","objects":1}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":69,"reported_epoch":53,"state":"active+clean","last_fresh":"2026-02-24T18:01:55.618250+0000","last_change":"2026-02-24T18:01:49.359419+0000","last_active":"2026-02-24T18:01:55.618250+0000","last_peered":"2026-02-24T18:01:55.618250+0000","last_clean":"2026-02-24T18:01:55.618250+0000","last_became_active":"2026-02-24T18:01:49.359167+0000","last_became_peered":"2026-02-24T18:01:49.359167+0000","last_unstale":"2026-02-24T18:01:55.618250+0000","last_undegraded":"2026-02-24T18:01:55.618250+0000","last_fullsized":"2026-02-24T18:01:55.618250+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-24T18:00:52.474019+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-24T18:00:52.474019+0000","last_clean_scrub_stamp":"2026-02-24T18:00:52.474019+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-25T19:47:00.731011+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00026963099999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":80,"reported_epoch":53,"state":"active+clean","last_fresh":"2026-02-24T18:01:55.618229+0000","last_change":"2026-02-24T18:01:49.359091+0000","last_active":"2026-02-24T18:01:55.618229+0000","last_peered":"2026-02-24T18:01:55.618229+0000","last_clean":"2026-02-24T18:01:55.618229+0000","last_became_active":"2026-02-24T18:01:49.358904+0000","last_became_peered":"2026-02-24T18:01:49.358904+0000","last_unstale":"2026-02-24T18:01:55.618229+0000","last_undegraded":"2026-02-24T18:01:55.618229+0000","last_fullsized":"2026-02-24T18:01:55.618229+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-24T18:00:52.474019+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-24T18:00:52.474019+0000","last_clean_scrub_stamp":"2026-02-24T18:00:52.474019+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-26T04:25:16.057495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9],"acting":[5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":12,"acting":12,"num_store_stats":9}],"osd_stats":[{"osd":9,"up_from":40,"seq":171798691856,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":27560,"kb_used_data":600,"kb_used_omap":25,"kb_used_meta":26918,"kb_avail":104830040,"statfs":{"total":107374182400,"available":107345960960,"internally_reserved":0,"allocated":614400,"data_stored":290302,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":25794,"internal_metadata":27564862},"hb_peers":[0,1,2,3,4,5,6,7,8],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":46,"seq":197568495632,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":104857600,"kb_used":27548,"kb_used_data":588,"kb_used_omap":26,"kb_used_meta":26917,"kb_avail":104830052,"statfs":{"total":107374182400,"available":107345973248,"internally_reserved":0,"allocated":602112,"data_stored":282636,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":27091,"internal_metadata":27563565},"hb_peers":[2,3,4,5,6,7,9],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":46,"seq":197568495632,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":27548,"kb_used_data":588,"kb_used_omap":26,"kb_used_meta":26917,"kb_avail":104830052,"statfs":{"total":107374182400,"available":107345973248,"internally_reserved":0,"allocated":602112,"data_stored":282636,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":27086,"internal_metadata":27563570},"hb_peers":[2,3,4,5,6,8,9],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":46,"seq":197568495632,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":27560,"kb_used_data":600,"kb_used_omap":25,"kb_used_meta":26918,"kb_avail":104830040,"statfs":{"total":107374182400,"available":107345960960,"internally_reserved":0,"allocated":614400,"data_stored":290302,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":25788,"internal_metadata":27564868},"hb_peers":[0,1,2,3,4,5,7,8,9],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":46,"seq":197568495632,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":27548,"kb_used_data":588,"kb_used_omap":26,"kb_used_meta":26917,"kb_avail":104830052,"statfs":{"total":107374182400,"available":107345973248,"internally_reserved":0,"allocated":602112,"data_stored":282636,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":27091,"internal_metadata":27563565},"hb_peers":[0,1,2,3,4,6,7,8,9],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":46,"seq":197568495632,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":27564,"kb_used_data":604,"kb_used_omap":28,"kb_used_meta":26915,"kb_avail":104830036,"statfs":{"total":107374182400,"available":107345956864,"internally_reserved":0,"allocated":618496,"data_stored":290321,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":29044,"internal_metadata":27561612},"hb_peers":[0,1,2,3,5,6,7,8,9],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":46,"seq":197568495632,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":27560,"kb_used_data":600,"kb_used_omap":25,"kb_used_meta":26918,"kb_avail":104830040,"statfs":{"total":107374182400,"available":107345960960,"internally_reserved":0,"allocated":614400,"data_stored":290302,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":26437,"internal_metadata":27564219},"hb_peers":[0,1,2,4,5,6,7,8,9],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":46,"seq":197568495632,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":104857600,"kb_used":27560,"kb_used_data":600,"kb_used_omap":22,"kb_used_meta":26921,"kb_avail":104830040,"statfs":{"total":107374182400,"available":107345960960,"internally_reserved":0,"allocated":614400,"data_stored":290302,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":22534,"internal_metadata":27568122},"hb_peers":[3,4,5,6,7,8,9],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":16,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":27540,"kb_used_data":580,"kb_used_omap":26,"kb_used_meta":26917,"kb_avail":104830060,"statfs":{"total":107374182400,"available":107345981440,"internally_reserved":0,"allocated":593920,"data_stored":274921,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":27091,"internal_metadata":27563565},"hb_peers":[2,3,4,5,6,7,8,9],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":56,"seq":240518168593,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":27572,"kb_used_data":612,"kb_used_omap":13,"kb_used_meta":26930,"kb_avail":104830028,"statfs":{"total":107374182400,"available":107345948672,"internally_reserved":0,"allocated":626688,"data_stored":298036,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14071,"internal_metadata":27576585},"hb_peers":[2,3,4,5,6,7,8,9],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-24T18:02:03.046 DEBUG:teuthology.orchestra.run.trial068:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-24T18:02:03.208 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-24T18:02:03.208 INFO:teuthology.orchestra.run.trial068.stderr:dumped all 2026-02-24T18:02:03.218 INFO:teuthology.orchestra.run.trial068.stdout:{"pg_ready":true,"pg_map":{"version":85,"stamp":"2026-02-24T18:02:01.468658+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":10,"num_per_pool_osds":10,"num_per_pool_omap_osds":8,"kb":1048576000,"kb_used":275560,"kb_used_data":5960,"kb_used_omap":246,"kb_used_meta":269193,"kb_avail":1048300440,"statfs":{"total":1073741824000,"available":1073459650560,"internally_reserved":0,"allocated":6103040,"data_stored":2872394,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":252027,"internal_metadata":275654533},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.095793"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":69,"reported_epoch":54,"state":"active+clean","last_fresh":"2026-02-24T18:01:56.374147+0000","last_change":"2026-02-24T18:01:49.359327+0000","last_active":"2026-02-24T18:01:56.374147+0000","last_peered":"2026-02-24T18:01:56.374147+0000","last_clean":"2026-02-24T18:01:56.374147+0000","last_became_active":"2026-02-24T18:01:49.359085+0000","last_became_peered":"2026-02-24T18:01:49.359085+0000","last_unstale":"2026-02-24T18:01:56.374147+0000","last_undegraded":"2026-02-24T18:01:56.374147+0000","last_fullsized":"2026-02-24T18:01:56.374147+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-24T18:00:52.474019+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-24T18:00:52.474019+0000","last_clean_scrub_stamp":"2026-02-24T18:00:52.474019+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-25T23:52:48.720387+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000255505,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":88,"reported_epoch":54,"state":"active+clean","last_fresh":"2026-02-24T18:01:56.374068+0000","last_change":"2026-02-24T18:01:49.359276+0000","last_active":"2026-02-24T18:01:56.374068+0000","last_peered":"2026-02-24T18:01:56.374068+0000","last_clean":"2026-02-24T18:01:56.374068+0000","last_became_active":"2026-02-24T18:01:49.359064+0000","last_became_peered":"2026-02-24T18:01:49.359064+0000","last_unstale":"2026-02-24T18:01:56.374068+0000","last_undegraded":"2026-02-24T18:01:56.374068+0000","last_fullsized":"2026-02-24T18:01:56.374068+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-24T18:00:52.474019+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-24T18:00:52.474019+0000","last_clean_scrub_stamp":"2026-02-24T18:00:52.474019+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-26T05:08:30.898058+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00029952599999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":96,"reported_epoch":55,"state":"active+undersized","last_fresh":"2026-02-24T18:01:57.376232+0000","last_change":"2026-02-24T18:01:57.376232+0000","last_active":"2026-02-24T18:01:57.376232+0000","last_peered":"2026-02-24T18:01:57.376232+0000","last_clean":"2026-02-24T18:01:54.618236+0000","last_became_active":"2026-02-24T18:01:56.374381+0000","last_became_peered":"2026-02-24T18:01:56.374381+0000","last_unstale":"2026-02-24T18:01:57.376232+0000","last_undegraded":"2026-02-24T18:01:57.376232+0000","last_fullsized":"2026-02-24T18:01:56.374083+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-24T18:00:52.474019+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-24T18:00:52.474019+0000","last_clean_scrub_stamp":"2026-02-24T18:00:52.474019+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-25T18:19:12.949262+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00028598099999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9],"acting":[9],"avail_no_missing":["9"],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":61,"reported_epoch":54,"state":"active+undersized+wait","last_fresh":"2026-02-24T18:01:56.374574+0000","last_change":"2026-02-24T18:01:56.374333+0000","last_active":"2026-02-24T18:01:56.374574+0000","last_peered":"2026-02-24T18:01:56.374574+0000","last_clean":"2026-02-24T18:01:46.156753+0000","last_became_active":"2026-02-24T18:01:56.374333+0000","last_became_peered":"2026-02-24T18:01:56.374333+0000","last_unstale":"2026-02-24T18:01:56.374574+0000","last_undegraded":"2026-02-24T18:01:56.374574+0000","last_fullsized":"2026-02-24T18:01:56.374090+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-24T18:00:52.474019+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-24T18:00:52.474019+0000","last_clean_scrub_stamp":"2026-02-24T18:00:52.474019+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-26T03:42:00.638077+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6],"acting":[6],"avail_no_missing":["6"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.3","version":"18'1","reported_seq":84,"reported_epoch":54,"state":"active+undersized+degraded+wait","last_fresh":"2026-02-24T18:01:56.374616+0000","last_change":"2026-02-24T18:01:56.374449+0000","last_active":"2026-02-24T18:01:56.374616+0000","last_peered":"2026-02-24T18:01:56.374616+0000","last_clean":"2026-02-24T18:01:54.618235+0000","last_became_active":"2026-02-24T18:01:56.374449+0000","last_became_peered":"2026-02-24T18:01:56.374449+0000","last_unstale":"2026-02-24T18:01:56.374616+0000","last_undegraded":"2026-02-24T18:01:56.374200+0000","last_fullsized":"2026-02-24T18:01:56.374158+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-24T18:00:52.474019+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-24T18:00:52.474019+0000","last_clean_scrub_stamp":"2026-02-24T18:00:52.474019+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-25T19:06:51.757172+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[{"shards":"3","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"20'2","reported_seq":92,"reported_epoch":54,"state":"active+undersized+degraded+wait","last_fresh":"2026-02-24T18:01:56.374391+0000","last_change":"2026-02-24T18:01:56.374282+0000","last_active":"2026-02-24T18:01:56.374391+0000","last_peered":"2026-02-24T18:01:56.374391+0000","last_clean":"2026-02-24T18:01:54.618334+0000","last_became_active":"2026-02-24T18:01:56.374282+0000","last_became_peered":"2026-02-24T18:01:56.374282+0000","last_unstale":"2026-02-24T18:01:56.374391+0000","last_undegraded":"2026-02-24T18:01:56.374112+0000","last_fullsized":"2026-02-24T18:01:56.374072+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-24T18:00:52.474019+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-24T18:00:52.474019+0000","last_clean_scrub_stamp":"2026-02-24T18:00:52.474019+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-26T04:49:05.797860+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00073412399999999995,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4],"acting":[4],"avail_no_missing":["4"],"object_location_counts":[{"shards":"4","objects":1}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":69,"reported_epoch":53,"state":"active+clean","last_fresh":"2026-02-24T18:01:55.618250+0000","last_change":"2026-02-24T18:01:49.359419+0000","last_active":"2026-02-24T18:01:55.618250+0000","last_peered":"2026-02-24T18:01:55.618250+0000","last_clean":"2026-02-24T18:01:55.618250+0000","last_became_active":"2026-02-24T18:01:49.359167+0000","last_became_peered":"2026-02-24T18:01:49.359167+0000","last_unstale":"2026-02-24T18:01:55.618250+0000","last_undegraded":"2026-02-24T18:01:55.618250+0000","last_fullsized":"2026-02-24T18:01:55.618250+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-24T18:00:52.474019+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-24T18:00:52.474019+0000","last_clean_scrub_stamp":"2026-02-24T18:00:52.474019+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-25T19:47:00.731011+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00026963099999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":80,"reported_epoch":53,"state":"active+clean","last_fresh":"2026-02-24T18:01:55.618229+0000","last_change":"2026-02-24T18:01:49.359091+0000","last_active":"2026-02-24T18:01:55.618229+0000","last_peered":"2026-02-24T18:01:55.618229+0000","last_clean":"2026-02-24T18:01:55.618229+0000","last_became_active":"2026-02-24T18:01:49.358904+0000","last_became_peered":"2026-02-24T18:01:49.358904+0000","last_unstale":"2026-02-24T18:01:55.618229+0000","last_undegraded":"2026-02-24T18:01:55.618229+0000","last_fullsized":"2026-02-24T18:01:55.618229+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-24T18:00:52.474019+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-24T18:00:52.474019+0000","last_clean_scrub_stamp":"2026-02-24T18:00:52.474019+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-26T04:25:16.057495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9],"acting":[5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":12,"acting":12,"num_store_stats":9}],"osd_stats":[{"osd":9,"up_from":40,"seq":171798691856,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":27560,"kb_used_data":600,"kb_used_omap":25,"kb_used_meta":26918,"kb_avail":104830040,"statfs":{"total":107374182400,"available":107345960960,"internally_reserved":0,"allocated":614400,"data_stored":290302,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":25794,"internal_metadata":27564862},"hb_peers":[0,1,2,3,4,5,6,7,8],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":46,"seq":197568495632,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":104857600,"kb_used":27548,"kb_used_data":588,"kb_used_omap":26,"kb_used_meta":26917,"kb_avail":104830052,"statfs":{"total":107374182400,"available":107345973248,"internally_reserved":0,"allocated":602112,"data_stored":282636,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":27091,"internal_metadata":27563565},"hb_peers":[2,3,4,5,6,7,9],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":46,"seq":197568495632,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":27548,"kb_used_data":588,"kb_used_omap":26,"kb_used_meta":26917,"kb_avail":104830052,"statfs":{"total":107374182400,"available":107345973248,"internally_reserved":0,"allocated":602112,"data_stored":282636,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":27086,"internal_metadata":27563570},"hb_peers":[2,3,4,5,6,8,9],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":46,"seq":197568495632,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":27560,"kb_used_data":600,"kb_used_omap":25,"kb_used_meta":26918,"kb_avail":104830040,"statfs":{"total":107374182400,"available":107345960960,"internally_reserved":0,"allocated":614400,"data_stored":290302,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":25788,"internal_metadata":27564868},"hb_peers":[0,1,2,3,4,5,7,8,9],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":46,"seq":197568495632,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":27548,"kb_used_data":588,"kb_used_omap":26,"kb_used_meta":26917,"kb_avail":104830052,"statfs":{"total":107374182400,"available":107345973248,"internally_reserved":0,"allocated":602112,"data_stored":282636,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":27091,"internal_metadata":27563565},"hb_peers":[0,1,2,3,4,6,7,8,9],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":46,"seq":197568495632,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":27564,"kb_used_data":604,"kb_used_omap":28,"kb_used_meta":26915,"kb_avail":104830036,"statfs":{"total":107374182400,"available":107345956864,"internally_reserved":0,"allocated":618496,"data_stored":290321,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":29044,"internal_metadata":27561612},"hb_peers":[0,1,2,3,5,6,7,8,9],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":46,"seq":197568495632,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":27560,"kb_used_data":600,"kb_used_omap":25,"kb_used_meta":26918,"kb_avail":104830040,"statfs":{"total":107374182400,"available":107345960960,"internally_reserved":0,"allocated":614400,"data_stored":290302,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":26437,"internal_metadata":27564219},"hb_peers":[0,1,2,4,5,6,7,8,9],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":46,"seq":197568495632,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":104857600,"kb_used":27560,"kb_used_data":600,"kb_used_omap":22,"kb_used_meta":26921,"kb_avail":104830040,"statfs":{"total":107374182400,"available":107345960960,"internally_reserved":0,"allocated":614400,"data_stored":290302,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":22534,"internal_metadata":27568122},"hb_peers":[3,4,5,6,7,8,9],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":16,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":27540,"kb_used_data":580,"kb_used_omap":26,"kb_used_meta":26917,"kb_avail":104830060,"statfs":{"total":107374182400,"available":107345981440,"internally_reserved":0,"allocated":593920,"data_stored":274921,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":27091,"internal_metadata":27563565},"hb_peers":[2,3,4,5,6,7,8,9],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":56,"seq":240518168593,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":27572,"kb_used_data":612,"kb_used_omap":13,"kb_used_meta":26930,"kb_avail":104830028,"statfs":{"total":107374182400,"available":107345948672,"internally_reserved":0,"allocated":626688,"data_stored":298036,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14071,"internal_metadata":27576585},"hb_peers":[2,3,4,5,6,7,8,9],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-24T18:02:03.219 DEBUG:teuthology.orchestra.run.trial068:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-24T18:02:03.369 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-24T18:02:03.369 INFO:teuthology.orchestra.run.trial068.stderr:dumped all 2026-02-24T18:02:03.379 INFO:teuthology.orchestra.run.trial068.stdout:{"pg_ready":true,"pg_map":{"version":85,"stamp":"2026-02-24T18:02:01.468658+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":10,"num_per_pool_osds":10,"num_per_pool_omap_osds":8,"kb":1048576000,"kb_used":275560,"kb_used_data":5960,"kb_used_omap":246,"kb_used_meta":269193,"kb_avail":1048300440,"statfs":{"total":1073741824000,"available":1073459650560,"internally_reserved":0,"allocated":6103040,"data_stored":2872394,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":252027,"internal_metadata":275654533},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.095793"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":69,"reported_epoch":54,"state":"active+clean","last_fresh":"2026-02-24T18:01:56.374147+0000","last_change":"2026-02-24T18:01:49.359327+0000","last_active":"2026-02-24T18:01:56.374147+0000","last_peered":"2026-02-24T18:01:56.374147+0000","last_clean":"2026-02-24T18:01:56.374147+0000","last_became_active":"2026-02-24T18:01:49.359085+0000","last_became_peered":"2026-02-24T18:01:49.359085+0000","last_unstale":"2026-02-24T18:01:56.374147+0000","last_undegraded":"2026-02-24T18:01:56.374147+0000","last_fullsized":"2026-02-24T18:01:56.374147+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-24T18:00:52.474019+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-24T18:00:52.474019+0000","last_clean_scrub_stamp":"2026-02-24T18:00:52.474019+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-25T23:52:48.720387+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000255505,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":88,"reported_epoch":54,"state":"active+clean","last_fresh":"2026-02-24T18:01:56.374068+0000","last_change":"2026-02-24T18:01:49.359276+0000","last_active":"2026-02-24T18:01:56.374068+0000","last_peered":"2026-02-24T18:01:56.374068+0000","last_clean":"2026-02-24T18:01:56.374068+0000","last_became_active":"2026-02-24T18:01:49.359064+0000","last_became_peered":"2026-02-24T18:01:49.359064+0000","last_unstale":"2026-02-24T18:01:56.374068+0000","last_undegraded":"2026-02-24T18:01:56.374068+0000","last_fullsized":"2026-02-24T18:01:56.374068+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-24T18:00:52.474019+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-24T18:00:52.474019+0000","last_clean_scrub_stamp":"2026-02-24T18:00:52.474019+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-26T05:08:30.898058+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00029952599999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":96,"reported_epoch":55,"state":"active+undersized","last_fresh":"2026-02-24T18:01:57.376232+0000","last_change":"2026-02-24T18:01:57.376232+0000","last_active":"2026-02-24T18:01:57.376232+0000","last_peered":"2026-02-24T18:01:57.376232+0000","last_clean":"2026-02-24T18:01:54.618236+0000","last_became_active":"2026-02-24T18:01:56.374381+0000","last_became_peered":"2026-02-24T18:01:56.374381+0000","last_unstale":"2026-02-24T18:01:57.376232+0000","last_undegraded":"2026-02-24T18:01:57.376232+0000","last_fullsized":"2026-02-24T18:01:56.374083+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-24T18:00:52.474019+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-24T18:00:52.474019+0000","last_clean_scrub_stamp":"2026-02-24T18:00:52.474019+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-25T18:19:12.949262+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00028598099999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9],"acting":[9],"avail_no_missing":["9"],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":61,"reported_epoch":54,"state":"active+undersized+wait","last_fresh":"2026-02-24T18:01:56.374574+0000","last_change":"2026-02-24T18:01:56.374333+0000","last_active":"2026-02-24T18:01:56.374574+0000","last_peered":"2026-02-24T18:01:56.374574+0000","last_clean":"2026-02-24T18:01:46.156753+0000","last_became_active":"2026-02-24T18:01:56.374333+0000","last_became_peered":"2026-02-24T18:01:56.374333+0000","last_unstale":"2026-02-24T18:01:56.374574+0000","last_undegraded":"2026-02-24T18:01:56.374574+0000","last_fullsized":"2026-02-24T18:01:56.374090+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-24T18:00:52.474019+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-24T18:00:52.474019+0000","last_clean_scrub_stamp":"2026-02-24T18:00:52.474019+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-26T03:42:00.638077+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6],"acting":[6],"avail_no_missing":["6"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.3","version":"18'1","reported_seq":84,"reported_epoch":54,"state":"active+undersized+degraded+wait","last_fresh":"2026-02-24T18:01:56.374616+0000","last_change":"2026-02-24T18:01:56.374449+0000","last_active":"2026-02-24T18:01:56.374616+0000","last_peered":"2026-02-24T18:01:56.374616+0000","last_clean":"2026-02-24T18:01:54.618235+0000","last_became_active":"2026-02-24T18:01:56.374449+0000","last_became_peered":"2026-02-24T18:01:56.374449+0000","last_unstale":"2026-02-24T18:01:56.374616+0000","last_undegraded":"2026-02-24T18:01:56.374200+0000","last_fullsized":"2026-02-24T18:01:56.374158+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-24T18:00:52.474019+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-24T18:00:52.474019+0000","last_clean_scrub_stamp":"2026-02-24T18:00:52.474019+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-25T19:06:51.757172+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[{"shards":"3","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"20'2","reported_seq":92,"reported_epoch":54,"state":"active+undersized+degraded+wait","last_fresh":"2026-02-24T18:01:56.374391+0000","last_change":"2026-02-24T18:01:56.374282+0000","last_active":"2026-02-24T18:01:56.374391+0000","last_peered":"2026-02-24T18:01:56.374391+0000","last_clean":"2026-02-24T18:01:54.618334+0000","last_became_active":"2026-02-24T18:01:56.374282+0000","last_became_peered":"2026-02-24T18:01:56.374282+0000","last_unstale":"2026-02-24T18:01:56.374391+0000","last_undegraded":"2026-02-24T18:01:56.374112+0000","last_fullsized":"2026-02-24T18:01:56.374072+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-24T18:00:52.474019+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-24T18:00:52.474019+0000","last_clean_scrub_stamp":"2026-02-24T18:00:52.474019+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-26T04:49:05.797860+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00073412399999999995,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4],"acting":[4],"avail_no_missing":["4"],"object_location_counts":[{"shards":"4","objects":1}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":69,"reported_epoch":53,"state":"active+clean","last_fresh":"2026-02-24T18:01:55.618250+0000","last_change":"2026-02-24T18:01:49.359419+0000","last_active":"2026-02-24T18:01:55.618250+0000","last_peered":"2026-02-24T18:01:55.618250+0000","last_clean":"2026-02-24T18:01:55.618250+0000","last_became_active":"2026-02-24T18:01:49.359167+0000","last_became_peered":"2026-02-24T18:01:49.359167+0000","last_unstale":"2026-02-24T18:01:55.618250+0000","last_undegraded":"2026-02-24T18:01:55.618250+0000","last_fullsized":"2026-02-24T18:01:55.618250+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-24T18:00:52.474019+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-24T18:00:52.474019+0000","last_clean_scrub_stamp":"2026-02-24T18:00:52.474019+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-25T19:47:00.731011+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00026963099999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":80,"reported_epoch":53,"state":"active+clean","last_fresh":"2026-02-24T18:01:55.618229+0000","last_change":"2026-02-24T18:01:49.359091+0000","last_active":"2026-02-24T18:01:55.618229+0000","last_peered":"2026-02-24T18:01:55.618229+0000","last_clean":"2026-02-24T18:01:55.618229+0000","last_became_active":"2026-02-24T18:01:49.358904+0000","last_became_peered":"2026-02-24T18:01:49.358904+0000","last_unstale":"2026-02-24T18:01:55.618229+0000","last_undegraded":"2026-02-24T18:01:55.618229+0000","last_fullsized":"2026-02-24T18:01:55.618229+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-24T18:00:52.474019+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-24T18:00:52.474019+0000","last_clean_scrub_stamp":"2026-02-24T18:00:52.474019+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-26T04:25:16.057495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9],"acting":[5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":12,"acting":12,"num_store_stats":9}],"osd_stats":[{"osd":9,"up_from":40,"seq":171798691856,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":27560,"kb_used_data":600,"kb_used_omap":25,"kb_used_meta":26918,"kb_avail":104830040,"statfs":{"total":107374182400,"available":107345960960,"internally_reserved":0,"allocated":614400,"data_stored":290302,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":25794,"internal_metadata":27564862},"hb_peers":[0,1,2,3,4,5,6,7,8],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":46,"seq":197568495632,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":104857600,"kb_used":27548,"kb_used_data":588,"kb_used_omap":26,"kb_used_meta":26917,"kb_avail":104830052,"statfs":{"total":107374182400,"available":107345973248,"internally_reserved":0,"allocated":602112,"data_stored":282636,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":27091,"internal_metadata":27563565},"hb_peers":[2,3,4,5,6,7,9],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":46,"seq":197568495632,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":27548,"kb_used_data":588,"kb_used_omap":26,"kb_used_meta":26917,"kb_avail":104830052,"statfs":{"total":107374182400,"available":107345973248,"internally_reserved":0,"allocated":602112,"data_stored":282636,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":27086,"internal_metadata":27563570},"hb_peers":[2,3,4,5,6,8,9],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":46,"seq":197568495632,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":27560,"kb_used_data":600,"kb_used_omap":25,"kb_used_meta":26918,"kb_avail":104830040,"statfs":{"total":107374182400,"available":107345960960,"internally_reserved":0,"allocated":614400,"data_stored":290302,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":25788,"internal_metadata":27564868},"hb_peers":[0,1,2,3,4,5,7,8,9],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":46,"seq":197568495632,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":27548,"kb_used_data":588,"kb_used_omap":26,"kb_used_meta":26917,"kb_avail":104830052,"statfs":{"total":107374182400,"available":107345973248,"internally_reserved":0,"allocated":602112,"data_stored":282636,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":27091,"internal_metadata":27563565},"hb_peers":[0,1,2,3,4,6,7,8,9],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":46,"seq":197568495632,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":27564,"kb_used_data":604,"kb_used_omap":28,"kb_used_meta":26915,"kb_avail":104830036,"statfs":{"total":107374182400,"available":107345956864,"internally_reserved":0,"allocated":618496,"data_stored":290321,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":29044,"internal_metadata":27561612},"hb_peers":[0,1,2,3,5,6,7,8,9],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":46,"seq":197568495632,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":27560,"kb_used_data":600,"kb_used_omap":25,"kb_used_meta":26918,"kb_avail":104830040,"statfs":{"total":107374182400,"available":107345960960,"internally_reserved":0,"allocated":614400,"data_stored":290302,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":26437,"internal_metadata":27564219},"hb_peers":[0,1,2,4,5,6,7,8,9],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":46,"seq":197568495632,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":104857600,"kb_used":27560,"kb_used_data":600,"kb_used_omap":22,"kb_used_meta":26921,"kb_avail":104830040,"statfs":{"total":107374182400,"available":107345960960,"internally_reserved":0,"allocated":614400,"data_stored":290302,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":22534,"internal_metadata":27568122},"hb_peers":[3,4,5,6,7,8,9],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":16,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":27540,"kb_used_data":580,"kb_used_omap":26,"kb_used_meta":26917,"kb_avail":104830060,"statfs":{"total":107374182400,"available":107345981440,"internally_reserved":0,"allocated":593920,"data_stored":274921,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":27091,"internal_metadata":27563565},"hb_peers":[2,3,4,5,6,7,8,9],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":56,"seq":240518168593,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":27572,"kb_used_data":612,"kb_used_omap":13,"kb_used_meta":26930,"kb_avail":104830028,"statfs":{"total":107374182400,"available":107345948672,"internally_reserved":0,"allocated":626688,"data_stored":298036,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14071,"internal_metadata":27576585},"hb_peers":[2,3,4,5,6,7,8,9],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-24T18:02:03.380 INFO:tasks.ceph.ceph_manager.ceph:PG 1.5 is not active+clean 2026-02-24T18:02:03.380 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '1.5', 'version': "0'0", 'reported_seq': 96, 'reported_epoch': 55, 'state': 'active+undersized', 'last_fresh': '2026-02-24T18:01:57.376232+0000', 'last_change': '2026-02-24T18:01:57.376232+0000', 'last_active': '2026-02-24T18:01:57.376232+0000', 'last_peered': '2026-02-24T18:01:57.376232+0000', 'last_clean': '2026-02-24T18:01:54.618236+0000', 'last_became_active': '2026-02-24T18:01:56.374381+0000', 'last_became_peered': '2026-02-24T18:01:56.374381+0000', 'last_unstale': '2026-02-24T18:01:57.376232+0000', 'last_undegraded': '2026-02-24T18:01:57.376232+0000', 'last_fullsized': '2026-02-24T18:01:56.374083+0000', 'mapping_epoch': 53, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 17, 'last_epoch_clean': 47, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2026-02-24T18:00:52.474019+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2026-02-24T18:00:52.474019+0000', 'last_clean_scrub_stamp': '2026-02-24T18:00:52.474019+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2026-02-25T18:19:12.949262+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.000285981, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [9], 'acting': [9], 'avail_no_missing': ['9'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 9, 'acting_primary': 9, 'purged_snaps': []} 2026-02-24T18:02:03.380 INFO:tasks.ceph.ceph_manager.ceph:PG 1.4 is not active+clean 2026-02-24T18:02:03.380 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '1.4', 'version': "0'0", 'reported_seq': 61, 'reported_epoch': 54, 'state': 'active+undersized+wait', 'last_fresh': '2026-02-24T18:01:56.374574+0000', 'last_change': '2026-02-24T18:01:56.374333+0000', 'last_active': '2026-02-24T18:01:56.374574+0000', 'last_peered': '2026-02-24T18:01:56.374574+0000', 'last_clean': '2026-02-24T18:01:46.156753+0000', 'last_became_active': '2026-02-24T18:01:56.374333+0000', 'last_became_peered': '2026-02-24T18:01:56.374333+0000', 'last_unstale': '2026-02-24T18:01:56.374574+0000', 'last_undegraded': '2026-02-24T18:01:56.374574+0000', 'last_fullsized': '2026-02-24T18:01:56.374090+0000', 'mapping_epoch': 53, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 17, 'last_epoch_clean': 47, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2026-02-24T18:00:52.474019+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2026-02-24T18:00:52.474019+0000', 'last_clean_scrub_stamp': '2026-02-24T18:00:52.474019+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2026-02-26T03:42:00.638077+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6], 'acting': [6], 'avail_no_missing': ['6'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2026-02-24T18:02:03.380 INFO:tasks.ceph.ceph_manager.ceph:PG 1.3 is not active+clean 2026-02-24T18:02:03.380 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '1.3', 'version': "18'1", 'reported_seq': 84, 'reported_epoch': 54, 'state': 'active+undersized+degraded+wait', 'last_fresh': '2026-02-24T18:01:56.374616+0000', 'last_change': '2026-02-24T18:01:56.374449+0000', 'last_active': '2026-02-24T18:01:56.374616+0000', 'last_peered': '2026-02-24T18:01:56.374616+0000', 'last_clean': '2026-02-24T18:01:54.618235+0000', 'last_became_active': '2026-02-24T18:01:56.374449+0000', 'last_became_peered': '2026-02-24T18:01:56.374449+0000', 'last_unstale': '2026-02-24T18:01:56.374616+0000', 'last_undegraded': '2026-02-24T18:01:56.374200+0000', 'last_fullsized': '2026-02-24T18:01:56.374158+0000', 'mapping_epoch': 53, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 17, 'last_epoch_clean': 47, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2026-02-24T18:00:52.474019+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2026-02-24T18:00:52.474019+0000', 'last_clean_scrub_stamp': '2026-02-24T18:00:52.474019+0000', 'objects_scrubbed': 0, 'log_size': 1, 'log_dups_size': 0, 'ondisk_log_size': 1, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2026-02-25T19:06:51.757172+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3], 'acting': [3], 'avail_no_missing': ['3'], 'object_location_counts': [{'shards': '3', 'objects': 1}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-02-24T18:02:03.381 INFO:tasks.ceph.ceph_manager.ceph:PG 1.2 is not active+clean 2026-02-24T18:02:03.381 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '1.2', 'version': "20'2", 'reported_seq': 92, 'reported_epoch': 54, 'state': 'active+undersized+degraded+wait', 'last_fresh': '2026-02-24T18:01:56.374391+0000', 'last_change': '2026-02-24T18:01:56.374282+0000', 'last_active': '2026-02-24T18:01:56.374391+0000', 'last_peered': '2026-02-24T18:01:56.374391+0000', 'last_clean': '2026-02-24T18:01:54.618334+0000', 'last_became_active': '2026-02-24T18:01:56.374282+0000', 'last_became_peered': '2026-02-24T18:01:56.374282+0000', 'last_unstale': '2026-02-24T18:01:56.374391+0000', 'last_undegraded': '2026-02-24T18:01:56.374112+0000', 'last_fullsized': '2026-02-24T18:01:56.374072+0000', 'mapping_epoch': 53, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 17, 'last_epoch_clean': 47, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2026-02-24T18:00:52.474019+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2026-02-24T18:00:52.474019+0000', 'last_clean_scrub_stamp': '2026-02-24T18:00:52.474019+0000', 'objects_scrubbed': 0, 'log_size': 2, 'log_dups_size': 0, 'ondisk_log_size': 2, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2026-02-26T04:49:05.797860+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.000734124, 'stat_sum': {'num_bytes': 19, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2, 'num_write_kb': 2, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 19, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4], 'acting': [4], 'avail_no_missing': ['4'], 'object_location_counts': [{'shards': '4', 'objects': 1}], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2026-02-24T18:02:03.381 DEBUG:teuthology.orchestra.run.trial068:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2026-02-24T18:02:03.565 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-24T18:02:03.565 INFO:teuthology.orchestra.run.trial068.stdout:{"fsid":"85c6d86c-aad0-4a30-9c50-f24175e8fcf5","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"noout flag(s) set","count":5},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 2/4 objects degraded (50.000%), 2 pgs degraded","count":2},"muted":false}},"mutes":[]},"election_epoch":3,"quorum":[0],"quorum_names":["a"],"quorum_age":82,"monmap":{"epoch":1,"min_mon_release_name":"tentacle","num_mons":1},"osdmap":{"epoch":57,"num_osds":10,"num_up_osds":10,"osd_up_since":1771956118,"num_in_osds":10,"osd_in_since":1771956043,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":4},{"state_name":"active+undersized+degraded+wait","count":2},{"state_name":"active+undersized+wait","count":1},{"state_name":"active+undersized","count":1}],"num_pgs":8,"num_pools":1,"num_objects":2,"data_bytes":19,"bytes_used":282173440,"bytes_avail":1073459650560,"bytes_total":1073741824000,"degraded_objects":2,"degraded_total":4,"degraded_ratio":0.5},"fsmap":{"epoch":1,"btime":"2026-02-24T18:00:41:319879+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs"],"services":{}},"servicemap":{"epoch":3,"modified":"2026-02-24T18:01:49.467350+0000","services":{}},"progress_events":{}} 2026-02-24T18:02:03.576 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2026-02-24T18:02:03.577 DEBUG:teuthology.orchestra.run.trial068:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-24T18:02:03.739 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-24T18:02:03.739 INFO:teuthology.orchestra.run.trial068.stderr:dumped all 2026-02-24T18:02:03.749 INFO:teuthology.orchestra.run.trial068.stdout:{"pg_ready":true,"pg_map":{"version":86,"stamp":"2026-02-24T18:02:03.468903+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":10,"num_per_pool_osds":10,"num_per_pool_omap_osds":8,"kb":1048576000,"kb_used":275944,"kb_used_data":6344,"kb_used_omap":260,"kb_used_meta":269179,"kb_avail":1048300056,"statfs":{"total":1073741824000,"available":1073459257344,"internally_reserved":0,"allocated":6496256,"data_stored":3125658,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":266349,"internal_metadata":275640211},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.093292"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":74,"reported_epoch":57,"state":"active+clean","last_fresh":"2026-02-24T18:01:59.381839+0000","last_change":"2026-02-24T18:01:49.359327+0000","last_active":"2026-02-24T18:01:59.381839+0000","last_peered":"2026-02-24T18:01:59.381839+0000","last_clean":"2026-02-24T18:01:59.381839+0000","last_became_active":"2026-02-24T18:01:49.359085+0000","last_became_peered":"2026-02-24T18:01:49.359085+0000","last_unstale":"2026-02-24T18:01:59.381839+0000","last_undegraded":"2026-02-24T18:01:59.381839+0000","last_fullsized":"2026-02-24T18:01:59.381839+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-24T18:00:52.474019+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-24T18:00:52.474019+0000","last_clean_scrub_stamp":"2026-02-24T18:00:52.474019+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-25T23:52:48.720387+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000255505,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":94,"reported_epoch":57,"state":"active+clean","last_fresh":"2026-02-24T18:01:59.519893+0000","last_change":"2026-02-24T18:01:49.359276+0000","last_active":"2026-02-24T18:01:59.519893+0000","last_peered":"2026-02-24T18:01:59.519893+0000","last_clean":"2026-02-24T18:01:59.519893+0000","last_became_active":"2026-02-24T18:01:49.359064+0000","last_became_peered":"2026-02-24T18:01:49.359064+0000","last_unstale":"2026-02-24T18:01:59.519893+0000","last_undegraded":"2026-02-24T18:01:59.519893+0000","last_fullsized":"2026-02-24T18:01:59.519893+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-24T18:00:52.474019+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-24T18:00:52.474019+0000","last_clean_scrub_stamp":"2026-02-24T18:00:52.474019+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-26T05:08:30.898058+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00029952599999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":106,"reported_epoch":57,"state":"active+clean","last_fresh":"2026-02-24T18:01:59.678373+0000","last_change":"2026-02-24T18:01:59.678303+0000","last_active":"2026-02-24T18:01:59.678373+0000","last_peered":"2026-02-24T18:01:59.678373+0000","last_clean":"2026-02-24T18:01:59.678373+0000","last_became_active":"2026-02-24T18:01:59.678129+0000","last_became_peered":"2026-02-24T18:01:59.678129+0000","last_unstale":"2026-02-24T18:01:59.678373+0000","last_undegraded":"2026-02-24T18:01:59.678373+0000","last_fullsized":"2026-02-24T18:01:59.678373+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-24T18:00:52.474019+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-24T18:00:52.474019+0000","last_clean_scrub_stamp":"2026-02-24T18:00:52.474019+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-25T19:04:42.594324+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00028598099999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0],"acting":[9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":83,"reported_epoch":57,"state":"active+clean","last_fresh":"2026-02-24T18:01:59.520647+0000","last_change":"2026-02-24T18:01:59.520598+0000","last_active":"2026-02-24T18:01:59.520647+0000","last_peered":"2026-02-24T18:01:59.520647+0000","last_clean":"2026-02-24T18:01:59.520647+0000","last_became_active":"2026-02-24T18:01:59.520409+0000","last_became_peered":"2026-02-24T18:01:59.520409+0000","last_unstale":"2026-02-24T18:01:59.520647+0000","last_undegraded":"2026-02-24T18:01:59.520647+0000","last_fullsized":"2026-02-24T18:01:59.520647+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-24T18:00:52.474019+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-24T18:00:52.474019+0000","last_clean_scrub_stamp":"2026-02-24T18:00:52.474019+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-25T18:19:12.949262+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.3","version":"18'1","reported_seq":96,"reported_epoch":57,"state":"active+clean","last_fresh":"2026-02-24T18:01:59.382923+0000","last_change":"2026-02-24T18:01:59.382832+0000","last_active":"2026-02-24T18:01:59.382923+0000","last_peered":"2026-02-24T18:01:59.382923+0000","last_clean":"2026-02-24T18:01:59.382923+0000","last_became_active":"2026-02-24T18:01:59.382604+0000","last_became_peered":"2026-02-24T18:01:59.382604+0000","last_unstale":"2026-02-24T18:01:59.382923+0000","last_undegraded":"2026-02-24T18:01:59.382923+0000","last_fullsized":"2026-02-24T18:01:59.382923+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-24T18:00:52.474019+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-24T18:00:52.474019+0000","last_clean_scrub_stamp":"2026-02-24T18:00:52.474019+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-26T03:46:41.523096+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"20'2","reported_seq":104,"reported_epoch":57,"state":"active+clean","last_fresh":"2026-02-24T18:01:59.382708+0000","last_change":"2026-02-24T18:01:59.382647+0000","last_active":"2026-02-24T18:01:59.382708+0000","last_peered":"2026-02-24T18:01:59.382708+0000","last_clean":"2026-02-24T18:01:59.382708+0000","last_became_active":"2026-02-24T18:01:59.382500+0000","last_became_peered":"2026-02-24T18:01:59.382500+0000","last_unstale":"2026-02-24T18:01:59.382708+0000","last_undegraded":"2026-02-24T18:01:59.382708+0000","last_fullsized":"2026-02-24T18:01:59.382708+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-24T18:00:52.474019+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-24T18:00:52.474019+0000","last_clean_scrub_stamp":"2026-02-24T18:00:52.474019+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-26T03:00:38.208451+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00073412399999999995,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":77,"reported_epoch":57,"state":"active+clean","last_fresh":"2026-02-24T18:01:59.677791+0000","last_change":"2026-02-24T18:01:49.359419+0000","last_active":"2026-02-24T18:01:59.677791+0000","last_peered":"2026-02-24T18:01:59.677791+0000","last_clean":"2026-02-24T18:01:59.677791+0000","last_became_active":"2026-02-24T18:01:49.359167+0000","last_became_peered":"2026-02-24T18:01:49.359167+0000","last_unstale":"2026-02-24T18:01:59.677791+0000","last_undegraded":"2026-02-24T18:01:59.677791+0000","last_fullsized":"2026-02-24T18:01:59.677791+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-24T18:00:52.474019+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-24T18:00:52.474019+0000","last_clean_scrub_stamp":"2026-02-24T18:00:52.474019+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-25T19:47:00.731011+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00026963099999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":88,"reported_epoch":57,"state":"active+clean","last_fresh":"2026-02-24T18:01:59.677712+0000","last_change":"2026-02-24T18:01:49.359091+0000","last_active":"2026-02-24T18:01:59.677712+0000","last_peered":"2026-02-24T18:01:59.677712+0000","last_clean":"2026-02-24T18:01:59.677712+0000","last_became_active":"2026-02-24T18:01:49.358904+0000","last_became_peered":"2026-02-24T18:01:49.358904+0000","last_unstale":"2026-02-24T18:01:59.677712+0000","last_undegraded":"2026-02-24T18:01:59.677712+0000","last_fullsized":"2026-02-24T18:01:59.677712+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-24T18:00:52.474019+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-24T18:00:52.474019+0000","last_clean_scrub_stamp":"2026-02-24T18:00:52.474019+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-26T04:25:16.057495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9],"acting":[5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":9}],"osd_stats":[{"osd":9,"up_from":40,"seq":171798691857,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":27596,"kb_used_data":636,"kb_used_omap":25,"kb_used_meta":26918,"kb_avail":104830004,"statfs":{"total":107374182400,"available":107345924096,"internally_reserved":0,"allocated":651264,"data_stored":314176,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":25794,"internal_metadata":27564862},"hb_peers":[0,1,2,3,4,5,6,7,8],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":46,"seq":197568495633,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":104857600,"kb_used":27596,"kb_used_data":636,"kb_used_omap":28,"kb_used_meta":26915,"kb_avail":104830004,"statfs":{"total":107374182400,"available":107345924096,"internally_reserved":0,"allocated":651264,"data_stored":314176,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":29044,"internal_metadata":27561612},"hb_peers":[0,1,2,3,4,5,6,7,9],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":46,"seq":197568495633,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":27596,"kb_used_data":636,"kb_used_omap":28,"kb_used_meta":26915,"kb_avail":104830004,"statfs":{"total":107374182400,"available":107345924096,"internally_reserved":0,"allocated":651264,"data_stored":314176,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":29690,"internal_metadata":27560966},"hb_peers":[0,1,2,3,4,5,6,8,9],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":46,"seq":197568495633,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":27596,"kb_used_data":636,"kb_used_omap":27,"kb_used_meta":26916,"kb_avail":104830004,"statfs":{"total":107374182400,"available":107345924096,"internally_reserved":0,"allocated":651264,"data_stored":314176,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":27741,"internal_metadata":27562915},"hb_peers":[0,1,2,3,4,5,7,8,9],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":46,"seq":197568495633,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":27596,"kb_used_data":636,"kb_used_omap":28,"kb_used_meta":26915,"kb_avail":104830004,"statfs":{"total":107374182400,"available":107345924096,"internally_reserved":0,"allocated":651264,"data_stored":314176,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":29695,"internal_metadata":27560961},"hb_peers":[0,1,2,3,4,6,7,8,9],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":46,"seq":197568495633,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":27600,"kb_used_data":640,"kb_used_omap":29,"kb_used_meta":26914,"kb_avail":104830000,"statfs":{"total":107374182400,"available":107345920000,"internally_reserved":0,"allocated":655360,"data_stored":314195,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":30346,"internal_metadata":27560310},"hb_peers":[0,1,2,3,5,6,7,8,9],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":46,"seq":197568495633,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":27596,"kb_used_data":636,"kb_used_omap":27,"kb_used_meta":26916,"kb_avail":104830004,"statfs":{"total":107374182400,"available":107345924096,"internally_reserved":0,"allocated":651264,"data_stored":314176,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":27739,"internal_metadata":27562917},"hb_peers":[0,1,2,4,5,6,7,8,9],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":46,"seq":197568495633,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":104857600,"kb_used":27596,"kb_used_data":636,"kb_used_omap":22,"kb_used_meta":26921,"kb_avail":104830004,"statfs":{"total":107374182400,"available":107345924096,"internally_reserved":0,"allocated":651264,"data_stored":314176,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":22534,"internal_metadata":27568122},"hb_peers":[0,1,3,4,5,6,7,8,9],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":56,"seq":240518168593,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":27600,"kb_used_data":640,"kb_used_omap":28,"kb_used_meta":26915,"kb_avail":104830000,"statfs":{"total":107374182400,"available":107345920000,"internally_reserved":0,"allocated":655360,"data_stored":314195,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":29695,"internal_metadata":27560961},"hb_peers":[0,2,3,4,5,6,7,8,9],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":56,"seq":240518168593,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":27572,"kb_used_data":612,"kb_used_omap":13,"kb_used_meta":26930,"kb_avail":104830028,"statfs":{"total":107374182400,"available":107345948672,"internally_reserved":0,"allocated":626688,"data_stored":298036,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14071,"internal_metadata":27576585},"hb_peers":[2,3,4,5,6,7,8,9],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-24T18:02:06.752 DEBUG:teuthology.orchestra.run.trial068:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-24T18:02:06.910 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-24T18:02:06.911 INFO:teuthology.orchestra.run.trial068.stderr:dumped all 2026-02-24T18:02:06.923 INFO:teuthology.orchestra.run.trial068.stdout:{"pg_ready":true,"pg_map":{"version":87,"stamp":"2026-02-24T18:02:05.469144+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":10,"num_per_pool_osds":10,"num_per_pool_omap_osds":8,"kb":1048576000,"kb_used":275968,"kb_used_data":6368,"kb_used_omap":260,"kb_used_meta":269179,"kb_avail":1048300032,"statfs":{"total":1073741824000,"available":1073459232768,"internally_reserved":0,"allocated":6520832,"data_stored":3141798,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":266349,"internal_metadata":275640211},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.000954"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":74,"reported_epoch":57,"state":"active+clean","last_fresh":"2026-02-24T18:01:59.381839+0000","last_change":"2026-02-24T18:01:49.359327+0000","last_active":"2026-02-24T18:01:59.381839+0000","last_peered":"2026-02-24T18:01:59.381839+0000","last_clean":"2026-02-24T18:01:59.381839+0000","last_became_active":"2026-02-24T18:01:49.359085+0000","last_became_peered":"2026-02-24T18:01:49.359085+0000","last_unstale":"2026-02-24T18:01:59.381839+0000","last_undegraded":"2026-02-24T18:01:59.381839+0000","last_fullsized":"2026-02-24T18:01:59.381839+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-24T18:00:52.474019+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-24T18:00:52.474019+0000","last_clean_scrub_stamp":"2026-02-24T18:00:52.474019+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-25T23:52:48.720387+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000255505,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":94,"reported_epoch":57,"state":"active+clean","last_fresh":"2026-02-24T18:01:59.519893+0000","last_change":"2026-02-24T18:01:49.359276+0000","last_active":"2026-02-24T18:01:59.519893+0000","last_peered":"2026-02-24T18:01:59.519893+0000","last_clean":"2026-02-24T18:01:59.519893+0000","last_became_active":"2026-02-24T18:01:49.359064+0000","last_became_peered":"2026-02-24T18:01:49.359064+0000","last_unstale":"2026-02-24T18:01:59.519893+0000","last_undegraded":"2026-02-24T18:01:59.519893+0000","last_fullsized":"2026-02-24T18:01:59.519893+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-24T18:00:52.474019+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-24T18:00:52.474019+0000","last_clean_scrub_stamp":"2026-02-24T18:00:52.474019+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-26T05:08:30.898058+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00029952599999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":106,"reported_epoch":57,"state":"active+clean","last_fresh":"2026-02-24T18:01:59.678373+0000","last_change":"2026-02-24T18:01:59.678303+0000","last_active":"2026-02-24T18:01:59.678373+0000","last_peered":"2026-02-24T18:01:59.678373+0000","last_clean":"2026-02-24T18:01:59.678373+0000","last_became_active":"2026-02-24T18:01:59.678129+0000","last_became_peered":"2026-02-24T18:01:59.678129+0000","last_unstale":"2026-02-24T18:01:59.678373+0000","last_undegraded":"2026-02-24T18:01:59.678373+0000","last_fullsized":"2026-02-24T18:01:59.678373+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-24T18:00:52.474019+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-24T18:00:52.474019+0000","last_clean_scrub_stamp":"2026-02-24T18:00:52.474019+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-25T19:04:42.594324+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00028598099999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0],"acting":[9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":83,"reported_epoch":57,"state":"active+clean","last_fresh":"2026-02-24T18:01:59.520647+0000","last_change":"2026-02-24T18:01:59.520598+0000","last_active":"2026-02-24T18:01:59.520647+0000","last_peered":"2026-02-24T18:01:59.520647+0000","last_clean":"2026-02-24T18:01:59.520647+0000","last_became_active":"2026-02-24T18:01:59.520409+0000","last_became_peered":"2026-02-24T18:01:59.520409+0000","last_unstale":"2026-02-24T18:01:59.520647+0000","last_undegraded":"2026-02-24T18:01:59.520647+0000","last_fullsized":"2026-02-24T18:01:59.520647+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-24T18:00:52.474019+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-24T18:00:52.474019+0000","last_clean_scrub_stamp":"2026-02-24T18:00:52.474019+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-25T18:19:12.949262+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.3","version":"18'1","reported_seq":96,"reported_epoch":57,"state":"active+clean","last_fresh":"2026-02-24T18:01:59.382923+0000","last_change":"2026-02-24T18:01:59.382832+0000","last_active":"2026-02-24T18:01:59.382923+0000","last_peered":"2026-02-24T18:01:59.382923+0000","last_clean":"2026-02-24T18:01:59.382923+0000","last_became_active":"2026-02-24T18:01:59.382604+0000","last_became_peered":"2026-02-24T18:01:59.382604+0000","last_unstale":"2026-02-24T18:01:59.382923+0000","last_undegraded":"2026-02-24T18:01:59.382923+0000","last_fullsized":"2026-02-24T18:01:59.382923+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-24T18:00:52.474019+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-24T18:00:52.474019+0000","last_clean_scrub_stamp":"2026-02-24T18:00:52.474019+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-26T03:46:41.523096+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"20'2","reported_seq":104,"reported_epoch":57,"state":"active+clean","last_fresh":"2026-02-24T18:01:59.382708+0000","last_change":"2026-02-24T18:01:59.382647+0000","last_active":"2026-02-24T18:01:59.382708+0000","last_peered":"2026-02-24T18:01:59.382708+0000","last_clean":"2026-02-24T18:01:59.382708+0000","last_became_active":"2026-02-24T18:01:59.382500+0000","last_became_peered":"2026-02-24T18:01:59.382500+0000","last_unstale":"2026-02-24T18:01:59.382708+0000","last_undegraded":"2026-02-24T18:01:59.382708+0000","last_fullsized":"2026-02-24T18:01:59.382708+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-24T18:00:52.474019+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-24T18:00:52.474019+0000","last_clean_scrub_stamp":"2026-02-24T18:00:52.474019+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-26T03:00:38.208451+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00073412399999999995,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":77,"reported_epoch":57,"state":"active+clean","last_fresh":"2026-02-24T18:01:59.677791+0000","last_change":"2026-02-24T18:01:49.359419+0000","last_active":"2026-02-24T18:01:59.677791+0000","last_peered":"2026-02-24T18:01:59.677791+0000","last_clean":"2026-02-24T18:01:59.677791+0000","last_became_active":"2026-02-24T18:01:49.359167+0000","last_became_peered":"2026-02-24T18:01:49.359167+0000","last_unstale":"2026-02-24T18:01:59.677791+0000","last_undegraded":"2026-02-24T18:01:59.677791+0000","last_fullsized":"2026-02-24T18:01:59.677791+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-24T18:00:52.474019+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-24T18:00:52.474019+0000","last_clean_scrub_stamp":"2026-02-24T18:00:52.474019+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-25T19:47:00.731011+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00026963099999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":88,"reported_epoch":57,"state":"active+clean","last_fresh":"2026-02-24T18:01:59.677712+0000","last_change":"2026-02-24T18:01:49.359091+0000","last_active":"2026-02-24T18:01:59.677712+0000","last_peered":"2026-02-24T18:01:59.677712+0000","last_clean":"2026-02-24T18:01:59.677712+0000","last_became_active":"2026-02-24T18:01:49.358904+0000","last_became_peered":"2026-02-24T18:01:49.358904+0000","last_unstale":"2026-02-24T18:01:59.677712+0000","last_undegraded":"2026-02-24T18:01:59.677712+0000","last_fullsized":"2026-02-24T18:01:59.677712+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-24T18:00:52.474019+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-24T18:00:52.474019+0000","last_clean_scrub_stamp":"2026-02-24T18:00:52.474019+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-26T04:25:16.057495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9],"acting":[5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":9}],"osd_stats":[{"osd":9,"up_from":40,"seq":171798691857,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":27596,"kb_used_data":636,"kb_used_omap":25,"kb_used_meta":26918,"kb_avail":104830004,"statfs":{"total":107374182400,"available":107345924096,"internally_reserved":0,"allocated":651264,"data_stored":314176,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":25794,"internal_metadata":27564862},"hb_peers":[0,1,2,3,4,5,6,7,8],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":46,"seq":197568495633,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":104857600,"kb_used":27596,"kb_used_data":636,"kb_used_omap":28,"kb_used_meta":26915,"kb_avail":104830004,"statfs":{"total":107374182400,"available":107345924096,"internally_reserved":0,"allocated":651264,"data_stored":314176,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":29044,"internal_metadata":27561612},"hb_peers":[0,1,2,3,4,5,6,7,9],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":46,"seq":197568495633,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":27596,"kb_used_data":636,"kb_used_omap":28,"kb_used_meta":26915,"kb_avail":104830004,"statfs":{"total":107374182400,"available":107345924096,"internally_reserved":0,"allocated":651264,"data_stored":314176,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":29690,"internal_metadata":27560966},"hb_peers":[0,1,2,3,4,5,6,8,9],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":46,"seq":197568495633,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":27596,"kb_used_data":636,"kb_used_omap":27,"kb_used_meta":26916,"kb_avail":104830004,"statfs":{"total":107374182400,"available":107345924096,"internally_reserved":0,"allocated":651264,"data_stored":314176,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":27741,"internal_metadata":27562915},"hb_peers":[0,1,2,3,4,5,7,8,9],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":46,"seq":197568495633,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":27596,"kb_used_data":636,"kb_used_omap":28,"kb_used_meta":26915,"kb_avail":104830004,"statfs":{"total":107374182400,"available":107345924096,"internally_reserved":0,"allocated":651264,"data_stored":314176,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":29695,"internal_metadata":27560961},"hb_peers":[0,1,2,3,4,6,7,8,9],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":46,"seq":197568495633,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":27600,"kb_used_data":640,"kb_used_omap":29,"kb_used_meta":26914,"kb_avail":104830000,"statfs":{"total":107374182400,"available":107345920000,"internally_reserved":0,"allocated":655360,"data_stored":314195,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":30346,"internal_metadata":27560310},"hb_peers":[0,1,2,3,5,6,7,8,9],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":46,"seq":197568495633,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":27596,"kb_used_data":636,"kb_used_omap":27,"kb_used_meta":26916,"kb_avail":104830004,"statfs":{"total":107374182400,"available":107345924096,"internally_reserved":0,"allocated":651264,"data_stored":314176,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":27739,"internal_metadata":27562917},"hb_peers":[0,1,2,4,5,6,7,8,9],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":46,"seq":197568495633,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":104857600,"kb_used":27596,"kb_used_data":636,"kb_used_omap":22,"kb_used_meta":26921,"kb_avail":104830004,"statfs":{"total":107374182400,"available":107345924096,"internally_reserved":0,"allocated":651264,"data_stored":314176,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":22534,"internal_metadata":27568122},"hb_peers":[0,1,3,4,5,6,7,8,9],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":56,"seq":240518168593,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":27600,"kb_used_data":640,"kb_used_omap":28,"kb_used_meta":26915,"kb_avail":104830000,"statfs":{"total":107374182400,"available":107345920000,"internally_reserved":0,"allocated":655360,"data_stored":314195,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":29695,"internal_metadata":27560961},"hb_peers":[0,2,3,4,5,6,7,8,9],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":56,"seq":240518168594,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":27596,"kb_used_data":636,"kb_used_omap":13,"kb_used_meta":26930,"kb_avail":104830004,"statfs":{"total":107374182400,"available":107345924096,"internally_reserved":0,"allocated":651264,"data_stored":314176,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14071,"internal_metadata":27576585},"hb_peers":[1,2,3,4,5,6,7,8,9],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-24T18:02:06.924 INFO:tasks.ceph.ceph_manager.ceph:clean! 2026-02-24T18:02:06.924 DEBUG:teuthology.orchestra.run.trial068:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-24T18:02:07.080 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-24T18:02:07.081 INFO:teuthology.orchestra.run.trial068.stderr:dumped all 2026-02-24T18:02:07.093 INFO:teuthology.orchestra.run.trial068.stdout:{"pg_ready":true,"pg_map":{"version":87,"stamp":"2026-02-24T18:02:05.469144+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":10,"num_per_pool_osds":10,"num_per_pool_omap_osds":8,"kb":1048576000,"kb_used":275968,"kb_used_data":6368,"kb_used_omap":260,"kb_used_meta":269179,"kb_avail":1048300032,"statfs":{"total":1073741824000,"available":1073459232768,"internally_reserved":0,"allocated":6520832,"data_stored":3141798,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":266349,"internal_metadata":275640211},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.000954"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":74,"reported_epoch":57,"state":"active+clean","last_fresh":"2026-02-24T18:01:59.381839+0000","last_change":"2026-02-24T18:01:49.359327+0000","last_active":"2026-02-24T18:01:59.381839+0000","last_peered":"2026-02-24T18:01:59.381839+0000","last_clean":"2026-02-24T18:01:59.381839+0000","last_became_active":"2026-02-24T18:01:49.359085+0000","last_became_peered":"2026-02-24T18:01:49.359085+0000","last_unstale":"2026-02-24T18:01:59.381839+0000","last_undegraded":"2026-02-24T18:01:59.381839+0000","last_fullsized":"2026-02-24T18:01:59.381839+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-24T18:00:52.474019+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-24T18:00:52.474019+0000","last_clean_scrub_stamp":"2026-02-24T18:00:52.474019+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-25T23:52:48.720387+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000255505,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":94,"reported_epoch":57,"state":"active+clean","last_fresh":"2026-02-24T18:01:59.519893+0000","last_change":"2026-02-24T18:01:49.359276+0000","last_active":"2026-02-24T18:01:59.519893+0000","last_peered":"2026-02-24T18:01:59.519893+0000","last_clean":"2026-02-24T18:01:59.519893+0000","last_became_active":"2026-02-24T18:01:49.359064+0000","last_became_peered":"2026-02-24T18:01:49.359064+0000","last_unstale":"2026-02-24T18:01:59.519893+0000","last_undegraded":"2026-02-24T18:01:59.519893+0000","last_fullsized":"2026-02-24T18:01:59.519893+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-24T18:00:52.474019+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-24T18:00:52.474019+0000","last_clean_scrub_stamp":"2026-02-24T18:00:52.474019+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-26T05:08:30.898058+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00029952599999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":106,"reported_epoch":57,"state":"active+clean","last_fresh":"2026-02-24T18:01:59.678373+0000","last_change":"2026-02-24T18:01:59.678303+0000","last_active":"2026-02-24T18:01:59.678373+0000","last_peered":"2026-02-24T18:01:59.678373+0000","last_clean":"2026-02-24T18:01:59.678373+0000","last_became_active":"2026-02-24T18:01:59.678129+0000","last_became_peered":"2026-02-24T18:01:59.678129+0000","last_unstale":"2026-02-24T18:01:59.678373+0000","last_undegraded":"2026-02-24T18:01:59.678373+0000","last_fullsized":"2026-02-24T18:01:59.678373+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-24T18:00:52.474019+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-24T18:00:52.474019+0000","last_clean_scrub_stamp":"2026-02-24T18:00:52.474019+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-25T19:04:42.594324+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00028598099999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0],"acting":[9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":83,"reported_epoch":57,"state":"active+clean","last_fresh":"2026-02-24T18:01:59.520647+0000","last_change":"2026-02-24T18:01:59.520598+0000","last_active":"2026-02-24T18:01:59.520647+0000","last_peered":"2026-02-24T18:01:59.520647+0000","last_clean":"2026-02-24T18:01:59.520647+0000","last_became_active":"2026-02-24T18:01:59.520409+0000","last_became_peered":"2026-02-24T18:01:59.520409+0000","last_unstale":"2026-02-24T18:01:59.520647+0000","last_undegraded":"2026-02-24T18:01:59.520647+0000","last_fullsized":"2026-02-24T18:01:59.520647+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-24T18:00:52.474019+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-24T18:00:52.474019+0000","last_clean_scrub_stamp":"2026-02-24T18:00:52.474019+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-25T18:19:12.949262+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.3","version":"18'1","reported_seq":96,"reported_epoch":57,"state":"active+clean","last_fresh":"2026-02-24T18:01:59.382923+0000","last_change":"2026-02-24T18:01:59.382832+0000","last_active":"2026-02-24T18:01:59.382923+0000","last_peered":"2026-02-24T18:01:59.382923+0000","last_clean":"2026-02-24T18:01:59.382923+0000","last_became_active":"2026-02-24T18:01:59.382604+0000","last_became_peered":"2026-02-24T18:01:59.382604+0000","last_unstale":"2026-02-24T18:01:59.382923+0000","last_undegraded":"2026-02-24T18:01:59.382923+0000","last_fullsized":"2026-02-24T18:01:59.382923+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-24T18:00:52.474019+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-24T18:00:52.474019+0000","last_clean_scrub_stamp":"2026-02-24T18:00:52.474019+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-26T03:46:41.523096+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"20'2","reported_seq":104,"reported_epoch":57,"state":"active+clean","last_fresh":"2026-02-24T18:01:59.382708+0000","last_change":"2026-02-24T18:01:59.382647+0000","last_active":"2026-02-24T18:01:59.382708+0000","last_peered":"2026-02-24T18:01:59.382708+0000","last_clean":"2026-02-24T18:01:59.382708+0000","last_became_active":"2026-02-24T18:01:59.382500+0000","last_became_peered":"2026-02-24T18:01:59.382500+0000","last_unstale":"2026-02-24T18:01:59.382708+0000","last_undegraded":"2026-02-24T18:01:59.382708+0000","last_fullsized":"2026-02-24T18:01:59.382708+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-24T18:00:52.474019+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-24T18:00:52.474019+0000","last_clean_scrub_stamp":"2026-02-24T18:00:52.474019+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-26T03:00:38.208451+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00073412399999999995,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":77,"reported_epoch":57,"state":"active+clean","last_fresh":"2026-02-24T18:01:59.677791+0000","last_change":"2026-02-24T18:01:49.359419+0000","last_active":"2026-02-24T18:01:59.677791+0000","last_peered":"2026-02-24T18:01:59.677791+0000","last_clean":"2026-02-24T18:01:59.677791+0000","last_became_active":"2026-02-24T18:01:49.359167+0000","last_became_peered":"2026-02-24T18:01:49.359167+0000","last_unstale":"2026-02-24T18:01:59.677791+0000","last_undegraded":"2026-02-24T18:01:59.677791+0000","last_fullsized":"2026-02-24T18:01:59.677791+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-24T18:00:52.474019+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-24T18:00:52.474019+0000","last_clean_scrub_stamp":"2026-02-24T18:00:52.474019+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-25T19:47:00.731011+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00026963099999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":88,"reported_epoch":57,"state":"active+clean","last_fresh":"2026-02-24T18:01:59.677712+0000","last_change":"2026-02-24T18:01:49.359091+0000","last_active":"2026-02-24T18:01:59.677712+0000","last_peered":"2026-02-24T18:01:59.677712+0000","last_clean":"2026-02-24T18:01:59.677712+0000","last_became_active":"2026-02-24T18:01:49.358904+0000","last_became_peered":"2026-02-24T18:01:49.358904+0000","last_unstale":"2026-02-24T18:01:59.677712+0000","last_undegraded":"2026-02-24T18:01:59.677712+0000","last_fullsized":"2026-02-24T18:01:59.677712+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-24T18:00:52.474019+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-24T18:00:52.474019+0000","last_clean_scrub_stamp":"2026-02-24T18:00:52.474019+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-26T04:25:16.057495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9],"acting":[5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":9}],"osd_stats":[{"osd":9,"up_from":40,"seq":171798691857,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":27596,"kb_used_data":636,"kb_used_omap":25,"kb_used_meta":26918,"kb_avail":104830004,"statfs":{"total":107374182400,"available":107345924096,"internally_reserved":0,"allocated":651264,"data_stored":314176,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":25794,"internal_metadata":27564862},"hb_peers":[0,1,2,3,4,5,6,7,8],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":46,"seq":197568495633,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":104857600,"kb_used":27596,"kb_used_data":636,"kb_used_omap":28,"kb_used_meta":26915,"kb_avail":104830004,"statfs":{"total":107374182400,"available":107345924096,"internally_reserved":0,"allocated":651264,"data_stored":314176,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":29044,"internal_metadata":27561612},"hb_peers":[0,1,2,3,4,5,6,7,9],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":46,"seq":197568495633,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":27596,"kb_used_data":636,"kb_used_omap":28,"kb_used_meta":26915,"kb_avail":104830004,"statfs":{"total":107374182400,"available":107345924096,"internally_reserved":0,"allocated":651264,"data_stored":314176,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":29690,"internal_metadata":27560966},"hb_peers":[0,1,2,3,4,5,6,8,9],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":46,"seq":197568495633,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":27596,"kb_used_data":636,"kb_used_omap":27,"kb_used_meta":26916,"kb_avail":104830004,"statfs":{"total":107374182400,"available":107345924096,"internally_reserved":0,"allocated":651264,"data_stored":314176,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":27741,"internal_metadata":27562915},"hb_peers":[0,1,2,3,4,5,7,8,9],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":46,"seq":197568495633,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":27596,"kb_used_data":636,"kb_used_omap":28,"kb_used_meta":26915,"kb_avail":104830004,"statfs":{"total":107374182400,"available":107345924096,"internally_reserved":0,"allocated":651264,"data_stored":314176,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":29695,"internal_metadata":27560961},"hb_peers":[0,1,2,3,4,6,7,8,9],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":46,"seq":197568495633,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":27600,"kb_used_data":640,"kb_used_omap":29,"kb_used_meta":26914,"kb_avail":104830000,"statfs":{"total":107374182400,"available":107345920000,"internally_reserved":0,"allocated":655360,"data_stored":314195,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":30346,"internal_metadata":27560310},"hb_peers":[0,1,2,3,5,6,7,8,9],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":46,"seq":197568495633,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":27596,"kb_used_data":636,"kb_used_omap":27,"kb_used_meta":26916,"kb_avail":104830004,"statfs":{"total":107374182400,"available":107345924096,"internally_reserved":0,"allocated":651264,"data_stored":314176,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":27739,"internal_metadata":27562917},"hb_peers":[0,1,2,4,5,6,7,8,9],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":46,"seq":197568495633,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":104857600,"kb_used":27596,"kb_used_data":636,"kb_used_omap":22,"kb_used_meta":26921,"kb_avail":104830004,"statfs":{"total":107374182400,"available":107345924096,"internally_reserved":0,"allocated":651264,"data_stored":314176,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":22534,"internal_metadata":27568122},"hb_peers":[0,1,3,4,5,6,7,8,9],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":56,"seq":240518168593,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":27600,"kb_used_data":640,"kb_used_omap":28,"kb_used_meta":26915,"kb_avail":104830000,"statfs":{"total":107374182400,"available":107345920000,"internally_reserved":0,"allocated":655360,"data_stored":314195,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":29695,"internal_metadata":27560961},"hb_peers":[0,2,3,4,5,6,7,8,9],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":56,"seq":240518168594,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":27596,"kb_used_data":636,"kb_used_omap":13,"kb_used_meta":26930,"kb_avail":104830004,"statfs":{"total":107374182400,"available":107345924096,"internally_reserved":0,"allocated":651264,"data_stored":314176,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14071,"internal_metadata":27576585},"hb_peers":[1,2,3,4,5,6,7,8,9],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-24T18:02:07.094 DEBUG:teuthology.orchestra.run.trial068:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2026-02-24T18:02:07.256 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-24T18:02:07.256 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":57,"fsid":"85c6d86c-aad0-4a30-9c50-f24175e8fcf5","created":"2026-02-24T18:00:41.320093+0000","modified":"2026-02-24T18:01:59.378035+0000","last_up_change":"2026-02-24T18:01:58.375337+0000","last_in_change":"2026-02-24T18:00:43.208336+0000","flags":"noout,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5800064,"flags_set":["noout","pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":10,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":10,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":false,"pools":[{"pool":1,"pool_name":"rbd","create_time":"2026-02-24T18:00:51.802289+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":2,"snap_epoch":20,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.5,"score_stable":2.5,"optimal_score":0.80000001192092896,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1.0000001192092896,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"0685fb8e-c8df-4155-aa67-77c8085c54fa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":15,"last_clean_end":55,"up_from":56,"up_thru":37,"down_at":53,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6826","nonce":4252813754},{"type":"v1","addr":"10.20.193.68:6828","nonce":4252813754}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6882","nonce":4255813754},{"type":"v1","addr":"10.20.193.68:6883","nonce":4255813754}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6839","nonce":4252813754},{"type":"v1","addr":"10.20.193.68:6841","nonce":4252813754}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6835","nonce":4252813754},{"type":"v1","addr":"10.20.193.68:6837","nonce":4252813754}]},"public_addr":"10.20.193.68:6828/4252813754","cluster_addr":"10.20.193.68:6883/4255813754","heartbeat_back_addr":"10.20.193.68:6841/4252813754","heartbeat_front_addr":"10.20.193.68:6837/4252813754","state":["exists","up"]},{"osd":1,"uuid":"e6c8aa5f-faf0-46d2-8185-92c1037749e2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":15,"last_clean_end":55,"up_from":56,"up_thru":56,"down_at":53,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":1878455324},{"type":"v1","addr":"10.20.193.68:6811","nonce":1878455324}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6830","nonce":1881455324},{"type":"v1","addr":"10.20.193.68:6831","nonce":1881455324}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":1878455324},{"type":"v1","addr":"10.20.193.68:6817","nonce":1878455324}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":1878455324},{"type":"v1","addr":"10.20.193.68:6815","nonce":1878455324}]},"public_addr":"10.20.193.68:6811/1878455324","cluster_addr":"10.20.193.68:6831/1881455324","heartbeat_back_addr":"10.20.193.68:6817/1878455324","heartbeat_front_addr":"10.20.193.68:6815/1878455324","state":["exists","up"]},{"osd":2,"uuid":"9de05692-da99-43ea-8eeb-7388526b5abc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":15,"last_clean_end":45,"up_from":46,"up_thru":0,"down_at":43,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6858","nonce":3653189349},{"type":"v1","addr":"10.20.193.68:6859","nonce":3653189349}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6845","nonce":3654189349},{"type":"v1","addr":"10.20.193.68:6868","nonce":3654189349}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6864","nonce":3653189349},{"type":"v1","addr":"10.20.193.68:6866","nonce":3653189349}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6862","nonce":3653189349},{"type":"v1","addr":"10.20.193.68:6863","nonce":3653189349}]},"public_addr":"10.20.193.68:6859/3653189349","cluster_addr":"10.20.193.68:6868/3654189349","heartbeat_back_addr":"10.20.193.68:6866/3653189349","heartbeat_front_addr":"10.20.193.68:6863/3653189349","state":["exists","up"]},{"osd":3,"uuid":"e221ca44-382e-4924-ae30-d1add04b6f0d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":15,"last_clean_end":45,"up_from":46,"up_thru":56,"down_at":43,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6865","nonce":1980192188},{"type":"v1","addr":"10.20.193.68:6867","nonce":1980192188}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6852","nonce":1981192188},{"type":"v1","addr":"10.20.193.68:6876","nonce":1981192188}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6872","nonce":1980192188},{"type":"v1","addr":"10.20.193.68:6873","nonce":1980192188}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6870","nonce":1980192188},{"type":"v1","addr":"10.20.193.68:6871","nonce":1980192188}]},"public_addr":"10.20.193.68:6867/1980192188","cluster_addr":"10.20.193.68:6876/1981192188","heartbeat_back_addr":"10.20.193.68:6873/1980192188","heartbeat_front_addr":"10.20.193.68:6871/1980192188","state":["exists","up"]},{"osd":4,"uuid":"59a23325-5017-4345-ad4b-6cde76f1f74d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":15,"last_clean_end":45,"up_from":46,"up_thru":56,"down_at":43,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6842","nonce":1162412721},{"type":"v1","addr":"10.20.193.68:6843","nonce":1162412721}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6853","nonce":1163412721},{"type":"v1","addr":"10.20.193.68:6877","nonce":1163412721}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6848","nonce":1162412721},{"type":"v1","addr":"10.20.193.68:6849","nonce":1162412721}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6846","nonce":1162412721},{"type":"v1","addr":"10.20.193.68:6847","nonce":1162412721}]},"public_addr":"10.20.193.68:6843/1162412721","cluster_addr":"10.20.193.68:6877/1163412721","heartbeat_back_addr":"10.20.193.68:6849/1162412721","heartbeat_front_addr":"10.20.193.68:6847/1162412721","state":["exists","up"]},{"osd":5,"uuid":"0d906a72-c6d6-4072-b5fb-383a18f376e4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":15,"last_clean_end":45,"up_from":46,"up_thru":46,"down_at":43,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6827","nonce":2140276670},{"type":"v1","addr":"10.20.193.68:6829","nonce":2140276670}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6820","nonce":2141276670},{"type":"v1","addr":"10.20.193.68:6821","nonce":2141276670}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6838","nonce":2140276670},{"type":"v1","addr":"10.20.193.68:6840","nonce":2140276670}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6834","nonce":2140276670},{"type":"v1","addr":"10.20.193.68:6836","nonce":2140276670}]},"public_addr":"10.20.193.68:6829/2140276670","cluster_addr":"10.20.193.68:6821/2141276670","heartbeat_back_addr":"10.20.193.68:6840/2140276670","heartbeat_front_addr":"10.20.193.68:6836/2140276670","state":["exists","up"]},{"osd":6,"uuid":"89445f92-371d-492c-bf3f-814aa6533ad0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":15,"last_clean_end":45,"up_from":46,"up_thru":53,"down_at":43,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6850","nonce":2652457072},{"type":"v1","addr":"10.20.193.68:6851","nonce":2652457072}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6860","nonce":2653457072},{"type":"v1","addr":"10.20.193.68:6869","nonce":2653457072}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6856","nonce":2652457072},{"type":"v1","addr":"10.20.193.68:6857","nonce":2652457072}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6854","nonce":2652457072},{"type":"v1","addr":"10.20.193.68:6855","nonce":2652457072}]},"public_addr":"10.20.193.68:6851/2652457072","cluster_addr":"10.20.193.68:6869/2653457072","heartbeat_back_addr":"10.20.193.68:6857/2652457072","heartbeat_front_addr":"10.20.193.68:6855/2652457072","state":["exists","up"]},{"osd":7,"uuid":"3a2e9d71-2ddc-4b3b-8417-e70a1794c3c2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":15,"last_clean_end":45,"up_from":46,"up_thru":46,"down_at":43,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6874","nonce":1182331931},{"type":"v1","addr":"10.20.193.68:6875","nonce":1182331931}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6844","nonce":1183331931},{"type":"v1","addr":"10.20.193.68:6861","nonce":1183331931}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6880","nonce":1182331931},{"type":"v1","addr":"10.20.193.68:6881","nonce":1182331931}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6878","nonce":1182331931},{"type":"v1","addr":"10.20.193.68:6879","nonce":1182331931}]},"public_addr":"10.20.193.68:6875/1182331931","cluster_addr":"10.20.193.68:6861/1183331931","heartbeat_back_addr":"10.20.193.68:6881/1182331931","heartbeat_front_addr":"10.20.193.68:6879/1182331931","state":["exists","up"]},{"osd":8,"uuid":"98acdd38-bad4-49c7-a681-772ff7e34dcc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":15,"last_clean_end":45,"up_from":46,"up_thru":17,"down_at":43,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3946693881},{"type":"v1","addr":"10.20.193.68:6803","nonce":3946693881}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3948693881},{"type":"v1","addr":"10.20.193.68:6813","nonce":3948693881}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3946693881},{"type":"v1","addr":"10.20.193.68:6809","nonce":3946693881}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3946693881},{"type":"v1","addr":"10.20.193.68:6807","nonce":3946693881}]},"public_addr":"10.20.193.68:6803/3946693881","cluster_addr":"10.20.193.68:6813/3948693881","heartbeat_back_addr":"10.20.193.68:6809/3946693881","heartbeat_front_addr":"10.20.193.68:6807/3946693881","state":["exists","up"]},{"osd":9,"uuid":"e2c6da1d-87c9-4ebf-8234-4356adbb2bb5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":15,"last_clean_end":39,"up_from":40,"up_thru":56,"down_at":37,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6818","nonce":3148430907},{"type":"v1","addr":"10.20.193.68:6819","nonce":3148430907}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3149430907},{"type":"v1","addr":"10.20.193.68:6805","nonce":3149430907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6824","nonce":3148430907},{"type":"v1","addr":"10.20.193.68:6825","nonce":3148430907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6822","nonce":3148430907},{"type":"v1","addr":"10.20.193.68:6823","nonce":3148430907}]},"public_addr":"10.20.193.68:6819/3148430907","cluster_addr":"10.20.193.68:6805/3149430907","heartbeat_back_addr":"10.20.193.68:6825/3148430907","heartbeat_front_addr":"10.20.193.68:6823/3148430907","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"2026-02-24T18:01:55.368629+0000","laggy_probability":0.65700000524520874,"laggy_interval":1,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-24T18:00:45.570554+0000","dead_epoch":54},{"osd":1,"down_stamp":"2026-02-24T18:01:55.368629+0000","laggy_probability":0.65700000524520874,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-24T18:00:45.565655+0000","dead_epoch":54},{"osd":2,"down_stamp":"2026-02-24T18:01:45.344419+0000","laggy_probability":0.30000001192092896,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-24T18:00:45.572517+0000","dead_epoch":44},{"osd":3,"down_stamp":"2026-02-24T18:01:45.344419+0000","laggy_probability":0.30000001192092896,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-24T18:00:45.573166+0000","dead_epoch":44},{"osd":4,"down_stamp":"2026-02-24T18:01:45.344419+0000","laggy_probability":0.30000001192092896,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-24T18:00:45.595183+0000","dead_epoch":44},{"osd":5,"down_stamp":"2026-02-24T18:01:45.344419+0000","laggy_probability":0.30000001192092896,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-24T18:00:45.560129+0000","dead_epoch":43},{"osd":6,"down_stamp":"2026-02-24T18:01:45.344419+0000","laggy_probability":0.30000001192092896,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-24T18:00:45.591572+0000","dead_epoch":44},{"osd":7,"down_stamp":"2026-02-24T18:01:45.344419+0000","laggy_probability":0.30000001192092896,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-24T18:00:45.639584+0000","dead_epoch":44},{"osd":8,"down_stamp":"2026-02-24T18:01:45.344419+0000","laggy_probability":0.50999999046325684,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-24T18:00:45.530957+0000","dead_epoch":43},{"osd":9,"down_stamp":"2026-02-24T18:01:38.329302+0000","laggy_probability":0.30000001192092896,"laggy_interval":1,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-24T18:00:45.534329+0000","dead_epoch":39}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-24T18:02:08.268 INFO:tasks.ceph:Scrubbing osd.0 2026-02-24T18:02:08.269 DEBUG:teuthology.orchestra.run.trial068:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 config set osd_debug_deep_scrub_sleep 0 2026-02-24T18:02:08.345 INFO:teuthology.orchestra.run.trial068.stdout:{ 2026-02-24T18:02:08.345 INFO:teuthology.orchestra.run.trial068.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2026-02-24T18:02:08.345 INFO:teuthology.orchestra.run.trial068.stdout:} 2026-02-24T18:02:08.355 DEBUG:teuthology.orchestra.run.trial068:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 0 2026-02-24T18:02:08.539 INFO:teuthology.orchestra.run.trial068.stderr:instructed osd(s) 0 to deep-scrub 2026-02-24T18:02:08.550 INFO:tasks.ceph:Scrubbing osd.1 2026-02-24T18:02:08.550 DEBUG:teuthology.orchestra.run.trial068:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 config set osd_debug_deep_scrub_sleep 0 2026-02-24T18:02:08.627 INFO:teuthology.orchestra.run.trial068.stdout:{ 2026-02-24T18:02:08.627 INFO:teuthology.orchestra.run.trial068.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2026-02-24T18:02:08.627 INFO:teuthology.orchestra.run.trial068.stdout:} 2026-02-24T18:02:08.638 DEBUG:teuthology.orchestra.run.trial068:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 1 2026-02-24T18:02:08.795 INFO:teuthology.orchestra.run.trial068.stderr:instructed osd(s) 1 to deep-scrub 2026-02-24T18:02:08.806 INFO:tasks.ceph:Scrubbing osd.2 2026-02-24T18:02:08.807 DEBUG:teuthology.orchestra.run.trial068:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 config set osd_debug_deep_scrub_sleep 0 2026-02-24T18:02:08.884 INFO:teuthology.orchestra.run.trial068.stdout:{ 2026-02-24T18:02:08.884 INFO:teuthology.orchestra.run.trial068.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2026-02-24T18:02:08.884 INFO:teuthology.orchestra.run.trial068.stdout:} 2026-02-24T18:02:08.895 DEBUG:teuthology.orchestra.run.trial068:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 2 2026-02-24T18:02:09.056 INFO:teuthology.orchestra.run.trial068.stderr:instructed osd(s) 2 to deep-scrub 2026-02-24T18:02:09.067 INFO:tasks.ceph:Scrubbing osd.3 2026-02-24T18:02:09.068 DEBUG:teuthology.orchestra.run.trial068:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 config set osd_debug_deep_scrub_sleep 0 2026-02-24T18:02:09.142 INFO:teuthology.orchestra.run.trial068.stdout:{ 2026-02-24T18:02:09.142 INFO:teuthology.orchestra.run.trial068.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2026-02-24T18:02:09.143 INFO:teuthology.orchestra.run.trial068.stdout:} 2026-02-24T18:02:09.151 DEBUG:teuthology.orchestra.run.trial068:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 3 2026-02-24T18:02:09.309 INFO:teuthology.orchestra.run.trial068.stderr:instructed osd(s) 3 to deep-scrub 2026-02-24T18:02:09.322 INFO:tasks.ceph:Scrubbing osd.4 2026-02-24T18:02:09.323 DEBUG:teuthology.orchestra.run.trial068:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.4 config set osd_debug_deep_scrub_sleep 0 2026-02-24T18:02:09.407 INFO:teuthology.orchestra.run.trial068.stdout:{ 2026-02-24T18:02:09.407 INFO:teuthology.orchestra.run.trial068.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2026-02-24T18:02:09.408 INFO:teuthology.orchestra.run.trial068.stdout:} 2026-02-24T18:02:09.416 DEBUG:teuthology.orchestra.run.trial068:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 4 2026-02-24T18:02:09.583 INFO:teuthology.orchestra.run.trial068.stderr:instructed osd(s) 4 to deep-scrub 2026-02-24T18:02:09.594 INFO:tasks.ceph:Scrubbing osd.5 2026-02-24T18:02:09.595 DEBUG:teuthology.orchestra.run.trial068:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.5 config set osd_debug_deep_scrub_sleep 0 2026-02-24T18:02:09.670 INFO:teuthology.orchestra.run.trial068.stdout:{ 2026-02-24T18:02:09.671 INFO:teuthology.orchestra.run.trial068.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2026-02-24T18:02:09.671 INFO:teuthology.orchestra.run.trial068.stdout:} 2026-02-24T18:02:09.682 DEBUG:teuthology.orchestra.run.trial068:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 5 2026-02-24T18:02:09.839 INFO:teuthology.orchestra.run.trial068.stderr:instructed osd(s) 5 to deep-scrub 2026-02-24T18:02:09.850 INFO:tasks.ceph:Scrubbing osd.6 2026-02-24T18:02:09.850 DEBUG:teuthology.orchestra.run.trial068:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.6 config set osd_debug_deep_scrub_sleep 0 2026-02-24T18:02:09.925 INFO:teuthology.orchestra.run.trial068.stdout:{ 2026-02-24T18:02:09.926 INFO:teuthology.orchestra.run.trial068.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2026-02-24T18:02:09.926 INFO:teuthology.orchestra.run.trial068.stdout:} 2026-02-24T18:02:09.935 DEBUG:teuthology.orchestra.run.trial068:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 6 2026-02-24T18:02:10.086 INFO:teuthology.orchestra.run.trial068.stderr:instructed osd(s) 6 to deep-scrub 2026-02-24T18:02:10.098 INFO:tasks.ceph:Scrubbing osd.7 2026-02-24T18:02:10.098 DEBUG:teuthology.orchestra.run.trial068:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.7 config set osd_debug_deep_scrub_sleep 0 2026-02-24T18:02:10.173 INFO:teuthology.orchestra.run.trial068.stdout:{ 2026-02-24T18:02:10.173 INFO:teuthology.orchestra.run.trial068.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2026-02-24T18:02:10.173 INFO:teuthology.orchestra.run.trial068.stdout:} 2026-02-24T18:02:10.182 DEBUG:teuthology.orchestra.run.trial068:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 7 2026-02-24T18:02:10.336 INFO:teuthology.orchestra.run.trial068.stderr:instructed osd(s) 7 to deep-scrub 2026-02-24T18:02:10.348 INFO:tasks.ceph:Scrubbing osd.8 2026-02-24T18:02:10.348 DEBUG:teuthology.orchestra.run.trial068:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.8 config set osd_debug_deep_scrub_sleep 0 2026-02-24T18:02:10.423 INFO:teuthology.orchestra.run.trial068.stdout:{ 2026-02-24T18:02:10.423 INFO:teuthology.orchestra.run.trial068.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2026-02-24T18:02:10.423 INFO:teuthology.orchestra.run.trial068.stdout:} 2026-02-24T18:02:10.432 DEBUG:teuthology.orchestra.run.trial068:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 8 2026-02-24T18:02:10.588 INFO:teuthology.orchestra.run.trial068.stderr:instructed osd(s) 8 to deep-scrub 2026-02-24T18:02:10.601 INFO:tasks.ceph:Scrubbing osd.9 2026-02-24T18:02:10.602 DEBUG:teuthology.orchestra.run.trial068:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.9 config set osd_debug_deep_scrub_sleep 0 2026-02-24T18:02:10.678 INFO:teuthology.orchestra.run.trial068.stdout:{ 2026-02-24T18:02:10.678 INFO:teuthology.orchestra.run.trial068.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2026-02-24T18:02:10.678 INFO:teuthology.orchestra.run.trial068.stdout:} 2026-02-24T18:02:10.688 DEBUG:teuthology.orchestra.run.trial068:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 9 2026-02-24T18:02:10.850 INFO:teuthology.orchestra.run.trial068.stderr:instructed osd(s) 9 to deep-scrub 2026-02-24T18:02:10.861 DEBUG:teuthology.orchestra.run.trial068:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-24T18:02:11.025 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-24T18:02:11.025 INFO:teuthology.orchestra.run.trial068.stderr:dumped all 2026-02-24T18:02:11.036 INFO:teuthology.orchestra.run.trial068.stdout:{"pg_ready":true,"pg_map":{"version":89,"stamp":"2026-02-24T18:02:09.469589+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":10,"num_per_pool_osds":10,"num_per_pool_omap_osds":8,"kb":1048576000,"kb_used":275968,"kb_used_data":6368,"kb_used_omap":260,"kb_used_meta":269179,"kb_avail":1048300032,"statfs":{"total":1073741824000,"available":1073459232768,"internally_reserved":0,"allocated":6520832,"data_stored":3141798,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":266349,"internal_metadata":275640211},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.089037"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":74,"reported_epoch":57,"state":"active+clean","last_fresh":"2026-02-24T18:01:59.381839+0000","last_change":"2026-02-24T18:01:49.359327+0000","last_active":"2026-02-24T18:01:59.381839+0000","last_peered":"2026-02-24T18:01:59.381839+0000","last_clean":"2026-02-24T18:01:59.381839+0000","last_became_active":"2026-02-24T18:01:49.359085+0000","last_became_peered":"2026-02-24T18:01:49.359085+0000","last_unstale":"2026-02-24T18:01:59.381839+0000","last_undegraded":"2026-02-24T18:01:59.381839+0000","last_fullsized":"2026-02-24T18:01:59.381839+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-24T18:00:52.474019+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-24T18:00:52.474019+0000","last_clean_scrub_stamp":"2026-02-24T18:00:52.474019+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-25T23:52:48.720387+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000255505,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":94,"reported_epoch":57,"state":"active+clean","last_fresh":"2026-02-24T18:01:59.519893+0000","last_change":"2026-02-24T18:01:49.359276+0000","last_active":"2026-02-24T18:01:59.519893+0000","last_peered":"2026-02-24T18:01:59.519893+0000","last_clean":"2026-02-24T18:01:59.519893+0000","last_became_active":"2026-02-24T18:01:49.359064+0000","last_became_peered":"2026-02-24T18:01:49.359064+0000","last_unstale":"2026-02-24T18:01:59.519893+0000","last_undegraded":"2026-02-24T18:01:59.519893+0000","last_fullsized":"2026-02-24T18:01:59.519893+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-24T18:00:52.474019+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-24T18:00:52.474019+0000","last_clean_scrub_stamp":"2026-02-24T18:00:52.474019+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-26T05:08:30.898058+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00029952599999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":106,"reported_epoch":57,"state":"active+clean","last_fresh":"2026-02-24T18:01:59.678373+0000","last_change":"2026-02-24T18:01:59.678303+0000","last_active":"2026-02-24T18:01:59.678373+0000","last_peered":"2026-02-24T18:01:59.678373+0000","last_clean":"2026-02-24T18:01:59.678373+0000","last_became_active":"2026-02-24T18:01:59.678129+0000","last_became_peered":"2026-02-24T18:01:59.678129+0000","last_unstale":"2026-02-24T18:01:59.678373+0000","last_undegraded":"2026-02-24T18:01:59.678373+0000","last_fullsized":"2026-02-24T18:01:59.678373+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-24T18:00:52.474019+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-24T18:00:52.474019+0000","last_clean_scrub_stamp":"2026-02-24T18:00:52.474019+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-25T19:04:42.594324+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00028598099999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0],"acting":[9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":83,"reported_epoch":57,"state":"active+clean","last_fresh":"2026-02-24T18:01:59.520647+0000","last_change":"2026-02-24T18:01:59.520598+0000","last_active":"2026-02-24T18:01:59.520647+0000","last_peered":"2026-02-24T18:01:59.520647+0000","last_clean":"2026-02-24T18:01:59.520647+0000","last_became_active":"2026-02-24T18:01:59.520409+0000","last_became_peered":"2026-02-24T18:01:59.520409+0000","last_unstale":"2026-02-24T18:01:59.520647+0000","last_undegraded":"2026-02-24T18:01:59.520647+0000","last_fullsized":"2026-02-24T18:01:59.520647+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-24T18:00:52.474019+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-24T18:00:52.474019+0000","last_clean_scrub_stamp":"2026-02-24T18:00:52.474019+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-25T18:19:12.949262+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.3","version":"18'1","reported_seq":96,"reported_epoch":57,"state":"active+clean","last_fresh":"2026-02-24T18:01:59.382923+0000","last_change":"2026-02-24T18:01:59.382832+0000","last_active":"2026-02-24T18:01:59.382923+0000","last_peered":"2026-02-24T18:01:59.382923+0000","last_clean":"2026-02-24T18:01:59.382923+0000","last_became_active":"2026-02-24T18:01:59.382604+0000","last_became_peered":"2026-02-24T18:01:59.382604+0000","last_unstale":"2026-02-24T18:01:59.382923+0000","last_undegraded":"2026-02-24T18:01:59.382923+0000","last_fullsized":"2026-02-24T18:01:59.382923+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-24T18:00:52.474019+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-24T18:00:52.474019+0000","last_clean_scrub_stamp":"2026-02-24T18:00:52.474019+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-26T03:46:41.523096+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"20'2","reported_seq":104,"reported_epoch":57,"state":"active+clean","last_fresh":"2026-02-24T18:01:59.382708+0000","last_change":"2026-02-24T18:01:59.382647+0000","last_active":"2026-02-24T18:01:59.382708+0000","last_peered":"2026-02-24T18:01:59.382708+0000","last_clean":"2026-02-24T18:01:59.382708+0000","last_became_active":"2026-02-24T18:01:59.382500+0000","last_became_peered":"2026-02-24T18:01:59.382500+0000","last_unstale":"2026-02-24T18:01:59.382708+0000","last_undegraded":"2026-02-24T18:01:59.382708+0000","last_fullsized":"2026-02-24T18:01:59.382708+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-24T18:00:52.474019+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-24T18:00:52.474019+0000","last_clean_scrub_stamp":"2026-02-24T18:00:52.474019+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-26T03:00:38.208451+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00073412399999999995,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":77,"reported_epoch":57,"state":"active+clean","last_fresh":"2026-02-24T18:01:59.677791+0000","last_change":"2026-02-24T18:01:49.359419+0000","last_active":"2026-02-24T18:01:59.677791+0000","last_peered":"2026-02-24T18:01:59.677791+0000","last_clean":"2026-02-24T18:01:59.677791+0000","last_became_active":"2026-02-24T18:01:49.359167+0000","last_became_peered":"2026-02-24T18:01:49.359167+0000","last_unstale":"2026-02-24T18:01:59.677791+0000","last_undegraded":"2026-02-24T18:01:59.677791+0000","last_fullsized":"2026-02-24T18:01:59.677791+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-24T18:00:52.474019+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-24T18:00:52.474019+0000","last_clean_scrub_stamp":"2026-02-24T18:00:52.474019+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-25T19:47:00.731011+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00026963099999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":88,"reported_epoch":57,"state":"active+clean","last_fresh":"2026-02-24T18:01:59.677712+0000","last_change":"2026-02-24T18:01:49.359091+0000","last_active":"2026-02-24T18:01:59.677712+0000","last_peered":"2026-02-24T18:01:59.677712+0000","last_clean":"2026-02-24T18:01:59.677712+0000","last_became_active":"2026-02-24T18:01:49.358904+0000","last_became_peered":"2026-02-24T18:01:49.358904+0000","last_unstale":"2026-02-24T18:01:59.677712+0000","last_undegraded":"2026-02-24T18:01:59.677712+0000","last_fullsized":"2026-02-24T18:01:59.677712+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-24T18:00:52.474019+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-24T18:00:52.474019+0000","last_clean_scrub_stamp":"2026-02-24T18:00:52.474019+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-26T04:25:16.057495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9],"acting":[5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":9}],"osd_stats":[{"osd":9,"up_from":40,"seq":171798691858,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":27596,"kb_used_data":636,"kb_used_omap":25,"kb_used_meta":26918,"kb_avail":104830004,"statfs":{"total":107374182400,"available":107345924096,"internally_reserved":0,"allocated":651264,"data_stored":314176,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":25794,"internal_metadata":27564862},"hb_peers":[0,1,2,3,4,5,6,7,8],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":46,"seq":197568495634,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":104857600,"kb_used":27596,"kb_used_data":636,"kb_used_omap":28,"kb_used_meta":26915,"kb_avail":104830004,"statfs":{"total":107374182400,"available":107345924096,"internally_reserved":0,"allocated":651264,"data_stored":314176,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":29044,"internal_metadata":27561612},"hb_peers":[0,1,2,3,4,5,6,7,9],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":46,"seq":197568495634,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":27596,"kb_used_data":636,"kb_used_omap":28,"kb_used_meta":26915,"kb_avail":104830004,"statfs":{"total":107374182400,"available":107345924096,"internally_reserved":0,"allocated":651264,"data_stored":314176,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":29690,"internal_metadata":27560966},"hb_peers":[0,1,2,3,4,5,6,8,9],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":46,"seq":197568495634,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":27596,"kb_used_data":636,"kb_used_omap":27,"kb_used_meta":26916,"kb_avail":104830004,"statfs":{"total":107374182400,"available":107345924096,"internally_reserved":0,"allocated":651264,"data_stored":314176,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":27741,"internal_metadata":27562915},"hb_peers":[0,1,2,3,4,5,7,8,9],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":46,"seq":197568495634,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":27596,"kb_used_data":636,"kb_used_omap":28,"kb_used_meta":26915,"kb_avail":104830004,"statfs":{"total":107374182400,"available":107345924096,"internally_reserved":0,"allocated":651264,"data_stored":314176,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":29695,"internal_metadata":27560961},"hb_peers":[0,1,2,3,4,6,7,8,9],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":46,"seq":197568495634,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":27600,"kb_used_data":640,"kb_used_omap":29,"kb_used_meta":26914,"kb_avail":104830000,"statfs":{"total":107374182400,"available":107345920000,"internally_reserved":0,"allocated":655360,"data_stored":314195,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":30346,"internal_metadata":27560310},"hb_peers":[0,1,2,3,5,6,7,8,9],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":46,"seq":197568495634,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":27596,"kb_used_data":636,"kb_used_omap":27,"kb_used_meta":26916,"kb_avail":104830004,"statfs":{"total":107374182400,"available":107345924096,"internally_reserved":0,"allocated":651264,"data_stored":314176,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":27739,"internal_metadata":27562917},"hb_peers":[0,1,2,4,5,6,7,8,9],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":46,"seq":197568495634,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":104857600,"kb_used":27596,"kb_used_data":636,"kb_used_omap":22,"kb_used_meta":26921,"kb_avail":104830004,"statfs":{"total":107374182400,"available":107345924096,"internally_reserved":0,"allocated":651264,"data_stored":314176,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":22534,"internal_metadata":27568122},"hb_peers":[0,1,3,4,5,6,7,8,9],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":56,"seq":240518168594,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":27600,"kb_used_data":640,"kb_used_omap":28,"kb_used_meta":26915,"kb_avail":104830000,"statfs":{"total":107374182400,"available":107345920000,"internally_reserved":0,"allocated":655360,"data_stored":314195,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":29695,"internal_metadata":27560961},"hb_peers":[0,2,3,4,5,6,7,8,9],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":56,"seq":240518168595,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":27596,"kb_used_data":636,"kb_used_omap":13,"kb_used_meta":26930,"kb_avail":104830004,"statfs":{"total":107374182400,"available":107345924096,"internally_reserved":0,"allocated":651264,"data_stored":314176,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14071,"internal_metadata":27576585},"hb_peers":[1,2,3,4,5,6,7,8,9],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-24T18:02:11.038 INFO:tasks.ceph:pgid 1.7 last_scrub_stamp 2026-02-24T18:00:52.474019+0000 time.struct_time(tm_year=2026, tm_mon=2, tm_mday=24, tm_hour=18, tm_min=0, tm_sec=52, tm_wday=1, tm_yday=55, tm_isdst=-1) <= time.struct_time(tm_year=2026, tm_mon=2, tm_mday=24, tm_hour=18, tm_min=2, tm_sec=7, tm_wday=1, tm_yday=55, tm_isdst=0) 2026-02-24T18:02:11.038 INFO:tasks.ceph:pgid 1.6 last_scrub_stamp 2026-02-24T18:00:52.474019+0000 time.struct_time(tm_year=2026, tm_mon=2, tm_mday=24, tm_hour=18, tm_min=0, tm_sec=52, tm_wday=1, tm_yday=55, tm_isdst=-1) <= time.struct_time(tm_year=2026, tm_mon=2, tm_mday=24, tm_hour=18, tm_min=2, tm_sec=7, tm_wday=1, tm_yday=55, tm_isdst=0) 2026-02-24T18:02:11.038 INFO:tasks.ceph:pgid 1.5 last_scrub_stamp 2026-02-24T18:00:52.474019+0000 time.struct_time(tm_year=2026, tm_mon=2, tm_mday=24, tm_hour=18, tm_min=0, tm_sec=52, tm_wday=1, tm_yday=55, tm_isdst=-1) <= time.struct_time(tm_year=2026, tm_mon=2, tm_mday=24, tm_hour=18, tm_min=2, tm_sec=7, tm_wday=1, tm_yday=55, tm_isdst=0) 2026-02-24T18:02:11.038 INFO:tasks.ceph:pgid 1.4 last_scrub_stamp 2026-02-24T18:00:52.474019+0000 time.struct_time(tm_year=2026, tm_mon=2, tm_mday=24, tm_hour=18, tm_min=0, tm_sec=52, tm_wday=1, tm_yday=55, tm_isdst=-1) <= time.struct_time(tm_year=2026, tm_mon=2, tm_mday=24, tm_hour=18, tm_min=2, tm_sec=7, tm_wday=1, tm_yday=55, tm_isdst=0) 2026-02-24T18:02:11.038 INFO:tasks.ceph:pgid 1.3 last_scrub_stamp 2026-02-24T18:00:52.474019+0000 time.struct_time(tm_year=2026, tm_mon=2, tm_mday=24, tm_hour=18, tm_min=0, tm_sec=52, tm_wday=1, tm_yday=55, tm_isdst=-1) <= time.struct_time(tm_year=2026, tm_mon=2, tm_mday=24, tm_hour=18, tm_min=2, tm_sec=7, tm_wday=1, tm_yday=55, tm_isdst=0) 2026-02-24T18:02:11.038 INFO:tasks.ceph:pgid 1.2 last_scrub_stamp 2026-02-24T18:00:52.474019+0000 time.struct_time(tm_year=2026, tm_mon=2, tm_mday=24, tm_hour=18, tm_min=0, tm_sec=52, tm_wday=1, tm_yday=55, tm_isdst=-1) <= time.struct_time(tm_year=2026, tm_mon=2, tm_mday=24, tm_hour=18, tm_min=2, tm_sec=7, tm_wday=1, tm_yday=55, tm_isdst=0) 2026-02-24T18:02:11.038 INFO:tasks.ceph:pgid 1.1 last_scrub_stamp 2026-02-24T18:00:52.474019+0000 time.struct_time(tm_year=2026, tm_mon=2, tm_mday=24, tm_hour=18, tm_min=0, tm_sec=52, tm_wday=1, tm_yday=55, tm_isdst=-1) <= time.struct_time(tm_year=2026, tm_mon=2, tm_mday=24, tm_hour=18, tm_min=2, tm_sec=7, tm_wday=1, tm_yday=55, tm_isdst=0) 2026-02-24T18:02:11.038 INFO:tasks.ceph:pgid 1.0 last_scrub_stamp 2026-02-24T18:00:52.474019+0000 time.struct_time(tm_year=2026, tm_mon=2, tm_mday=24, tm_hour=18, tm_min=0, tm_sec=52, tm_wday=1, tm_yday=55, tm_isdst=-1) <= time.struct_time(tm_year=2026, tm_mon=2, tm_mday=24, tm_hour=18, tm_min=2, tm_sec=7, tm_wday=1, tm_yday=55, tm_isdst=0) 2026-02-24T18:02:11.038 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2026-02-24T18:02:31.039 DEBUG:teuthology.orchestra.run.trial068:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-24T18:02:31.201 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-24T18:02:31.201 INFO:teuthology.orchestra.run.trial068.stderr:dumped all 2026-02-24T18:02:31.211 INFO:teuthology.orchestra.run.trial068.stdout:{"pg_ready":true,"pg_map":{"version":99,"stamp":"2026-02-24T18:02:29.471542+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":5,"ondisk_log_size":5,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":10,"num_per_pool_osds":10,"num_per_pool_omap_osds":8,"kb":1048576000,"kb_used":275968,"kb_used_data":6368,"kb_used_omap":260,"kb_used_meta":269179,"kb_avail":1048300032,"statfs":{"total":1073741824000,"available":1073459232768,"internally_reserved":0,"allocated":6520832,"data_stored":3141798,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":266349,"internal_metadata":275640211},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001034"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":82,"reported_epoch":57,"state":"active+clean","last_fresh":"2026-02-24T18:02:11.033211+0000","last_change":"2026-02-24T18:02:11.033211+0000","last_active":"2026-02-24T18:02:11.033211+0000","last_peered":"2026-02-24T18:02:11.033211+0000","last_clean":"2026-02-24T18:02:11.033211+0000","last_became_active":"2026-02-24T18:01:49.359085+0000","last_became_peered":"2026-02-24T18:01:49.359085+0000","last_unstale":"2026-02-24T18:02:11.033211+0000","last_undegraded":"2026-02-24T18:02:11.033211+0000","last_fullsized":"2026-02-24T18:02:11.033211+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-24T18:02:11.033157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-24T18:02:11.033157+0000","last_clean_scrub_stamp":"2026-02-24T18:02:11.033157+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-26T02:57:42.308558+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000255505,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":102,"reported_epoch":57,"state":"active+clean","last_fresh":"2026-02-24T18:02:10.526895+0000","last_change":"2026-02-24T18:02:10.526895+0000","last_active":"2026-02-24T18:02:10.526895+0000","last_peered":"2026-02-24T18:02:10.526895+0000","last_clean":"2026-02-24T18:02:10.526895+0000","last_became_active":"2026-02-24T18:01:49.359064+0000","last_became_peered":"2026-02-24T18:01:49.359064+0000","last_unstale":"2026-02-24T18:02:10.526895+0000","last_undegraded":"2026-02-24T18:02:10.526895+0000","last_fullsized":"2026-02-24T18:02:10.526895+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-24T18:02:10.526853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-24T18:02:10.526853+0000","last_clean_scrub_stamp":"2026-02-24T18:02:10.526853+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-25T18:46:16.097196+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00029952599999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":114,"reported_epoch":57,"state":"active+clean","last_fresh":"2026-02-24T18:02:11.527919+0000","last_change":"2026-02-24T18:02:11.527919+0000","last_active":"2026-02-24T18:02:11.527919+0000","last_peered":"2026-02-24T18:02:11.527919+0000","last_clean":"2026-02-24T18:02:11.527919+0000","last_became_active":"2026-02-24T18:01:59.678129+0000","last_became_peered":"2026-02-24T18:01:59.678129+0000","last_unstale":"2026-02-24T18:02:11.527919+0000","last_undegraded":"2026-02-24T18:02:11.527919+0000","last_fullsized":"2026-02-24T18:02:11.527919+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-24T18:02:11.527870+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-24T18:02:11.527870+0000","last_clean_scrub_stamp":"2026-02-24T18:02:11.527870+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-25T23:55:56.629108+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00028598099999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0],"acting":[9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":91,"reported_epoch":57,"state":"active+clean","last_fresh":"2026-02-24T18:02:09.518315+0000","last_change":"2026-02-24T18:02:09.518315+0000","last_active":"2026-02-24T18:02:09.518315+0000","last_peered":"2026-02-24T18:02:09.518315+0000","last_clean":"2026-02-24T18:02:09.518315+0000","last_became_active":"2026-02-24T18:01:59.520409+0000","last_became_peered":"2026-02-24T18:01:59.520409+0000","last_unstale":"2026-02-24T18:02:09.518315+0000","last_undegraded":"2026-02-24T18:02:09.518315+0000","last_fullsized":"2026-02-24T18:02:09.518315+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-24T18:02:09.518253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-24T18:02:09.518253+0000","last_clean_scrub_stamp":"2026-02-24T18:02:09.518253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-26T05:34:46.810244+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.3","version":"57'2","reported_seq":105,"reported_epoch":57,"state":"active+clean","last_fresh":"2026-02-24T18:02:09.999078+0000","last_change":"2026-02-24T18:02:09.999078+0000","last_active":"2026-02-24T18:02:09.999078+0000","last_peered":"2026-02-24T18:02:09.999078+0000","last_clean":"2026-02-24T18:02:09.999078+0000","last_became_active":"2026-02-24T18:01:59.382604+0000","last_became_peered":"2026-02-24T18:01:59.382604+0000","last_unstale":"2026-02-24T18:02:09.999078+0000","last_undegraded":"2026-02-24T18:02:09.999078+0000","last_fullsized":"2026-02-24T18:02:09.999078+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"57'2","last_scrub_stamp":"2026-02-24T18:02:09.999013+0000","last_deep_scrub":"57'2","last_deep_scrub_stamp":"2026-02-24T18:02:09.999013+0000","last_clean_scrub_stamp":"2026-02-24T18:02:09.999013+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2026-02-25T23:00:12.427534+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"57'3","reported_seq":113,"reported_epoch":57,"state":"active+clean","last_fresh":"2026-02-24T18:02:10.454030+0000","last_change":"2026-02-24T18:02:10.454030+0000","last_active":"2026-02-24T18:02:10.454030+0000","last_peered":"2026-02-24T18:02:10.454030+0000","last_clean":"2026-02-24T18:02:10.454030+0000","last_became_active":"2026-02-24T18:01:59.382500+0000","last_became_peered":"2026-02-24T18:01:59.382500+0000","last_unstale":"2026-02-24T18:02:10.454030+0000","last_undegraded":"2026-02-24T18:02:10.454030+0000","last_fullsized":"2026-02-24T18:02:10.454030+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"57'3","last_scrub_stamp":"2026-02-24T18:02:10.453982+0000","last_deep_scrub":"57'3","last_deep_scrub_stamp":"2026-02-24T18:02:10.453982+0000","last_clean_scrub_stamp":"2026-02-24T18:02:10.453982+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-26T04:26:13.975486+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00073412399999999995,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":85,"reported_epoch":57,"state":"active+clean","last_fresh":"2026-02-24T18:02:10.916428+0000","last_change":"2026-02-24T18:02:10.916428+0000","last_active":"2026-02-24T18:02:10.916428+0000","last_peered":"2026-02-24T18:02:10.916428+0000","last_clean":"2026-02-24T18:02:10.916428+0000","last_became_active":"2026-02-24T18:01:49.359167+0000","last_became_peered":"2026-02-24T18:01:49.359167+0000","last_unstale":"2026-02-24T18:02:10.916428+0000","last_undegraded":"2026-02-24T18:02:10.916428+0000","last_fullsized":"2026-02-24T18:02:10.916428+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-24T18:02:10.916353+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-24T18:02:10.916353+0000","last_clean_scrub_stamp":"2026-02-24T18:02:10.916353+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-25T18:20:31.391596+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00026963099999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":96,"reported_epoch":57,"state":"active+clean","last_fresh":"2026-02-24T18:02:09.937995+0000","last_change":"2026-02-24T18:02:09.937995+0000","last_active":"2026-02-24T18:02:09.937995+0000","last_peered":"2026-02-24T18:02:09.937995+0000","last_clean":"2026-02-24T18:02:09.937995+0000","last_became_active":"2026-02-24T18:01:49.358904+0000","last_became_peered":"2026-02-24T18:01:49.358904+0000","last_unstale":"2026-02-24T18:02:09.937995+0000","last_undegraded":"2026-02-24T18:02:09.937995+0000","last_fullsized":"2026-02-24T18:02:09.937995+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-24T18:02:09.937935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-24T18:02:09.937935+0000","last_clean_scrub_stamp":"2026-02-24T18:02:09.937935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-25T18:41:28.030416+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9],"acting":[5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":5,"ondisk_log_size":5,"up":16,"acting":16,"num_store_stats":9}],"osd_stats":[{"osd":9,"up_from":40,"seq":171798691862,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":27596,"kb_used_data":636,"kb_used_omap":25,"kb_used_meta":26918,"kb_avail":104830004,"statfs":{"total":107374182400,"available":107345924096,"internally_reserved":0,"allocated":651264,"data_stored":314176,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":25794,"internal_metadata":27564862},"hb_peers":[0,1,2,3,4,5,6,7,8],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":46,"seq":197568495638,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":104857600,"kb_used":27596,"kb_used_data":636,"kb_used_omap":28,"kb_used_meta":26915,"kb_avail":104830004,"statfs":{"total":107374182400,"available":107345924096,"internally_reserved":0,"allocated":651264,"data_stored":314176,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":29044,"internal_metadata":27561612},"hb_peers":[0,1,2,3,4,5,6,7,9],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":46,"seq":197568495638,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":27596,"kb_used_data":636,"kb_used_omap":28,"kb_used_meta":26915,"kb_avail":104830004,"statfs":{"total":107374182400,"available":107345924096,"internally_reserved":0,"allocated":651264,"data_stored":314176,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":29690,"internal_metadata":27560966},"hb_peers":[0,1,2,3,4,5,6,8,9],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":46,"seq":197568495638,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":27596,"kb_used_data":636,"kb_used_omap":27,"kb_used_meta":26916,"kb_avail":104830004,"statfs":{"total":107374182400,"available":107345924096,"internally_reserved":0,"allocated":651264,"data_stored":314176,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":27741,"internal_metadata":27562915},"hb_peers":[0,1,2,3,4,5,7,8,9],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":46,"seq":197568495638,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":27596,"kb_used_data":636,"kb_used_omap":28,"kb_used_meta":26915,"kb_avail":104830004,"statfs":{"total":107374182400,"available":107345924096,"internally_reserved":0,"allocated":651264,"data_stored":314176,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":29695,"internal_metadata":27560961},"hb_peers":[0,1,2,3,4,6,7,8,9],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":46,"seq":197568495638,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":27600,"kb_used_data":640,"kb_used_omap":29,"kb_used_meta":26914,"kb_avail":104830000,"statfs":{"total":107374182400,"available":107345920000,"internally_reserved":0,"allocated":655360,"data_stored":314195,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":30346,"internal_metadata":27560310},"hb_peers":[0,1,2,3,5,6,7,8,9],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":46,"seq":197568495638,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":27596,"kb_used_data":636,"kb_used_omap":27,"kb_used_meta":26916,"kb_avail":104830004,"statfs":{"total":107374182400,"available":107345924096,"internally_reserved":0,"allocated":651264,"data_stored":314176,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":27739,"internal_metadata":27562917},"hb_peers":[0,1,2,4,5,6,7,8,9],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":46,"seq":197568495638,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":104857600,"kb_used":27596,"kb_used_data":636,"kb_used_omap":22,"kb_used_meta":26921,"kb_avail":104830004,"statfs":{"total":107374182400,"available":107345924096,"internally_reserved":0,"allocated":651264,"data_stored":314176,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":22534,"internal_metadata":27568122},"hb_peers":[0,1,3,4,5,6,7,8,9],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":56,"seq":240518168598,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":27600,"kb_used_data":640,"kb_used_omap":28,"kb_used_meta":26915,"kb_avail":104830000,"statfs":{"total":107374182400,"available":107345920000,"internally_reserved":0,"allocated":655360,"data_stored":314195,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":29695,"internal_metadata":27560961},"hb_peers":[0,2,3,4,5,6,7,8,9],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":56,"seq":240518168599,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":27596,"kb_used_data":636,"kb_used_omap":13,"kb_used_meta":26930,"kb_avail":104830004,"statfs":{"total":107374182400,"available":107345924096,"internally_reserved":0,"allocated":651264,"data_stored":314176,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14071,"internal_metadata":27576585},"hb_peers":[1,2,3,4,5,6,7,8,9],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-24T18:02:31.214 DEBUG:teuthology.orchestra.run.trial068:> sudo ceph --cluster ceph config set global mon_health_to_clog false 2026-02-24T18:02:31.388 INFO:teuthology.misc:Shutting down mds daemons... 2026-02-24T18:02:31.388 INFO:teuthology.misc:Shutting down osd daemons... 2026-02-24T18:02:31.388 DEBUG:tasks.ceph.osd.0:waiting for process to exit 2026-02-24T18:02:31.388 INFO:teuthology.orchestra.run:waiting for 300 2026-02-24T18:02:31.474 INFO:tasks.ceph.osd.0:Stopped 2026-02-24T18:02:31.475 DEBUG:tasks.ceph.osd.1:waiting for process to exit 2026-02-24T18:02:31.475 INFO:teuthology.orchestra.run:waiting for 300 2026-02-24T18:02:31.531 INFO:tasks.ceph.osd.1:Stopped 2026-02-24T18:02:31.532 DEBUG:tasks.ceph.osd.2:waiting for process to exit 2026-02-24T18:02:31.532 INFO:teuthology.orchestra.run:waiting for 300 2026-02-24T18:02:31.623 INFO:tasks.ceph.osd.2:Stopped 2026-02-24T18:02:31.623 DEBUG:tasks.ceph.osd.3:waiting for process to exit 2026-02-24T18:02:31.624 INFO:teuthology.orchestra.run:waiting for 300 2026-02-24T18:02:31.694 INFO:tasks.ceph.osd.3:Stopped 2026-02-24T18:02:31.694 DEBUG:tasks.ceph.osd.4:waiting for process to exit 2026-02-24T18:02:31.694 INFO:teuthology.orchestra.run:waiting for 300 2026-02-24T18:02:31.769 INFO:tasks.ceph.osd.4:Stopped 2026-02-24T18:02:31.770 DEBUG:tasks.ceph.osd.5:waiting for process to exit 2026-02-24T18:02:31.770 INFO:teuthology.orchestra.run:waiting for 300 2026-02-24T18:02:35.401 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.5 is failed for ~0s 2026-02-24T18:02:37.871 INFO:tasks.ceph.osd.5:Stopped 2026-02-24T18:02:37.872 DEBUG:tasks.ceph.osd.6:waiting for process to exit 2026-02-24T18:02:37.872 INFO:teuthology.orchestra.run:waiting for 300 2026-02-24T18:02:37.940 INFO:tasks.ceph.osd.6:Stopped 2026-02-24T18:02:37.940 DEBUG:tasks.ceph.osd.7:waiting for process to exit 2026-02-24T18:02:37.940 INFO:teuthology.orchestra.run:waiting for 300 2026-02-24T18:02:38.013 INFO:tasks.ceph.osd.7:Stopped 2026-02-24T18:02:38.013 DEBUG:tasks.ceph.osd.8:waiting for process to exit 2026-02-24T18:02:38.013 INFO:teuthology.orchestra.run:waiting for 300 2026-02-24T18:02:38.073 INFO:tasks.ceph.osd.8:Stopped 2026-02-24T18:02:38.073 DEBUG:tasks.ceph.osd.9:waiting for process to exit 2026-02-24T18:02:38.073 INFO:teuthology.orchestra.run:waiting for 300 2026-02-24T18:02:40.602 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.9 is failed for ~0s 2026-02-24T18:02:40.602 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.5 has been restored 2026-02-24T18:02:43.494 INFO:tasks.ceph.mgr.x.trial068.stderr:2026-02-24T18:02:43.488+0000 7f0d175fe640 -1 calc_pg_upmaps abort due to max <= 0 2026-02-24T18:02:44.174 INFO:tasks.ceph.osd.9:Stopped 2026-02-24T18:02:44.174 INFO:teuthology.misc:Shutting down mgr daemons... 2026-02-24T18:02:44.174 DEBUG:tasks.ceph.mgr.x:waiting for process to exit 2026-02-24T18:02:44.174 INFO:teuthology.orchestra.run:waiting for 300 2026-02-24T18:02:44.215 INFO:tasks.ceph.mgr.x:Stopped 2026-02-24T18:02:44.215 INFO:teuthology.misc:Shutting down mon daemons... 2026-02-24T18:02:44.216 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2026-02-24T18:02:44.216 INFO:teuthology.orchestra.run:waiting for 300 2026-02-24T18:02:44.228 INFO:tasks.ceph.mon.a:Stopped 2026-02-24T18:02:44.228 INFO:tasks.ceph:Checking cluster log for badness... 2026-02-24T18:02:44.228 DEBUG:teuthology.orchestra.run.trial068:> sudo grep -E '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/ceph.log | grep -E -v 'but it is still running' | grep -E -v 'overall HEALTH_' | grep -E -v '\(OSDMAP_FLAGS\)' | grep -E -v '\(OSD_' | grep -E -v '\(PG_' | grep -E -v '\(POOL_APP_NOT_ENABLED\)' | grep -E -v '\(MDS_ALL_DOWN\)' | grep -E -v '\(MDS_UP_LESS_THAN_MAX\)' | head -n 1 2026-02-24T18:02:44.283 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-0 on ubuntu@trial068.front.sepia.ceph.com 2026-02-24T18:02:44.283 DEBUG:teuthology.orchestra.run.trial068:> sync && sudo umount -f /var/lib/ceph/osd/ceph-0 2026-02-24T18:02:44.338 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-1 on ubuntu@trial068.front.sepia.ceph.com 2026-02-24T18:02:44.338 DEBUG:teuthology.orchestra.run.trial068:> sync && sudo umount -f /var/lib/ceph/osd/ceph-1 2026-02-24T18:02:44.389 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-2 on ubuntu@trial068.front.sepia.ceph.com 2026-02-24T18:02:44.389 DEBUG:teuthology.orchestra.run.trial068:> sync && sudo umount -f /var/lib/ceph/osd/ceph-2 2026-02-24T18:02:44.440 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-3 on ubuntu@trial068.front.sepia.ceph.com 2026-02-24T18:02:44.440 DEBUG:teuthology.orchestra.run.trial068:> sync && sudo umount -f /var/lib/ceph/osd/ceph-3 2026-02-24T18:02:44.492 INFO:tasks.ceph:Archiving mon data... 2026-02-24T18:02:44.494 DEBUG:teuthology.misc:Transferring archived files from trial068:/var/lib/ceph/mon/ceph-a to /home/teuthworker/mnt/teuthology/yuriw-2026-02-24_16:03:54-rados-wip-yuri8-testing-2026-02-23-2128-distro-default-trial/68571/data/mon.a.tgz 2026-02-24T18:02:44.494 DEBUG:teuthology.orchestra.run.trial068:> mktemp 2026-02-24T18:02:44.542 INFO:teuthology.orchestra.run.trial068.stdout:/tmp/tmp.hmCR86u4ha 2026-02-24T18:02:44.542 DEBUG:teuthology.orchestra.run.trial068:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-a -- . > /tmp/tmp.hmCR86u4ha 2026-02-24T18:02:44.620 DEBUG:teuthology.orchestra.run.trial068:> sudo chmod 0666 /tmp/tmp.hmCR86u4ha 2026-02-24T18:02:44.718 DEBUG:teuthology.orchestra.remote:trial068:/tmp/tmp.hmCR86u4ha is 224KB 2026-02-24T18:02:44.771 DEBUG:teuthology.orchestra.run.trial068:> rm -fr /tmp/tmp.hmCR86u4ha 2026-02-24T18:02:44.774 INFO:tasks.ceph:Cleaning ceph cluster... 2026-02-24T18:02:44.774 DEBUG:teuthology.orchestra.run.trial068:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2026-02-24T18:02:44.869 INFO:teuthology.util.scanner:summary_data or yaml_file is empty! 2026-02-24T18:02:44.869 INFO:tasks.ceph:Archiving crash dumps... 2026-02-24T18:02:44.871 DEBUG:teuthology.misc:Transferring archived files from trial068:/var/lib/ceph/crash to /home/teuthworker/mnt/teuthology/yuriw-2026-02-24_16:03:54-rados-wip-yuri8-testing-2026-02-23-2128-distro-default-trial/68571/remote/trial068/crash 2026-02-24T18:02:44.872 DEBUG:teuthology.orchestra.run.trial068:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2026-02-24T18:02:44.917 INFO:tasks.ceph:Compressing logs... 2026-02-24T18:02:44.917 DEBUG:teuthology.orchestra.run.trial068:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2026-02-24T18:02:44.965 INFO:teuthology.orchestra.run.trial068.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.24670.log 2026-02-24T18:02:44.966 INFO:teuthology.orchestra.run.trial068.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.19290.log 2026-02-24T18:02:44.966 INFO:teuthology.orchestra.run.trial068.stderr:/var/log/ceph/ceph-client.admin.24670.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26711.log 2026-02-24T18:02:44.966 INFO:teuthology.orchestra.run.trial068.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.24670.log.gz 2026-02-24T18:02:44.966 INFO:teuthology.orchestra.run.trial068.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.19665.log 2026-02-24T18:02:44.966 INFO:teuthology.orchestra.run.trial068.stderr:/var/log/ceph/ceph-client.admin.19290.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.19290.log.gz 2026-02-24T18:02:44.967 INFO:teuthology.orchestra.run.trial068.stderr:/var/log/ceph/ceph-client.admin.26711.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.26711.log.gz 2026-02-24T18:02:44.967 INFO:teuthology.orchestra.run.trial068.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.24915.log 2026-02-24T18:02:44.967 INFO:teuthology.orchestra.run.trial068.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.25870.log 2026-02-24T18:02:44.967 INFO:teuthology.orchestra.run.trial068.stderr:/var/log/ceph/ceph-client.admin.19665.log: /var/log/ceph/ceph-client.admin.24915.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.19665.log.gz 2026-02-24T18:02:44.967 INFO:teuthology.orchestra.run.trial068.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.24915.log.gz 2026-02-24T18:02:44.967 INFO:teuthology.orchestra.run.trial068.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.24937.log 2026-02-24T18:02:44.968 INFO:teuthology.orchestra.run.trial068.stderr:/var/log/ceph/ceph-client.admin.25870.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27394.log 2026-02-24T18:02:44.968 INFO:teuthology.orchestra.run.trial068.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.25870.log.gz 2026-02-24T18:02:44.968 INFO:teuthology.orchestra.run.trial068.stderr:/var/log/ceph/ceph-client.admin.24937.log: gzip -5 --verbose -- /var/log/ceph/ceph.tmp-client.admin.11345.log 2026-02-24T18:02:44.968 INFO:teuthology.orchestra.run.trial068.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.24937.log.gz 2026-02-24T18:02:44.968 INFO:teuthology.orchestra.run.trial068.stderr:/var/log/ceph/ceph-client.admin.27394.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26822.log 2026-02-24T18:02:44.968 INFO:teuthology.orchestra.run.trial068.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.27394.log.gz 2026-02-24T18:02:44.969 INFO:teuthology.orchestra.run.trial068.stderr:/var/log/ceph/ceph.tmp-client.admin.11345.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27316.log 2026-02-24T18:02:44.969 INFO:teuthology.orchestra.run.trial068.stderr: 0.0% -- replaced with /var/log/ceph/ceph.tmp-client.admin.11345.log.gz 2026-02-24T18:02:44.969 INFO:teuthology.orchestra.run.trial068.stderr:/var/log/ceph/ceph-client.admin.26822.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.24968.log 2026-02-24T18:02:44.969 INFO:teuthology.orchestra.run.trial068.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.26822.log.gz 2026-02-24T18:02:44.969 INFO:teuthology.orchestra.run.trial068.stderr:/var/log/ceph/ceph-client.admin.27316.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27212.log 2026-02-24T18:02:44.969 INFO:teuthology.orchestra.run.trial068.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.27316.log.gz 2026-02-24T18:02:44.970 INFO:teuthology.orchestra.run.trial068.stderr:/var/log/ceph/ceph-client.admin.24968.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.25972.log 2026-02-24T18:02:44.970 INFO:teuthology.orchestra.run.trial068.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.24968.log.gz 2026-02-24T18:02:44.970 INFO:teuthology.orchestra.run.trial068.stderr:/var/log/ceph/ceph-client.admin.27212.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.24503.log 2026-02-24T18:02:44.970 INFO:teuthology.orchestra.run.trial068.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.27212.log.gz 2026-02-24T18:02:44.970 INFO:teuthology.orchestra.run.trial068.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26069.log 2026-02-24T18:02:44.970 INFO:teuthology.orchestra.run.trial068.stderr:/var/log/ceph/ceph-client.admin.25972.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.25972.log.gz 2026-02-24T18:02:44.971 INFO:teuthology.orchestra.run.trial068.stderr:/var/log/ceph/ceph-client.admin.24503.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27472.log 2026-02-24T18:02:44.971 INFO:teuthology.orchestra.run.trial068.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.24503.log.gz 2026-02-24T18:02:44.971 INFO:teuthology.orchestra.run.trial068.stderr:/var/log/ceph/ceph-client.admin.26069.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.26069.log.gz 2026-02-24T18:02:44.971 INFO:teuthology.orchestra.run.trial068.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.19472.log 2026-02-24T18:02:44.971 INFO:teuthology.orchestra.run.trial068.stderr:/var/log/ceph/ceph-client.admin.27472.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26268.log 2026-02-24T18:02:44.971 INFO:teuthology.orchestra.run.trial068.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.27472.log.gz 2026-02-24T18:02:44.971 INFO:teuthology.orchestra.run.trial068.stderr:/var/log/ceph/ceph-client.admin.19472.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.19472.log.gz 2026-02-24T18:02:44.971 INFO:teuthology.orchestra.run.trial068.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27004.log 2026-02-24T18:02:44.972 INFO:teuthology.orchestra.run.trial068.stderr:/var/log/ceph/ceph-client.admin.26268.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.26268.log.gz 2026-02-24T18:02:44.972 INFO:teuthology.orchestra.run.trial068.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26194.log 2026-02-24T18:02:44.972 INFO:teuthology.orchestra.run.trial068.stderr:/var/log/ceph/ceph-client.admin.27004.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26441.log 2026-02-24T18:02:44.972 INFO:teuthology.orchestra.run.trial068.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.27004.log.gz 2026-02-24T18:02:44.972 INFO:teuthology.orchestra.run.trial068.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.24681.log 2026-02-24T18:02:44.972 INFO:teuthology.orchestra.run.trial068.stderr:/var/log/ceph/ceph-client.admin.26194.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.26194.log.gz 2026-02-24T18:02:44.972 INFO:teuthology.orchestra.run.trial068.stderr:/var/log/ceph/ceph-client.admin.26441.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.26441.log.gz 2026-02-24T18:02:44.973 INFO:teuthology.orchestra.run.trial068.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27056.log 2026-02-24T18:02:44.973 INFO:teuthology.orchestra.run.trial068.stderr:/var/log/ceph/ceph-client.admin.24681.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.25222.log 2026-02-24T18:02:44.973 INFO:teuthology.orchestra.run.trial068.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.24681.log.gz 2026-02-24T18:02:44.973 INFO:teuthology.orchestra.run.trial068.stderr:/var/log/ceph/ceph-client.admin.27056.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26119.log 2026-02-24T18:02:44.973 INFO:teuthology.orchestra.run.trial068.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.27056.log.gz 2026-02-24T18:02:44.973 INFO:teuthology.orchestra.run.trial068.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.25894.log 2026-02-24T18:02:44.974 INFO:teuthology.orchestra.run.trial068.stderr:/var/log/ceph/ceph-client.admin.25222.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.25222.log.gz/var/log/ceph/ceph-client.admin.26119.log: 2026-02-24T18:02:44.974 INFO:teuthology.orchestra.run.trial068.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.26119.log.gz 2026-02-24T18:02:44.974 INFO:teuthology.orchestra.run.trial068.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.24940.log 2026-02-24T18:02:44.974 INFO:teuthology.orchestra.run.trial068.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.3.log 2026-02-24T18:02:44.974 INFO:teuthology.orchestra.run.trial068.stderr:/var/log/ceph/ceph-client.admin.25894.log: /var/log/ceph/ceph-client.admin.24940.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.25894.log.gz 2026-02-24T18:02:44.974 INFO:teuthology.orchestra.run.trial068.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.24940.log.gz 2026-02-24T18:02:44.974 INFO:teuthology.orchestra.run.trial068.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27082.log 2026-02-24T18:02:44.975 INFO:teuthology.orchestra.run.trial068.stderr:/var/log/ceph/ceph-osd.3.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.25946.log 2026-02-24T18:02:44.975 INFO:teuthology.orchestra.run.trial068.stderr:/var/log/ceph/ceph-client.admin.27082.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26342.log 2026-02-24T18:02:44.975 INFO:teuthology.orchestra.run.trial068.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.27082.log.gz 2026-02-24T18:02:44.975 INFO:teuthology.orchestra.run.trial068.stderr:/var/log/ceph/ceph-client.admin.25946.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26900.log 2026-02-24T18:02:44.975 INFO:teuthology.orchestra.run.trial068.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.25946.log.gz 2026-02-24T18:02:44.976 INFO:teuthology.orchestra.run.trial068.stderr:gzip -5/var/log/ceph/ceph-client.admin.26342.log: --verbose -- /var/log/ceph/ceph-client.admin.25458.log 2026-02-24T18:02:44.976 INFO:teuthology.orchestra.run.trial068.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.26342.log.gz 2026-02-24T18:02:44.976 INFO:teuthology.orchestra.run.trial068.stderr:/var/log/ceph/ceph-client.admin.26900.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27678.log 2026-02-24T18:02:44.976 INFO:teuthology.orchestra.run.trial068.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.26900.log.gz 2026-02-24T18:02:44.976 INFO:teuthology.orchestra.run.trial068.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.24478.log 2026-02-24T18:02:44.977 INFO:teuthology.orchestra.run.trial068.stderr:/var/log/ceph/ceph-client.admin.25458.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.25458.log.gz 2026-02-24T18:02:44.977 INFO:teuthology.orchestra.run.trial068.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.19354.log 2026-02-24T18:02:44.977 INFO:teuthology.orchestra.run.trial068.stderr:/var/log/ceph/ceph-client.admin.27678.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.27678.log.gz 2026-02-24T18:02:44.977 INFO:teuthology.orchestra.run.trial068.stderr:/var/log/ceph/ceph-client.admin.24478.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.24478.log.gz 2026-02-24T18:02:44.977 INFO:teuthology.orchestra.run.trial068.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.24528.log 2026-02-24T18:02:44.979 INFO:teuthology.orchestra.run.trial068.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27628.log 2026-02-24T18:02:44.979 INFO:teuthology.orchestra.run.trial068.stderr:/var/log/ceph/ceph-client.admin.19354.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.19354.log.gz 2026-02-24T18:02:44.979 INFO:teuthology.orchestra.run.trial068.stderr:/var/log/ceph/ceph-client.admin.24528.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26952.log 2026-02-24T18:02:44.979 INFO:teuthology.orchestra.run.trial068.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.24528.log.gz 2026-02-24T18:02:44.979 INFO:teuthology.orchestra.run.trial068.stderr:/var/log/ceph/ceph-client.admin.27628.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.27628.log.gz 2026-02-24T18:02:44.979 INFO:teuthology.orchestra.run.trial068.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.24688.log 2026-02-24T18:02:44.979 INFO:teuthology.orchestra.run.trial068.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.19593.log 2026-02-24T18:02:44.979 INFO:teuthology.orchestra.run.trial068.stderr:/var/log/ceph/ceph-client.admin.26952.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.26952.log.gz 2026-02-24T18:02:44.979 INFO:teuthology.orchestra.run.trial068.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.5.log 2026-02-24T18:02:44.979 INFO:teuthology.orchestra.run.trial068.stderr:/var/log/ceph/ceph-client.admin.24688.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.24688.log.gz 2026-02-24T18:02:44.980 INFO:teuthology.orchestra.run.trial068.stderr:/var/log/ceph/ceph-client.admin.19593.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26564.log 2026-02-24T18:02:44.980 INFO:teuthology.orchestra.run.trial068.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.19593.log.gz 2026-02-24T18:02:44.980 INFO:teuthology.orchestra.run.trial068.stderr:gzip/var/log/ceph/ceph-osd.5.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.24548.log 2026-02-24T18:02:44.980 INFO:teuthology.orchestra.run.trial068.stderr:/var/log/ceph/ceph-client.admin.26564.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.24574.log 2026-02-24T18:02:44.980 INFO:teuthology.orchestra.run.trial068.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.26564.log.gz 2026-02-24T18:02:44.980 INFO:teuthology.orchestra.run.trial068.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mgr.x.log 2026-02-24T18:02:44.980 INFO:teuthology.orchestra.run.trial068.stderr:/var/log/ceph/ceph-client.admin.24548.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.24548.log.gz 2026-02-24T18:02:44.981 INFO:teuthology.orchestra.run.trial068.stderr:/var/log/ceph/ceph-client.admin.24574.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.24574.log.gz 2026-02-24T18:02:44.981 INFO:teuthology.orchestra.run.trial068.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26516.log 2026-02-24T18:02:44.981 INFO:teuthology.orchestra.run.trial068.stderr:/var/log/ceph/ceph-mgr.x.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.9.log 2026-02-24T18:02:44.981 INFO:teuthology.orchestra.run.trial068.stderr:/var/log/ceph/ceph-client.admin.26516.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27238.log 2026-02-24T18:02:44.981 INFO:teuthology.orchestra.run.trial068.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.26516.log.gz 2026-02-24T18:02:44.981 INFO:teuthology.orchestra.run.trial068.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.6.log 2026-02-24T18:02:44.981 INFO:teuthology.orchestra.run.trial068.stderr:/var/log/ceph/ceph-osd.9.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26243.log 2026-02-24T18:02:44.981 INFO:teuthology.orchestra.run.trial068.stderr:/var/log/ceph/ceph-client.admin.27238.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.27238.log.gz 2026-02-24T18:02:44.982 INFO:teuthology.orchestra.run.trial068.stderr:/var/log/ceph/ceph-osd.6.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27108.log 2026-02-24T18:02:44.982 INFO:teuthology.orchestra.run.trial068.stderr:/var/log/ceph/ceph-client.admin.26243.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.26243.log.gz 2026-02-24T18:02:44.982 INFO:teuthology.orchestra.run.trial068.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.25172.log 2026-02-24T18:02:44.982 INFO:teuthology.orchestra.run.trial068.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.25997.log 2026-02-24T18:02:44.982 INFO:teuthology.orchestra.run.trial068.stderr:/var/log/ceph/ceph-client.admin.27108.log: /var/log/ceph/ceph-client.admin.25172.log: gzip -5 --verbose -- 0.0% /var/log/ceph/ceph-client.admin.24679.log 2026-02-24T18:02:44.982 INFO:teuthology.orchestra.run.trial068.stderr: -- replaced with /var/log/ceph/ceph-client.admin.27108.log.gz 2026-02-24T18:02:44.983 INFO:teuthology.orchestra.run.trial068.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.25172.log.gz 2026-02-24T18:02:44.983 INFO:teuthology.orchestra.run.trial068.stderr:/var/log/ceph/ceph-client.admin.25997.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.25997.log.gz 2026-02-24T18:02:44.983 INFO:teuthology.orchestra.run.trial068.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26848.log 2026-02-24T18:02:44.983 INFO:teuthology.orchestra.run.trial068.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26636.log 2026-02-24T18:02:44.983 INFO:teuthology.orchestra.run.trial068.stderr:/var/log/ceph/ceph-client.admin.24679.log: /var/log/ceph/ceph-client.admin.26848.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.24935.log 2026-02-24T18:02:44.983 INFO:teuthology.orchestra.run.trial068.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.24679.log.gz 2026-02-24T18:02:44.983 INFO:teuthology.orchestra.run.trial068.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.26848.log.gz 2026-02-24T18:02:44.984 INFO:teuthology.orchestra.run.trial068.stderr:/var/log/ceph/ceph-client.admin.26636.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.26636.log.gz 2026-02-24T18:02:44.984 INFO:teuthology.orchestra.run.trial068.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.24684.log 2026-02-24T18:02:44.984 INFO:teuthology.orchestra.run.trial068.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mon.a.log 2026-02-24T18:02:44.984 INFO:teuthology.orchestra.run.trial068.stderr:/var/log/ceph/ceph-client.admin.24935.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.24935.log.gz 2026-02-24T18:02:44.984 INFO:teuthology.orchestra.run.trial068.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.25795.log 2026-02-24T18:02:44.984 INFO:teuthology.orchestra.run.trial068.stderr:/var/log/ceph/ceph-client.admin.24684.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.24684.log.gz 2026-02-24T18:02:44.985 INFO:teuthology.orchestra.run.trial068.stderr:/var/log/ceph/ceph-mon.a.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26293.log 2026-02-24T18:02:44.985 INFO:teuthology.orchestra.run.trial068.stderr:/var/log/ceph/ceph-client.admin.25795.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.25229.log 2026-02-24T18:02:44.985 INFO:teuthology.orchestra.run.trial068.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.25795.log.gz 2026-02-24T18:02:44.985 INFO:teuthology.orchestra.run.trial068.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.8.log 2026-02-24T18:02:44.985 INFO:teuthology.orchestra.run.trial068.stderr:/var/log/ceph/ceph-client.admin.26293.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.26293.log.gz 2026-02-24T18:02:44.986 INFO:teuthology.orchestra.run.trial068.stderr:/var/log/ceph/ceph-client.admin.25229.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27342.log 2026-02-24T18:02:44.986 INFO:teuthology.orchestra.run.trial068.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.25229.log.gz 2026-02-24T18:02:44.986 INFO:teuthology.orchestra.run.trial068.stderr:/var/log/ceph/ceph-osd.8.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.24949.log 2026-02-24T18:02:44.986 INFO:teuthology.orchestra.run.trial068.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.19641.log 2026-02-24T18:02:44.986 INFO:teuthology.orchestra.run.trial068.stderr:/var/log/ceph/ceph-client.admin.27342.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.27342.log.gz 2026-02-24T18:02:44.986 INFO:teuthology.orchestra.run.trial068.stderr:/var/log/ceph/ceph-client.admin.24949.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27550.log 2026-02-24T18:02:44.986 INFO:teuthology.orchestra.run.trial068.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.24949.log.gz 2026-02-24T18:02:44.987 INFO:teuthology.orchestra.run.trial068.stderr:/var/log/ceph/ceph-client.admin.19641.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.25228.log 2026-02-24T18:02:44.987 INFO:teuthology.orchestra.run.trial068.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.19641.log.gz 2026-02-24T18:02:44.987 INFO:teuthology.orchestra.run.trial068.stderr:/var/log/ceph/ceph-client.admin.27550.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.25920.log 2026-02-24T18:02:44.987 INFO:teuthology.orchestra.run.trial068.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.27550.log.gz 2026-02-24T18:02:44.987 INFO:teuthology.orchestra.run.trial068.stderr:/var/log/ceph/ceph-client.admin.25228.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.25228.log.gz 2026-02-24T18:02:44.987 INFO:teuthology.orchestra.run.trial068.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27186.log 2026-02-24T18:02:44.988 INFO:teuthology.orchestra.run.trial068.stderr:/var/log/ceph/ceph-client.admin.25920.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26466.log 2026-02-24T18:02:44.988 INFO:teuthology.orchestra.run.trial068.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.25920.log.gz 2026-02-24T18:02:44.988 INFO:teuthology.orchestra.run.trial068.stderr:/var/log/ceph/ceph-client.admin.27186.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.25674.log 2026-02-24T18:02:44.988 INFO:teuthology.orchestra.run.trial068.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.27186.log.gz 2026-02-24T18:02:44.988 INFO:teuthology.orchestra.run.trial068.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26686.log 2026-02-24T18:02:44.988 INFO:teuthology.orchestra.run.trial068.stderr:/var/log/ceph/ceph-client.admin.26466.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.26466.log.gz 2026-02-24T18:02:44.989 INFO:teuthology.orchestra.run.trial068.stderr:/var/log/ceph/ceph-client.admin.25674.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26978.log 2026-02-24T18:02:44.989 INFO:teuthology.orchestra.run.trial068.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.25674.log.gz 2026-02-24T18:02:44.989 INFO:teuthology.orchestra.run.trial068.stderr:/var/log/ceph/ceph-client.admin.26686.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27030.log 2026-02-24T18:02:44.989 INFO:teuthology.orchestra.run.trial068.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.26686.log.gz 2026-02-24T18:02:44.989 INFO:teuthology.orchestra.run.trial068.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.19545.log 2026-02-24T18:02:44.989 INFO:teuthology.orchestra.run.trial068.stderr:/var/log/ceph/ceph-client.admin.26978.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.26978.log.gz 2026-02-24T18:02:44.989 INFO:teuthology.orchestra.run.trial068.stderr:/var/log/ceph/ceph-client.admin.27030.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26588.log 2026-02-24T18:02:44.990 INFO:teuthology.orchestra.run.trial068.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.27030.log.gz 2026-02-24T18:02:44.990 INFO:teuthology.orchestra.run.trial068.stderr:/var/log/ceph/ceph-client.admin.19545.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26736.log 2026-02-24T18:02:44.990 INFO:teuthology.orchestra.run.trial068.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.19545.log.gz 2026-02-24T18:02:44.990 INFO:teuthology.orchestra.run.trial068.stderr:/var/log/ceph/ceph-client.admin.26588.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26540.log 2026-02-24T18:02:44.990 INFO:teuthology.orchestra.run.trial068.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.26588.log.gz 2026-02-24T18:02:44.990 INFO:teuthology.orchestra.run.trial068.stderr:/var/log/ceph/ceph-client.admin.26736.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.25224.log 2026-02-24T18:02:44.990 INFO:teuthology.orchestra.run.trial068.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.26736.log.gz 2026-02-24T18:02:44.991 INFO:teuthology.orchestra.run.trial068.stderr:/var/log/ceph/ceph-client.admin.26540.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26318.log 2026-02-24T18:02:44.991 INFO:teuthology.orchestra.run.trial068.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.26540.log.gz 2026-02-24T18:02:44.991 INFO:teuthology.orchestra.run.trial068.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.7.log 2026-02-24T18:02:44.991 INFO:teuthology.orchestra.run.trial068.stderr:/var/log/ceph/ceph-client.admin.25224.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.25224.log.gz 2026-02-24T18:02:44.991 INFO:teuthology.orchestra.run.trial068.stderr:/var/log/ceph/ceph-client.admin.26318.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.25598.log 2026-02-24T18:02:44.991 INFO:teuthology.orchestra.run.trial068.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.26318.log.gz 2026-02-24T18:02:44.992 INFO:teuthology.orchestra.run.trial068.stderr:/var/log/ceph/ceph-osd.7.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.24950.log 2026-02-24T18:02:44.992 INFO:teuthology.orchestra.run.trial068.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27368.log 2026-02-24T18:02:44.992 INFO:teuthology.orchestra.run.trial068.stderr:/var/log/ceph/ceph-client.admin.25598.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.25598.log.gz 2026-02-24T18:02:44.992 INFO:teuthology.orchestra.run.trial068.stderr:/var/log/ceph/ceph-client.admin.24950.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27524.log 2026-02-24T18:02:44.992 INFO:teuthology.orchestra.run.trial068.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.24950.log.gz 2026-02-24T18:02:44.993 INFO:teuthology.orchestra.run.trial068.stderr:/var/log/ceph/ceph-client.admin.27368.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.19721.log 2026-02-24T18:02:44.993 INFO:teuthology.orchestra.run.trial068.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.27368.log.gz 2026-02-24T18:02:44.993 INFO:teuthology.orchestra.run.trial068.stderr:/var/log/ceph/ceph-client.admin.27524.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26612.log 2026-02-24T18:02:44.993 INFO:teuthology.orchestra.run.trial068.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.27524.log.gz 2026-02-24T18:02:44.993 INFO:teuthology.orchestra.run.trial068.stderr:/var/log/ceph/ceph-client.admin.19721.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27576.log 2026-02-24T18:02:44.993 INFO:teuthology.orchestra.run.trial068.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.19721.log.gz 2026-02-24T18:02:44.994 INFO:teuthology.orchestra.run.trial068.stderr:/var/log/ceph/ceph-client.admin.26612.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.25698.log 2026-02-24T18:02:44.994 INFO:teuthology.orchestra.run.trial068.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.26612.log.gz 2026-02-24T18:02:44.994 INFO:teuthology.orchestra.run.trial068.stderr:/var/log/ceph/ceph-client.admin.27576.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.27576.log.gz 2026-02-24T18:02:44.994 INFO:teuthology.orchestra.run.trial068.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.24677.log 2026-02-24T18:02:44.995 INFO:teuthology.orchestra.run.trial068.stderr:gzip -5 --verbose --/var/log/ceph/ceph-client.admin.25698.log: /var/log/ceph/ceph-client.admin.25570.log 2026-02-24T18:02:44.995 INFO:teuthology.orchestra.run.trial068.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.25698.log.gz 2026-02-24T18:02:44.995 INFO:teuthology.orchestra.run.trial068.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.25770.log 2026-02-24T18:02:44.995 INFO:teuthology.orchestra.run.trial068.stderr:/var/log/ceph/ceph-client.admin.24677.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.24677.log.gz 2026-02-24T18:02:44.995 INFO:teuthology.orchestra.run.trial068.stderr:/var/log/ceph/ceph-client.admin.25570.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.25570.log.gz 2026-02-24T18:02:44.995 INFO:teuthology.orchestra.run.trial068.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.19617.log 2026-02-24T18:02:44.996 INFO:teuthology.orchestra.run.trial068.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.25820.log 2026-02-24T18:02:44.996 INFO:teuthology.orchestra.run.trial068.stderr:/var/log/ceph/ceph-client.admin.25770.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.25770.log.gz 2026-02-24T18:02:44.996 INFO:teuthology.orchestra.run.trial068.stderr:/var/log/ceph/ceph-client.admin.19617.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.19617.log.gz 2026-02-24T18:02:44.996 INFO:teuthology.orchestra.run.trial068.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.19448.log 2026-02-24T18:02:44.996 INFO:teuthology.orchestra.run.trial068.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26021.log 2026-02-24T18:02:44.996 INFO:teuthology.orchestra.run.trial068.stderr:/var/log/ceph/ceph-client.admin.25820.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.25820.log.gz 2026-02-24T18:02:44.997 INFO:teuthology.orchestra.run.trial068.stderr:/var/log/ceph/ceph-client.admin.19448.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.19448.log.gz 2026-02-24T18:02:44.997 INFO:teuthology.orchestra.run.trial068.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2026-02-24T18:02:44.997 INFO:teuthology.orchestra.run.trial068.stderr:/var/log/ceph/ceph-client.admin.26021.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.2.log 2026-02-24T18:02:44.997 INFO:teuthology.orchestra.run.trial068.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.26021.log.gz 2026-02-24T18:02:44.997 INFO:teuthology.orchestra.run.trial068.stderr:/var/log/ceph/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27602.log 2026-02-24T18:02:44.997 INFO:teuthology.orchestra.run.trial068.stderr:/var/log/ceph/ceph-osd.2.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26874.log 2026-02-24T18:02:44.997 INFO:teuthology.orchestra.run.trial068.stderr: 89.7% -- replaced with /var/log/ceph/ceph.audit.log.gz 2026-02-24T18:02:44.997 INFO:teuthology.orchestra.run.trial068.stderr:/var/log/ceph/ceph-client.admin.27602.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26416.log 2026-02-24T18:02:44.998 INFO:teuthology.orchestra.run.trial068.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.27602.log.gz 2026-02-24T18:02:44.998 INFO:teuthology.orchestra.run.trial068.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.log 2026-02-24T18:02:44.998 INFO:teuthology.orchestra.run.trial068.stderr:/var/log/ceph/ceph-client.admin.26874.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.26874.log.gz 2026-02-24T18:02:44.998 INFO:teuthology.orchestra.run.trial068.stderr:/var/log/ceph/ceph-client.admin.26416.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.26416.log.gz 2026-02-24T18:02:44.998 INFO:teuthology.orchestra.run.trial068.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26926.log 2026-02-24T18:02:44.998 INFO:teuthology.orchestra.run.trial068.stderr:/var/log/ceph/ceph.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27290.log 2026-02-24T18:02:44.999 INFO:teuthology.orchestra.run.trial068.stderr:/var/log/ceph/ceph-client.admin.26926.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.26926.log.gz 2026-02-24T18:02:44.999 INFO:teuthology.orchestra.run.trial068.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27498.log 2026-02-24T18:02:44.999 INFO:teuthology.orchestra.run.trial068.stderr:/var/log/ceph/ceph-client.admin.27290.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.25845.log 2026-02-24T18:02:44.999 INFO:teuthology.orchestra.run.trial068.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.27290.log.gz 2026-02-24T18:02:44.999 INFO:teuthology.orchestra.run.trial068.stderr:/var/log/ceph/ceph-client.admin.27498.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.27498.log.gz 2026-02-24T18:02:44.999 INFO:teuthology.orchestra.run.trial068.stderr: 86.7% -- replaced with /var/log/ceph/ceph.log.gz 2026-02-24T18:02:44.999 INFO:teuthology.orchestra.run.trial068.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.25227.log 2026-02-24T18:02:45.000 INFO:teuthology.orchestra.run.trial068.stderr:/var/log/ceph/ceph-client.admin.25845.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.4.log 2026-02-24T18:02:45.000 INFO:teuthology.orchestra.run.trial068.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.25845.log.gz 2026-02-24T18:02:45.000 INFO:teuthology.orchestra.run.trial068.stderr:/var/log/ceph/ceph-client.admin.25227.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26144.log 2026-02-24T18:02:45.000 INFO:teuthology.orchestra.run.trial068.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.25227.log.gz 2026-02-24T18:02:45.000 INFO:teuthology.orchestra.run.trial068.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.24626.log 2026-02-24T18:02:45.000 INFO:teuthology.orchestra.run.trial068.stderr:/var/log/ceph/ceph-osd.4.log: /var/log/ceph/ceph-client.admin.26144.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.26144.log.gz 2026-02-24T18:02:45.001 INFO:teuthology.orchestra.run.trial068.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.25746.log 2026-02-24T18:02:45.001 INFO:teuthology.orchestra.run.trial068.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.25622.log 2026-02-24T18:02:45.001 INFO:teuthology.orchestra.run.trial068.stderr:/var/log/ceph/ceph-client.admin.24626.log: /var/log/ceph/ceph-client.admin.25746.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.24626.log.gz 2026-02-24T18:02:45.001 INFO:teuthology.orchestra.run.trial068.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.25746.log.gz 2026-02-24T18:02:45.001 INFO:teuthology.orchestra.run.trial068.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26045.log 2026-02-24T18:02:45.001 INFO:teuthology.orchestra.run.trial068.stderr:/var/log/ceph/ceph-client.admin.25622.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.25622.log.gz 2026-02-24T18:02:45.001 INFO:teuthology.orchestra.run.trial068.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27134.log 2026-02-24T18:02:45.001 INFO:teuthology.orchestra.run.trial068.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.25226.log 2026-02-24T18:02:45.002 INFO:teuthology.orchestra.run.trial068.stderr:/var/log/ceph/ceph-client.admin.26045.log: /var/log/ceph/ceph-client.admin.27134.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.26045.log.gz 2026-02-24T18:02:45.002 INFO:teuthology.orchestra.run.trial068.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.27134.log.gz 2026-02-24T18:02:45.002 INFO:teuthology.orchestra.run.trial068.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.24600.log 2026-02-24T18:02:45.002 INFO:teuthology.orchestra.run.trial068.stderr:/var/log/ceph/ceph-client.admin.25226.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.25226.log.gz 2026-02-24T18:02:45.002 INFO:teuthology.orchestra.run.trial068.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27160.log 2026-02-24T18:02:45.003 INFO:teuthology.orchestra.run.trial068.stderr: 91.6% -- replaced with /var/log/ceph/ceph-mgr.x.log.gz 2026-02-24T18:02:45.003 INFO:teuthology.orchestra.run.trial068.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26094.log 2026-02-24T18:02:45.003 INFO:teuthology.orchestra.run.trial068.stderr:/var/log/ceph/ceph-client.admin.24600.log: /var/log/ceph/ceph-client.admin.27160.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.24600.log.gz 2026-02-24T18:02:45.003 INFO:teuthology.orchestra.run.trial068.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.27160.log.gz 2026-02-24T18:02:45.003 INFO:teuthology.orchestra.run.trial068.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.24687.log 2026-02-24T18:02:45.003 INFO:teuthology.orchestra.run.trial068.stderr:/var/log/ceph/ceph-client.admin.26094.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.20333.log 2026-02-24T18:02:45.003 INFO:teuthology.orchestra.run.trial068.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.26094.log.gz 2026-02-24T18:02:45.004 INFO:teuthology.orchestra.run.trial068.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.24916.log/var/log/ceph/ceph-client.admin.24687.log: 2026-02-24T18:02:45.004 INFO:teuthology.orchestra.run.trial068.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.24687.log.gz 2026-02-24T18:02:45.004 INFO:teuthology.orchestra.run.trial068.stderr:/var/log/ceph/ceph-client.admin.20333.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.20333.log.gz 2026-02-24T18:02:45.004 INFO:teuthology.orchestra.run.trial068.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27446.log 2026-02-24T18:02:45.004 INFO:teuthology.orchestra.run.trial068.stderr:/var/log/ceph/ceph-client.admin.24916.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.24686.log 2026-02-24T18:02:45.004 INFO:teuthology.orchestra.run.trial068.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.24916.log.gz 2026-02-24T18:02:45.005 INFO:teuthology.orchestra.run.trial068.stderr:/var/log/ceph/ceph-client.admin.27446.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.27446.log.gz 2026-02-24T18:02:45.005 INFO:teuthology.orchestra.run.trial068.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26391.log 2026-02-24T18:02:45.005 INFO:teuthology.orchestra.run.trial068.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.19404.log 2026-02-24T18:02:45.005 INFO:teuthology.orchestra.run.trial068.stderr:/var/log/ceph/ceph-client.admin.24686.log: /var/log/ceph/ceph-client.admin.26391.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.24686.log.gz 0.0% 2026-02-24T18:02:45.005 INFO:teuthology.orchestra.run.trial068.stderr: -- replaced with /var/log/ceph/ceph-client.admin.26391.log.gz 2026-02-24T18:02:45.005 INFO:teuthology.orchestra.run.trial068.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.24952.log 2026-02-24T18:02:45.005 INFO:teuthology.orchestra.run.trial068.stderr: 92.2% -- replaced with /var/log/ceph/ceph-osd.5.log.gz 2026-02-24T18:02:45.006 INFO:teuthology.orchestra.run.trial068.stderr:/var/log/ceph/ceph-client.admin.19404.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27264.log 2026-02-24T18:02:45.006 INFO:teuthology.orchestra.run.trial068.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.19404.log.gz 2026-02-24T18:02:45.006 INFO:teuthology.orchestra.run.trial068.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26491.log 2026-02-24T18:02:45.006 INFO:teuthology.orchestra.run.trial068.stderr:/var/log/ceph/ceph-client.admin.24952.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.24952.log.gz 2026-02-24T18:02:45.006 INFO:teuthology.orchestra.run.trial068.stderr:/var/log/ceph/ceph-client.admin.27264.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.27264.log.gz 2026-02-24T18:02:45.006 INFO:teuthology.orchestra.run.trial068.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.0.log 2026-02-24T18:02:45.007 INFO:teuthology.orchestra.run.trial068.stderr:/var/log/ceph/ceph-client.admin.26491.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.19497.log 2026-02-24T18:02:45.007 INFO:teuthology.orchestra.run.trial068.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.26491.log.gz 2026-02-24T18:02:45.007 INFO:teuthology.orchestra.run.trial068.stderr:/var/log/ceph/ceph-osd.0.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.24685.log 2026-02-24T18:02:45.007 INFO:teuthology.orchestra.run.trial068.stderr:/var/log/ceph/ceph-client.admin.19497.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.19497.log.gz 2026-02-24T18:02:45.007 INFO:teuthology.orchestra.run.trial068.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27654.log 2026-02-24T18:02:45.007 INFO:teuthology.orchestra.run.trial068.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.19521.log 2026-02-24T18:02:45.008 INFO:teuthology.orchestra.run.trial068.stderr:/var/log/ceph/ceph-client.admin.24685.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.24685.log.gz 2026-02-24T18:02:45.008 INFO:teuthology.orchestra.run.trial068.stderr:/var/log/ceph/ceph-client.admin.27654.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27420.log 2026-02-24T18:02:45.008 INFO:teuthology.orchestra.run.trial068.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.27654.log.gz 2026-02-24T18:02:45.008 INFO:teuthology.orchestra.run.trial068.stderr:/var/log/ceph/ceph-client.admin.19521.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.25220.log 2026-02-24T18:02:45.008 INFO:teuthology.orchestra.run.trial068.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.19521.log.gz 2026-02-24T18:02:45.009 INFO:teuthology.orchestra.run.trial068.stderr:/var/log/ceph/ceph-client.admin.27420.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.27420.log.gz 2026-02-24T18:02:45.009 INFO:teuthology.orchestra.run.trial068.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.25219.log 2026-02-24T18:02:45.009 INFO:teuthology.orchestra.run.trial068.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26761.log 2026-02-24T18:02:45.009 INFO:teuthology.orchestra.run.trial068.stderr:/var/log/ceph/ceph-client.admin.25220.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.25220.log.gz 2026-02-24T18:02:45.009 INFO:teuthology.orchestra.run.trial068.stderr:/var/log/ceph/ceph-client.admin.25219.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.25219.log.gz 2026-02-24T18:02:45.009 INFO:teuthology.orchestra.run.trial068.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.25484.log 2026-02-24T18:02:45.010 INFO:teuthology.orchestra.run.trial068.stderr:/var/log/ceph/ceph-client.admin.26761.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.1.log 2026-02-24T18:02:45.010 INFO:teuthology.orchestra.run.trial068.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.26761.log.gz 2026-02-24T18:02:45.010 INFO:teuthology.orchestra.run.trial068.stderr:/var/log/ceph/ceph-client.admin.25484.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26169.log 2026-02-24T18:02:45.010 INFO:teuthology.orchestra.run.trial068.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.25484.log.gz 2026-02-24T18:02:45.010 INFO:teuthology.orchestra.run.trial068.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.25432.log 2026-02-24T18:02:45.010 INFO:teuthology.orchestra.run.trial068.stderr:/var/log/ceph/ceph-osd.1.log: /var/log/ceph/ceph-client.admin.26169.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.26169.log.gz 2026-02-24T18:02:45.011 INFO:teuthology.orchestra.run.trial068.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26219.log 2026-02-24T18:02:45.011 INFO:teuthology.orchestra.run.trial068.stderr:/var/log/ceph/ceph-client.admin.25432.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.24675.log 2026-02-24T18:02:45.011 INFO:teuthology.orchestra.run.trial068.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.25432.log.gz 2026-02-24T18:02:45.011 INFO:teuthology.orchestra.run.trial068.stderr:/var/log/ceph/ceph-client.admin.26219.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26661.log 2026-02-24T18:02:45.011 INFO:teuthology.orchestra.run.trial068.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.26219.log.gz 2026-02-24T18:02:45.012 INFO:teuthology.orchestra.run.trial068.stderr:/var/log/ceph/ceph-client.admin.24675.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.24675.log.gz 2026-02-24T18:02:45.012 INFO:teuthology.orchestra.run.trial068.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26366.log 2026-02-24T18:02:45.012 INFO:teuthology.orchestra.run.trial068.stderr:/var/log/ceph/ceph-client.admin.26661.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.19569.log 2026-02-24T18:02:45.012 INFO:teuthology.orchestra.run.trial068.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.26661.log.gz 2026-02-24T18:02:45.012 INFO:teuthology.orchestra.run.trial068.stderr:/var/log/ceph/ceph-client.admin.26366.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.26366.log.gzgzip 2026-02-24T18:02:45.012 INFO:teuthology.orchestra.run.trial068.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.24939.log 2026-02-24T18:02:45.013 INFO:teuthology.orchestra.run.trial068.stderr:/var/log/ceph/ceph-client.admin.19569.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.19569.log.gz 2026-02-24T18:02:45.013 INFO:teuthology.orchestra.run.trial068.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.25176.log 2026-02-24T18:02:45.013 INFO:teuthology.orchestra.run.trial068.stderr:/var/log/ceph/ceph-client.admin.24939.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.24939.log.gz 2026-02-24T18:02:45.013 INFO:teuthology.orchestra.run.trial068.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.25722.log 2026-02-24T18:02:45.013 INFO:teuthology.orchestra.run.trial068.stderr:/var/log/ceph/ceph-client.admin.25176.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26796.log 2026-02-24T18:02:45.013 INFO:teuthology.orchestra.run.trial068.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.25176.log.gz 2026-02-24T18:02:45.014 INFO:teuthology.orchestra.run.trial068.stderr: 92.4% -- replaced with /var/log/ceph/ceph-osd.8.log.gz 2026-02-24T18:02:45.014 INFO:teuthology.orchestra.run.trial068.stderr:/var/log/ceph/ceph-client.admin.25722.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.25722.log.gz 2026-02-24T18:02:45.014 INFO:teuthology.orchestra.run.trial068.stderr:/var/log/ceph/ceph-client.admin.26796.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.26796.log.gz 2026-02-24T18:02:45.014 INFO:teuthology.orchestra.run.trial068.stderr: 92.3% -- replaced with /var/log/ceph/ceph-osd.3.log.gz 2026-02-24T18:02:45.019 INFO:teuthology.orchestra.run.trial068.stderr: 92.1% -- replaced with /var/log/ceph/ceph-osd.6.log.gz 2026-02-24T18:02:45.020 INFO:teuthology.orchestra.run.trial068.stderr: 92.2% -- replaced with /var/log/ceph/ceph-osd.9.log.gz 2026-02-24T18:02:45.023 INFO:teuthology.orchestra.run.trial068.stderr: 92.5% -- replaced with /var/log/ceph/ceph-osd.2.log.gz 2026-02-24T18:02:45.024 INFO:teuthology.orchestra.run.trial068.stderr: 92.2% -- replaced with /var/log/ceph/ceph-osd.7.log.gz 2026-02-24T18:02:45.034 INFO:teuthology.orchestra.run.trial068.stderr: 92.2% -- replaced with /var/log/ceph/ceph-osd.0.log.gz 2026-02-24T18:02:45.039 INFO:teuthology.orchestra.run.trial068.stderr: 92.3% -- replaced with /var/log/ceph/ceph-osd.4.log.gz 2026-02-24T18:02:45.047 INFO:teuthology.orchestra.run.trial068.stderr: 92.3% -- replaced with /var/log/ceph/ceph-osd.1.log.gz 2026-02-24T18:02:45.070 INFO:teuthology.orchestra.run.trial068.stderr: 88.8% -- replaced with /var/log/ceph/ceph-mon.a.log.gz 2026-02-24T18:02:45.071 INFO:teuthology.orchestra.run.trial068.stderr: 2026-02-24T18:02:45.071 INFO:teuthology.orchestra.run.trial068.stderr:real 0m0.112s 2026-02-24T18:02:45.072 INFO:teuthology.orchestra.run.trial068.stderr:user 0m0.516s 2026-02-24T18:02:45.072 INFO:teuthology.orchestra.run.trial068.stderr:sys 0m0.061s 2026-02-24T18:02:45.072 INFO:tasks.ceph:Archiving logs... 2026-02-24T18:02:45.073 DEBUG:teuthology.misc:Transferring archived files from trial068:/var/log/ceph to /home/teuthworker/mnt/teuthology/yuriw-2026-02-24_16:03:54-rados-wip-yuri8-testing-2026-02-23-2128-distro-default-trial/68571/remote/trial068/log 2026-02-24T18:02:45.074 DEBUG:teuthology.orchestra.run.trial068:> sudo tar c -f - -C /var/log/ceph -- . 2026-02-24T18:02:45.253 DEBUG:teuthology.run_tasks:Unwinding manager install 2026-02-24T18:02:45.265 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2026-02-24T18:02:45.265 DEBUG:teuthology.orchestra.run.trial068:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2026-02-24T18:02:45.353 DEBUG:teuthology.run_tasks:Unwinding manager clock 2026-02-24T18:02:45.363 INFO:teuthology.task.clock:Checking final clock skew... 2026-02-24T18:02:45.364 DEBUG:teuthology.orchestra.run.trial068:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2026-02-24T18:02:45.550 INFO:teuthology.orchestra.run.trial068.stdout: remote refid st t when poll reach delay offset jitter 2026-02-24T18:02:45.550 INFO:teuthology.orchestra.run.trial068.stdout:============================================================================== 2026-02-24T18:02:45.550 INFO:teuthology.orchestra.run.trial068.stdout: ntp.maxhost.io 132.163.97.4 2 u 131 64 16 88.329 -7.465 5.290 2026-02-24T18:02:45.550 INFO:teuthology.orchestra.run.trial068.stdout:*sushi.ruselabs. 2.148.140.233 2 u 59 64 17 73.426 +2.997 1.875 2026-02-24T18:02:45.550 INFO:teuthology.orchestra.run.trial068.stdout:+static-72-87-88 173.162.192.156 2 u 67 64 17 14.594 +0.555 2.472 2026-02-24T18:02:45.550 INFO:teuthology.orchestra.run.trial068.stdout:+23.186.168.129 132.163.97.4 2 u 1 64 37 69.954 +2.496 6.026 2026-02-24T18:02:45.551 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2026-02-24T18:02:45.562 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2026-02-24T18:02:45.563 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2026-02-24T18:02:45.576 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2026-02-24T18:02:45.594 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2026-02-24T18:02:45.617 INFO:teuthology.task.internal:Duration was 412.344622 seconds 2026-02-24T18:02:45.617 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2026-02-24T18:02:45.635 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2026-02-24T18:02:45.635 DEBUG:teuthology.orchestra.run.trial068:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2026-02-24T18:02:45.635 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.9 has been restored 2026-02-24T18:02:45.665 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2026-02-24T18:02:45.665 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@trial068.front.sepia.ceph.com 2026-02-24T18:02:45.666 DEBUG:teuthology.orchestra.run.trial068:> grep -E --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | grep -E -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | grep -E -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2026-02-24T18:02:45.720 INFO:teuthology.task.internal.syslog:Gathering journactl... 2026-02-24T18:02:45.720 DEBUG:teuthology.orchestra.run.trial068:> sudo journalctl > /home/ubuntu/cephtest/archive/syslog/journalctl.log 2026-02-24T18:02:45.991 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2026-02-24T18:02:45.991 DEBUG:teuthology.orchestra.run.trial068:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --max-args=1 --max-procs=0 --verbose --no-run-if-empty -- gzip -5 --verbose -- 2026-02-24T18:02:45.999 INFO:teuthology.orchestra.run.trial068.stderr:gzip -5 --verbose -- /home/ubuntu/cephtest/archive/syslog/misc.log 2026-02-24T18:02:46.001 INFO:teuthology.orchestra.run.trial068.stderr:gzip -5 --verbose -- /home/ubuntu/cephtest/archive/syslog/journalctl.log 2026-02-24T18:02:46.001 INFO:teuthology.orchestra.run.trial068.stderr:/home/ubuntu/cephtest/archive/syslog/misc.log: 0.0% -- replaced with /home/ubuntu/cephtest/archive/syslog/misc.log.gz 2026-02-24T18:02:46.001 INFO:teuthology.orchestra.run.trial068.stderr:gzip -5 --verbose -- /home/ubuntu/cephtest/archive/syslog/kern.log 2026-02-24T18:02:46.001 INFO:teuthology.orchestra.run.trial068.stderr:/home/ubuntu/cephtest/archive/syslog/journalctl.log: /home/ubuntu/cephtest/archive/syslog/kern.log: 0.0% -- replaced with /home/ubuntu/cephtest/archive/syslog/kern.log.gz 2026-02-24T18:02:46.069 INFO:teuthology.orchestra.run.trial068.stderr: 81.5% -- replaced with /home/ubuntu/cephtest/archive/syslog/journalctl.log.gz 2026-02-24T18:02:46.070 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2026-02-24T18:02:46.082 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2026-02-24T18:02:46.083 DEBUG:teuthology.orchestra.run.trial068:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2026-02-24T18:02:46.121 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2026-02-24T18:02:46.138 DEBUG:teuthology.orchestra.run.trial068:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2026-02-24T18:02:46.170 INFO:teuthology.orchestra.run.trial068.stdout:kernel.core_pattern = core 2026-02-24T18:02:46.179 DEBUG:teuthology.orchestra.run.trial068:> test -e /home/ubuntu/cephtest/archive/coredump 2026-02-24T18:02:46.228 DEBUG:teuthology.orchestra.run:got remote process result: 1 2026-02-24T18:02:46.228 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2026-02-24T18:02:46.240 INFO:teuthology.task.internal:Transferring archived files... 2026-02-24T18:02:46.240 DEBUG:teuthology.misc:Transferring archived files from trial068:/home/ubuntu/cephtest/archive to /home/teuthworker/mnt/teuthology/yuriw-2026-02-24_16:03:54-rados-wip-yuri8-testing-2026-02-23-2128-distro-default-trial/68571/remote/trial068 2026-02-24T18:02:46.240 DEBUG:teuthology.orchestra.run.trial068:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2026-02-24T18:02:46.291 INFO:teuthology.task.internal:Removing archive directory... 2026-02-24T18:02:46.291 DEBUG:teuthology.orchestra.run.trial068:> rm -rf -- /home/ubuntu/cephtest/archive 2026-02-24T18:02:46.325 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2026-02-24T18:02:46.337 INFO:teuthology.task.internal:Not uploading archives. 2026-02-24T18:02:46.337 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2026-02-24T18:02:46.350 INFO:teuthology.task.internal:Tidying up after the test... 2026-02-24T18:02:46.350 DEBUG:teuthology.orchestra.run.trial068:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2026-02-24T18:02:46.370 INFO:teuthology.orchestra.run.trial068.stdout: 1311292 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Feb 24 18:02 /home/ubuntu/cephtest 2026-02-24T18:02:46.371 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2026-02-24T18:02:46.383 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2026-02-24T18:02:46.416 INFO:teuthology.run:Summary data: description: rados/singleton-nomsgr/{all/health-warnings mon_election/connectivity rados supported-random-distro$/{ubuntu_latest}} duration: 412.3446216583252 flavor: default owner: scheduled_yuriw@soko04.front.sepia.ceph.com success: true 2026-02-24T18:02:46.416 DEBUG:teuthology.report:Pushing job info to https://paddles-paddles.apps.pok.os.sepia.ceph.com 2026-02-24T18:02:46.509 INFO:teuthology.run:pass