2026-02-25T11:54:48.509 INFO:root:teuthology version: 1.2.4.dev9+gfa17720d0 2026-02-25T11:54:48.525 DEBUG:teuthology.report:Pushing job info to https://paddles-paddles.apps.pok.os.sepia.ceph.com 2026-02-25T11:54:48.602 INFO:teuthology.run:Config: archive_path: /home/teuthworker/mnt/teuthology/nmordech-2026-02-25_11:36:23-rados-wip-rocky10-branch-of-the-day-2026-02-24-1771941190-distro-default-trial/69872 branch: wip-rocky10-branch-of-the-day-2026-02-24-1771941190 description: rados/singleton-nomsgr/{all/health-warnings mon_election/connectivity rados supported-random-distro$/{ubuntu_latest}} email: null first_in_suite: false flavor: default job_id: '69872' kernel: branch: distro kdb: 1 sha1: distro ktype: distro last_in_suite: false machine_type: trial name: nmordech-2026-02-25_11:36:23-rados-wip-rocky10-branch-of-the-day-2026-02-24-1771941190-distro-default-trial no_nested_subset: false os_type: ubuntu os_version: '22.04' overrides: admin_socket: branch: wip-rocky10-branch-of-the-day-2026-02-24-1771941190 ceph: conf: global: mon election default strategy: 3 mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 30 debug ms: 1 debug paxos: 20 mon scrub interval: 300 osd: bluestore zero block detection: true debug ms: 1 debug osd: 20 osd debug verify cached snaps: true osd debug verify missing on start: true osd mclock override recovery settings: true osd mclock profile: high_recovery_ops osd mclock skip benchmark: true osd op queue: debug_random osd op queue cut off: debug_random flavor: default log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) sha1: 506b402a3d3a240d0bfe073b4769eaffdf84497b ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} cephadm: distro-suffix: rockylinux-10 install: ceph: flavor: default sha1: 506b402a3d3a240d0bfe073b4769eaffdf84497b workunit: branch: wip-rocky10-branch-of-the-day-2026-02-24-1771941190 sha1: 506b402a3d3a240d0bfe073b4769eaffdf84497b owner: scheduled_nmordech@soko04.front.sepia.ceph.com priority: 10 repo: https://git.ceph.com/ceph-ci.git roles: - - mon.a - mgr.x - osd.0 - osd.1 - osd.2 - osd.3 - osd.4 - osd.5 - osd.6 - osd.7 - osd.8 - osd.9 - client.0 seed: 317 sha1: 506b402a3d3a240d0bfe073b4769eaffdf84497b sleep_before_teardown: 0 subset: 111/120000 suite: rados suite_branch: wip-rocky10-branch-of-the-day-2026-02-24-1771941190 suite_path: /home/teuthworker/src/git.ceph.com_ceph-c_506b402a3d3a240d0bfe073b4769eaffdf84497b/qa suite_relpath: qa suite_repo: https://git.ceph.com/ceph-ci.git suite_sha1: 506b402a3d3a240d0bfe073b4769eaffdf84497b targets: trial117.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBJFfl9t2An17wPEskxXrqDLtD3lvUT6PBH8Egb5x1vc3SHIZuO4Xga9Z9XZIdDLcmbBOFUwlEWs56vvPpXmd+3E= tasks: - install: null - ceph: conf: osd: osd max object name len: 400 osd max object namespace len: 64 log-ignorelist: - but it is still running - overall HEALTH_ - \(OSDMAP_FLAGS\) - \(OSD_ - \(PG_ - \(POOL_APP_NOT_ENABLED\) pre-mgr-commands: - sudo ceph config set mgr mgr_pool false --force - workunit: clients: all: - rados/test_health_warnings.sh teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: fa17720d0088c3ac28e473468bfc79eeaff5cd38 timestamp: 2026-02-25_11:36:23 tube: trial user: nmordech verbose: false worker_log: /home/teuthworker/mnt/teuthology/worker_logs/dispatcher.trial.904699 2026-02-25T11:54:48.602 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph-c_506b402a3d3a240d0bfe073b4769eaffdf84497b/qa; will attempt to use it 2026-02-25T11:54:48.602 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph-c_506b402a3d3a240d0bfe073b4769eaffdf84497b/qa/tasks 2026-02-25T11:54:48.602 INFO:teuthology.run_tasks:Running task internal.check_packages... 2026-02-25T11:54:48.604 INFO:teuthology.task.internal:Checking packages... 2026-02-25T11:54:48.604 INFO:teuthology.task.internal:Checking packages for os_type 'ubuntu', flavor 'default' and ceph hash '506b402a3d3a240d0bfe073b4769eaffdf84497b' 2026-02-25T11:54:48.604 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2026-02-25T11:54:48.604 INFO:teuthology.packaging:ref: None 2026-02-25T11:54:48.604 INFO:teuthology.packaging:tag: None 2026-02-25T11:54:48.604 INFO:teuthology.packaging:branch: wip-rocky10-branch-of-the-day-2026-02-24-1771941190 2026-02-25T11:54:48.604 INFO:teuthology.packaging:sha1: 506b402a3d3a240d0bfe073b4769eaffdf84497b 2026-02-25T11:54:48.604 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&ref=wip-rocky10-branch-of-the-day-2026-02-24-1771941190 2026-02-25T11:54:48.725 INFO:teuthology.task.internal:Found packages for ceph version 20.3.0-5580-g506b402a-1jammy 2026-02-25T11:54:48.726 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2026-02-25T11:54:48.741 INFO:teuthology.task.internal:no buildpackages task found 2026-02-25T11:54:48.741 INFO:teuthology.run_tasks:Running task internal.save_config... 2026-02-25T11:54:48.751 INFO:teuthology.task.internal:Saving configuration 2026-02-25T11:54:48.761 INFO:teuthology.run_tasks:Running task internal.check_lock... 2026-02-25T11:54:48.773 INFO:teuthology.task.internal.check_lock:Checking locks... 2026-02-25T11:54:48.847 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'trial117.front.sepia.ceph.com', 'description': '/home/teuthworker/mnt/teuthology/nmordech-2026-02-25_11:36:23-rados-wip-rocky10-branch-of-the-day-2026-02-24-1771941190-distro-default-trial/69872', 'up': True, 'machine_type': 'trial', 'is_vm': False, 'vm_host': None, 'os_type': 'ubuntu', 'os_version': '22.04', 'arch': 'x86_64', 'locked': True, 'locked_since': '2026-02-25 11:51:19.866687', 'locked_by': 'scheduled_nmordech@soko04.front.sepia.ceph.com', 'mac_address': None, 'ssh_pub_key': 'ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAICc6kJQjbefHkSgTamAhqdrFQjhDLudxSoLpOPlpm730'} 2026-02-25T11:54:48.847 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2026-02-25T11:54:48.861 INFO:teuthology.task.internal:roles: ubuntu@trial117.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'osd.8', 'osd.9', 'client.0'] 2026-02-25T11:54:48.861 INFO:teuthology.run_tasks:Running task console_log... 2026-02-25T11:54:48.943 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x766ace8151c0>, signals=[15]) 2026-02-25T11:54:48.943 INFO:teuthology.run_tasks:Running task internal.connect... 2026-02-25T11:54:48.954 INFO:teuthology.task.internal:Opening connections... 2026-02-25T11:54:48.954 DEBUG:teuthology.task.internal:connecting to ubuntu@trial117.front.sepia.ceph.com 2026-02-25T11:54:48.955 DEBUG:teuthology.orchestra.connection:{'hostname': 'trial117.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2026-02-25T11:54:49.020 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2026-02-25T11:54:49.037 DEBUG:teuthology.orchestra.run.trial117:> uname -m 2026-02-25T11:54:49.098 INFO:teuthology.orchestra.run.trial117.stdout:x86_64 2026-02-25T11:54:49.098 DEBUG:teuthology.orchestra.run.trial117:> cat /etc/os-release 2026-02-25T11:54:49.146 INFO:teuthology.orchestra.run.trial117.stdout:PRETTY_NAME="Ubuntu 22.04.5 LTS" 2026-02-25T11:54:49.146 INFO:teuthology.orchestra.run.trial117.stdout:NAME="Ubuntu" 2026-02-25T11:54:49.146 INFO:teuthology.orchestra.run.trial117.stdout:VERSION_ID="22.04" 2026-02-25T11:54:49.146 INFO:teuthology.orchestra.run.trial117.stdout:VERSION="22.04.5 LTS (Jammy Jellyfish)" 2026-02-25T11:54:49.146 INFO:teuthology.orchestra.run.trial117.stdout:VERSION_CODENAME=jammy 2026-02-25T11:54:49.146 INFO:teuthology.orchestra.run.trial117.stdout:ID=ubuntu 2026-02-25T11:54:49.146 INFO:teuthology.orchestra.run.trial117.stdout:ID_LIKE=debian 2026-02-25T11:54:49.147 INFO:teuthology.orchestra.run.trial117.stdout:HOME_URL="https://www.ubuntu.com/" 2026-02-25T11:54:49.147 INFO:teuthology.orchestra.run.trial117.stdout:SUPPORT_URL="https://help.ubuntu.com/" 2026-02-25T11:54:49.147 INFO:teuthology.orchestra.run.trial117.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" 2026-02-25T11:54:49.147 INFO:teuthology.orchestra.run.trial117.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" 2026-02-25T11:54:49.147 INFO:teuthology.orchestra.run.trial117.stdout:UBUNTU_CODENAME=jammy 2026-02-25T11:54:49.147 INFO:teuthology.lock.ops:Updating trial117.front.sepia.ceph.com on lock server 2026-02-25T11:54:49.218 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2026-02-25T11:54:49.232 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2026-02-25T11:54:49.246 INFO:teuthology.task.internal:Checking for old test directory... 2026-02-25T11:54:49.246 DEBUG:teuthology.orchestra.run.trial117:> test '!' -e /home/ubuntu/cephtest 2026-02-25T11:54:49.249 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2026-02-25T11:54:49.261 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2026-02-25T11:54:49.261 DEBUG:teuthology.orchestra.run.trial117:> test -z $(ls -A /var/lib/ceph) 2026-02-25T11:54:49.293 INFO:teuthology.orchestra.run.trial117.stderr:ls: cannot access '/var/lib/ceph': No such file or directory 2026-02-25T11:54:49.294 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2026-02-25T11:54:49.387 INFO:teuthology.run_tasks:Running task kernel... 2026-02-25T11:54:49.407 INFO:teuthology.task.kernel:normalize config orig: {'branch': 'distro', 'kdb': 1, 'sha1': 'distro'} 2026-02-25T11:54:49.407 INFO:teuthology.task.kernel:config {'mon.a': {'branch': 'distro', 'kdb': 1, 'sha1': 'distro'}}, timeout 300 2026-02-25T11:54:49.407 DEBUG:teuthology.orchestra.run.trial117:> test -f /run/.containerenv -o -f /.dockerenv 2026-02-25T11:54:49.410 DEBUG:teuthology.orchestra.run:got remote process result: 1 2026-02-25T11:54:49.410 DEBUG:teuthology.orchestra.run.trial117:> uname -r 2026-02-25T11:54:49.458 INFO:teuthology.orchestra.run.trial117.stdout:5.15.0-170-generic 2026-02-25T11:54:49.458 INFO:teuthology.task.kernel:Running kernel on trial117: 5.15.0-170-generic 2026-02-25T11:54:49.458 DEBUG:teuthology.orchestra.run.trial117:> sudo apt-get clean 2026-02-25T11:54:49.516 DEBUG:teuthology.orchestra.run.trial117:> sudo apt-get update 2026-02-25T11:54:50.011 INFO:teuthology.orchestra.run.trial117.stdout:Hit:1 https://archive.ubuntu.com/ubuntu jammy InRelease 2026-02-25T11:54:50.100 INFO:teuthology.orchestra.run.trial117.stdout:Get:2 https://archive.ubuntu.com/ubuntu jammy-updates InRelease [128 kB] 2026-02-25T11:54:50.437 INFO:teuthology.orchestra.run.trial117.stdout:Get:3 https://archive.ubuntu.com/ubuntu jammy-security InRelease [129 kB] 2026-02-25T11:54:50.585 INFO:teuthology.orchestra.run.trial117.stdout:Get:4 https://archive.ubuntu.com/ubuntu jammy-backports InRelease [127 kB] 2026-02-25T11:54:50.725 INFO:teuthology.orchestra.run.trial117.stdout:Get:5 https://archive.ubuntu.com/ubuntu jammy-updates/main amd64 Packages [3275 kB] 2026-02-25T11:54:51.124 INFO:teuthology.orchestra.run.trial117.stdout:Get:6 https://archive.ubuntu.com/ubuntu jammy-updates/main Translation-en [498 kB] 2026-02-25T11:54:51.137 INFO:teuthology.orchestra.run.trial117.stdout:Get:7 https://archive.ubuntu.com/ubuntu jammy-updates/main amd64 c-n-f Metadata [19.2 kB] 2026-02-25T11:54:51.137 INFO:teuthology.orchestra.run.trial117.stdout:Get:8 https://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 Packages [1256 kB] 2026-02-25T11:54:51.172 INFO:teuthology.orchestra.run.trial117.stdout:Get:9 https://archive.ubuntu.com/ubuntu jammy-updates/universe Translation-en [314 kB] 2026-02-25T11:54:51.178 INFO:teuthology.orchestra.run.trial117.stdout:Get:10 https://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 c-n-f Metadata [30.3 kB] 2026-02-25T11:54:51.179 INFO:teuthology.orchestra.run.trial117.stdout:Get:11 https://archive.ubuntu.com/ubuntu jammy-updates/restricted amd64 Packages [5400 kB] 2026-02-25T11:54:51.338 INFO:teuthology.orchestra.run.trial117.stdout:Get:12 https://archive.ubuntu.com/ubuntu jammy-updates/restricted Translation-en [1025 kB] 2026-02-25T11:54:51.359 INFO:teuthology.orchestra.run.trial117.stdout:Get:13 https://archive.ubuntu.com/ubuntu jammy-security/main amd64 Packages [3007 kB] 2026-02-25T11:54:51.447 INFO:teuthology.orchestra.run.trial117.stdout:Get:14 https://archive.ubuntu.com/ubuntu jammy-security/main Translation-en [430 kB] 2026-02-25T11:54:51.456 INFO:teuthology.orchestra.run.trial117.stdout:Get:15 https://archive.ubuntu.com/ubuntu jammy-security/main amd64 c-n-f Metadata [14.1 kB] 2026-02-25T11:54:51.456 INFO:teuthology.orchestra.run.trial117.stdout:Get:16 https://archive.ubuntu.com/ubuntu jammy-security/universe amd64 Packages [1018 kB] 2026-02-25T11:54:51.490 INFO:teuthology.orchestra.run.trial117.stdout:Get:17 https://archive.ubuntu.com/ubuntu jammy-security/universe Translation-en [225 kB] 2026-02-25T11:54:51.499 INFO:teuthology.orchestra.run.trial117.stdout:Get:18 https://archive.ubuntu.com/ubuntu jammy-security/universe amd64 c-n-f Metadata [22.7 kB] 2026-02-25T11:54:51.499 INFO:teuthology.orchestra.run.trial117.stdout:Get:19 https://archive.ubuntu.com/ubuntu jammy-security/restricted amd64 Packages [5217 kB] 2026-02-25T11:54:51.646 INFO:teuthology.orchestra.run.trial117.stdout:Get:20 https://archive.ubuntu.com/ubuntu jammy-security/restricted Translation-en [993 kB] 2026-02-25T11:54:55.101 INFO:teuthology.orchestra.run.trial117.stdout:Fetched 23.1 MB in 3s (7934 kB/s) 2026-02-25T11:54:55.625 INFO:teuthology.orchestra.run.trial117.stdout:Reading package lists... 2026-02-25T11:54:55.638 DEBUG:teuthology.orchestra.run.trial117:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2026-02-25T11:54:55.670 INFO:teuthology.orchestra.run.trial117.stdout:Reading package lists... 2026-02-25T11:54:55.777 INFO:teuthology.orchestra.run.trial117.stdout:Building dependency tree... 2026-02-25T11:54:55.778 INFO:teuthology.orchestra.run.trial117.stdout:Reading state information... 2026-02-25T11:54:55.863 INFO:teuthology.orchestra.run.trial117.stdout:The following packages were automatically installed and are no longer required: 2026-02-25T11:54:55.863 INFO:teuthology.orchestra.run.trial117.stdout: kpartx libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 2026-02-25T11:54:55.863 INFO:teuthology.orchestra.run.trial117.stdout: libgfxdr0 libglusterfs0 libiscsi7 libpmemobj1 libsgutils2-2 sg3-utils 2026-02-25T11:54:55.863 INFO:teuthology.orchestra.run.trial117.stdout: sg3-utils-udev 2026-02-25T11:54:55.863 INFO:teuthology.orchestra.run.trial117.stdout:Use 'sudo apt autoremove' to remove them. 2026-02-25T11:54:55.864 INFO:teuthology.orchestra.run.trial117.stdout:The following additional packages will be installed: 2026-02-25T11:54:55.864 INFO:teuthology.orchestra.run.trial117.stdout: linux-generic linux-headers-5.15.0-171 linux-headers-5.15.0-171-generic 2026-02-25T11:54:55.864 INFO:teuthology.orchestra.run.trial117.stdout: linux-headers-generic linux-image-5.15.0-171-generic 2026-02-25T11:54:55.864 INFO:teuthology.orchestra.run.trial117.stdout: linux-modules-5.15.0-171-generic linux-modules-extra-5.15.0-171-generic 2026-02-25T11:54:55.864 INFO:teuthology.orchestra.run.trial117.stdout:Suggested packages: 2026-02-25T11:54:55.864 INFO:teuthology.orchestra.run.trial117.stdout: fdutils linux-doc | linux-source-5.15.0 linux-tools 2026-02-25T11:54:55.877 INFO:teuthology.orchestra.run.trial117.stdout:The following NEW packages will be installed: 2026-02-25T11:54:55.877 INFO:teuthology.orchestra.run.trial117.stdout: linux-headers-5.15.0-171 linux-headers-5.15.0-171-generic 2026-02-25T11:54:55.877 INFO:teuthology.orchestra.run.trial117.stdout: linux-image-5.15.0-171-generic linux-modules-5.15.0-171-generic 2026-02-25T11:54:55.877 INFO:teuthology.orchestra.run.trial117.stdout: linux-modules-extra-5.15.0-171-generic 2026-02-25T11:54:55.877 INFO:teuthology.orchestra.run.trial117.stdout:The following packages will be upgraded: 2026-02-25T11:54:55.877 INFO:teuthology.orchestra.run.trial117.stdout: linux-generic linux-headers-generic linux-image-generic 2026-02-25T11:54:56.293 INFO:teuthology.orchestra.run.trial117.stdout:3 upgraded, 5 newly installed, 0 to remove and 29 not upgraded. 2026-02-25T11:54:56.293 INFO:teuthology.orchestra.run.trial117.stdout:Need to get 113 MB of archives. 2026-02-25T11:54:56.294 INFO:teuthology.orchestra.run.trial117.stdout:After this operation, 585 MB of additional disk space will be used. 2026-02-25T11:54:56.294 INFO:teuthology.orchestra.run.trial117.stdout:Get:1 https://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-modules-5.15.0-171-generic amd64 5.15.0-171.181 [22.7 MB] 2026-02-25T11:54:57.741 INFO:teuthology.orchestra.run.trial117.stdout:Get:2 https://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-image-5.15.0-171-generic amd64 5.15.0-171.181 [11.6 MB] 2026-02-25T11:54:58.140 INFO:teuthology.orchestra.run.trial117.stdout:Get:3 https://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-modules-extra-5.15.0-171-generic amd64 5.15.0-171.181 [63.9 MB] 2026-02-25T11:55:00.049 INFO:teuthology.orchestra.run.trial117.stdout:Get:4 https://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-generic amd64 5.15.0.171.160 [1658 B] 2026-02-25T11:55:00.049 INFO:teuthology.orchestra.run.trial117.stdout:Get:5 https://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-image-generic amd64 5.15.0.171.160 [2486 B] 2026-02-25T11:55:00.049 INFO:teuthology.orchestra.run.trial117.stdout:Get:6 https://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-headers-5.15.0-171 all 5.15.0-171.181 [12.4 MB] 2026-02-25T11:55:00.393 INFO:teuthology.orchestra.run.trial117.stdout:Get:7 https://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-headers-5.15.0-171-generic amd64 5.15.0-171.181 [2849 kB] 2026-02-25T11:55:00.472 INFO:teuthology.orchestra.run.trial117.stdout:Get:8 https://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-headers-generic amd64 5.15.0.171.160 [2332 B] 2026-02-25T11:55:00.671 INFO:teuthology.orchestra.run.trial117.stdout:Fetched 113 MB in 5s (24.7 MB/s) 2026-02-25T11:55:00.691 INFO:teuthology.orchestra.run.trial117.stdout:Selecting previously unselected package linux-modules-5.15.0-171-generic. 2026-02-25T11:55:00.861 INFO:teuthology.orchestra.run.trial117.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 167442 files and directories currently installed.) 2026-02-25T11:55:00.863 INFO:teuthology.orchestra.run.trial117.stdout:Preparing to unpack .../0-linux-modules-5.15.0-171-generic_5.15.0-171.181_amd64.deb ... 2026-02-25T11:55:00.864 INFO:teuthology.orchestra.run.trial117.stdout:Unpacking linux-modules-5.15.0-171-generic (5.15.0-171.181) ... 2026-02-25T11:55:01.212 INFO:teuthology.orchestra.run.trial117.stdout:Selecting previously unselected package linux-image-5.15.0-171-generic. 2026-02-25T11:55:01.221 INFO:teuthology.orchestra.run.trial117.stdout:Preparing to unpack .../1-linux-image-5.15.0-171-generic_5.15.0-171.181_amd64.deb ... 2026-02-25T11:55:01.229 INFO:teuthology.orchestra.run.trial117.stdout:Unpacking linux-image-5.15.0-171-generic (5.15.0-171.181) ... 2026-02-25T11:55:01.270 INFO:teuthology.orchestra.run.trial117.stdout:Selecting previously unselected package linux-modules-extra-5.15.0-171-generic. 2026-02-25T11:55:01.279 INFO:teuthology.orchestra.run.trial117.stdout:Preparing to unpack .../2-linux-modules-extra-5.15.0-171-generic_5.15.0-171.181_amd64.deb ... 2026-02-25T11:55:01.279 INFO:teuthology.orchestra.run.trial117.stdout:Unpacking linux-modules-extra-5.15.0-171-generic (5.15.0-171.181) ... 2026-02-25T11:55:02.394 INFO:teuthology.orchestra.run.trial117.stdout:Preparing to unpack .../3-linux-generic_5.15.0.171.160_amd64.deb ... 2026-02-25T11:55:02.395 INFO:teuthology.orchestra.run.trial117.stdout:Unpacking linux-generic (5.15.0.171.160) over (5.15.0.170.159) ... 2026-02-25T11:55:02.420 INFO:teuthology.orchestra.run.trial117.stdout:Preparing to unpack .../4-linux-image-generic_5.15.0.171.160_amd64.deb ... 2026-02-25T11:55:02.421 INFO:teuthology.orchestra.run.trial117.stdout:Unpacking linux-image-generic (5.15.0.171.160) over (5.15.0.170.159) ... 2026-02-25T11:55:02.439 INFO:teuthology.orchestra.run.trial117.stdout:Selecting previously unselected package linux-headers-5.15.0-171. 2026-02-25T11:55:02.449 INFO:teuthology.orchestra.run.trial117.stdout:Preparing to unpack .../5-linux-headers-5.15.0-171_5.15.0-171.181_all.deb ... 2026-02-25T11:55:02.449 INFO:teuthology.orchestra.run.trial117.stdout:Unpacking linux-headers-5.15.0-171 (5.15.0-171.181) ... 2026-02-25T11:55:03.823 INFO:teuthology.orchestra.run.trial117.stdout:Selecting previously unselected package linux-headers-5.15.0-171-generic. 2026-02-25T11:55:03.837 INFO:teuthology.orchestra.run.trial117.stdout:Preparing to unpack .../6-linux-headers-5.15.0-171-generic_5.15.0-171.181_amd64.deb ... 2026-02-25T11:55:03.838 INFO:teuthology.orchestra.run.trial117.stdout:Unpacking linux-headers-5.15.0-171-generic (5.15.0-171.181) ... 2026-02-25T11:55:04.428 INFO:teuthology.orchestra.run.trial117.stdout:Preparing to unpack .../7-linux-headers-generic_5.15.0.171.160_amd64.deb ... 2026-02-25T11:55:04.428 INFO:teuthology.orchestra.run.trial117.stdout:Unpacking linux-headers-generic (5.15.0.171.160) over (5.15.0.170.159) ... 2026-02-25T11:55:04.455 INFO:teuthology.orchestra.run.trial117.stdout:Setting up linux-headers-5.15.0-171 (5.15.0-171.181) ... 2026-02-25T11:55:04.456 INFO:teuthology.orchestra.run.trial117.stdout:Setting up linux-headers-5.15.0-171-generic (5.15.0-171.181) ... 2026-02-25T11:55:04.457 INFO:teuthology.orchestra.run.trial117.stdout:Setting up linux-headers-generic (5.15.0.171.160) ... 2026-02-25T11:55:04.459 INFO:teuthology.orchestra.run.trial117.stdout:Setting up linux-image-5.15.0-171-generic (5.15.0-171.181) ... 2026-02-25T11:55:05.244 INFO:teuthology.orchestra.run.trial117.stdout:I: /boot/vmlinuz.old is now a symlink to vmlinuz-5.15.0-170-generic 2026-02-25T11:55:05.245 INFO:teuthology.orchestra.run.trial117.stdout:I: /boot/initrd.img.old is now a symlink to initrd.img-5.15.0-170-generic 2026-02-25T11:55:05.245 INFO:teuthology.orchestra.run.trial117.stdout:I: /boot/vmlinuz is now a symlink to vmlinuz-5.15.0-171-generic 2026-02-25T11:55:05.245 INFO:teuthology.orchestra.run.trial117.stdout:I: /boot/initrd.img is now a symlink to initrd.img-5.15.0-171-generic 2026-02-25T11:55:05.252 INFO:teuthology.orchestra.run.trial117.stdout:Setting up linux-modules-extra-5.15.0-171-generic (5.15.0-171.181) ... 2026-02-25T11:55:06.062 INFO:teuthology.orchestra.run.trial117.stdout:Setting up linux-modules-5.15.0-171-generic (5.15.0-171.181) ... 2026-02-25T11:55:06.866 INFO:teuthology.orchestra.run.trial117.stdout:Setting up linux-image-generic (5.15.0.171.160) ... 2026-02-25T11:55:06.867 INFO:teuthology.orchestra.run.trial117.stdout:Setting up linux-generic (5.15.0.171.160) ... 2026-02-25T11:55:06.869 INFO:teuthology.orchestra.run.trial117.stdout:Processing triggers for linux-image-5.15.0-171-generic (5.15.0-171.181) ... 2026-02-25T11:55:06.874 INFO:teuthology.orchestra.run.trial117.stdout:/etc/kernel/postinst.d/initramfs-tools: 2026-02-25T11:55:06.874 INFO:teuthology.orchestra.run.trial117.stdout:update-initramfs: Generating /boot/initrd.img-5.15.0-171-generic 2026-02-25T11:55:13.890 INFO:teuthology.orchestra.run.trial117.stdout:W: Possible missing firmware /lib/firmware/ast_dp501_fw.bin for module ast 2026-02-25T11:55:20.499 INFO:teuthology.orchestra.run.trial117.stdout:/etc/kernel/postinst.d/zz-update-grub: 2026-02-25T11:55:20.499 INFO:teuthology.orchestra.run.trial117.stdout:Sourcing file `/etc/default/grub' 2026-02-25T11:55:20.519 INFO:teuthology.orchestra.run.trial117.stdout:Sourcing file `/etc/default/grub.d/50-curtin-settings.cfg' 2026-02-25T11:55:20.520 INFO:teuthology.orchestra.run.trial117.stdout:Sourcing file `/etc/default/grub.d/init-select.cfg' 2026-02-25T11:55:20.523 INFO:teuthology.orchestra.run.trial117.stdout:Generating grub configuration file ... 2026-02-25T11:55:20.608 INFO:teuthology.orchestra.run.trial117.stdout:Found linux image: /boot/vmlinuz-5.15.0-171-generic 2026-02-25T11:55:20.615 INFO:teuthology.orchestra.run.trial117.stdout:Found initrd image: /boot/initrd.img-5.15.0-171-generic 2026-02-25T11:55:20.790 INFO:teuthology.orchestra.run.trial117.stdout:Found linux image: /boot/vmlinuz-5.15.0-170-generic 2026-02-25T11:55:20.796 INFO:teuthology.orchestra.run.trial117.stdout:Found initrd image: /boot/initrd.img-5.15.0-170-generic 2026-02-25T11:55:20.840 INFO:teuthology.orchestra.run.trial117.stdout:Found linux image: /boot/vmlinuz-5.15.0-164-generic 2026-02-25T11:55:20.846 INFO:teuthology.orchestra.run.trial117.stdout:Found initrd image: /boot/initrd.img-5.15.0-164-generic 2026-02-25T11:55:20.920 INFO:teuthology.orchestra.run.trial117.stdout:Warning: os-prober will not be executed to detect other bootable partitions. 2026-02-25T11:55:20.920 INFO:teuthology.orchestra.run.trial117.stdout:Systems on them will not be added to the GRUB boot configuration. 2026-02-25T11:55:20.920 INFO:teuthology.orchestra.run.trial117.stdout:Check GRUB_DISABLE_OS_PROBER documentation entry. 2026-02-25T11:55:20.922 INFO:teuthology.orchestra.run.trial117.stdout:Adding boot menu entry for UEFI Firmware Settings ... 2026-02-25T11:55:20.932 INFO:teuthology.orchestra.run.trial117.stdout:done 2026-02-25T11:55:21.341 INFO:teuthology.orchestra.run.trial117.stdout: 2026-02-25T11:55:21.341 INFO:teuthology.orchestra.run.trial117.stdout:Pending kernel upgrade! 2026-02-25T11:55:21.341 INFO:teuthology.orchestra.run.trial117.stdout: 2026-02-25T11:55:21.341 INFO:teuthology.orchestra.run.trial117.stdout:Running kernel version: 2026-02-25T11:55:21.341 INFO:teuthology.orchestra.run.trial117.stdout: 5.15.0-170-generic 2026-02-25T11:55:21.341 INFO:teuthology.orchestra.run.trial117.stdout: 2026-02-25T11:55:21.341 INFO:teuthology.orchestra.run.trial117.stdout:Diagnostics: 2026-02-25T11:55:21.341 INFO:teuthology.orchestra.run.trial117.stdout: The currently running kernel version is not the expected kernel version 5.15.0-171-generic. 2026-02-25T11:55:21.342 INFO:teuthology.orchestra.run.trial117.stdout: 2026-02-25T11:55:21.342 INFO:teuthology.orchestra.run.trial117.stdout:Restarting the system to load the new kernel will not be handled automatically, so you should consider rebooting. [Return] 2026-02-25T11:55:21.342 INFO:teuthology.orchestra.run.trial117.stdout: 2026-02-25T11:55:21.342 INFO:teuthology.orchestra.run.trial117.stdout:The processor microcode seems to be up-to-date. 2026-02-25T11:55:21.342 INFO:teuthology.orchestra.run.trial117.stdout: 2026-02-25T11:55:21.342 INFO:teuthology.orchestra.run.trial117.stdout:No services need to be restarted. 2026-02-25T11:55:21.342 INFO:teuthology.orchestra.run.trial117.stdout: 2026-02-25T11:55:21.342 INFO:teuthology.orchestra.run.trial117.stdout:No containers need to be restarted. 2026-02-25T11:55:21.342 INFO:teuthology.orchestra.run.trial117.stdout: 2026-02-25T11:55:21.342 INFO:teuthology.orchestra.run.trial117.stdout:No user sessions are running outdated binaries. 2026-02-25T11:55:21.342 INFO:teuthology.orchestra.run.trial117.stdout: 2026-02-25T11:55:21.342 INFO:teuthology.orchestra.run.trial117.stdout:No VM guests are running outdated hypervisor (qemu) binaries on this host. 2026-02-25T11:55:22.185 DEBUG:teuthology.orchestra.run.trial117:> dpkg -s linux-image-generic 2026-02-25T11:55:22.236 INFO:teuthology.orchestra.run.trial117.stdout:Package: linux-image-generic 2026-02-25T11:55:22.236 INFO:teuthology.orchestra.run.trial117.stdout:Status: install ok installed 2026-02-25T11:55:22.236 INFO:teuthology.orchestra.run.trial117.stdout:Priority: optional 2026-02-25T11:55:22.236 INFO:teuthology.orchestra.run.trial117.stdout:Section: kernel 2026-02-25T11:55:22.236 INFO:teuthology.orchestra.run.trial117.stdout:Installed-Size: 22 2026-02-25T11:55:22.236 INFO:teuthology.orchestra.run.trial117.stdout:Maintainer: Ubuntu Kernel Team 2026-02-25T11:55:22.236 INFO:teuthology.orchestra.run.trial117.stdout:Architecture: amd64 2026-02-25T11:55:22.237 INFO:teuthology.orchestra.run.trial117.stdout:Source: linux-meta 2026-02-25T11:55:22.237 INFO:teuthology.orchestra.run.trial117.stdout:Version: 5.15.0.171.160 2026-02-25T11:55:22.237 INFO:teuthology.orchestra.run.trial117.stdout:Provides: spl-modules (= 2.1.5-1ubuntu6~22.04.6), v4l2loopback-modules (= 0.12.7-2ubuntu2~22.04.1), virtualbox-guest-modules (= 5.15.0-171), wireguard-modules (= 1.0.0), zfs-modules (= 2.1.5-1ubuntu6~22.04.6) 2026-02-25T11:55:22.237 INFO:teuthology.orchestra.run.trial117.stdout:Depends: linux-image-5.15.0-171-generic, linux-modules-extra-5.15.0-171-generic, linux-firmware, intel-microcode, amd64-microcode 2026-02-25T11:55:22.237 INFO:teuthology.orchestra.run.trial117.stdout:Recommends: thermald 2026-02-25T11:55:22.237 INFO:teuthology.orchestra.run.trial117.stdout:Description: Generic Linux kernel image 2026-02-25T11:55:22.237 INFO:teuthology.orchestra.run.trial117.stdout: This package will always depend on the latest generic kernel image 2026-02-25T11:55:22.237 INFO:teuthology.orchestra.run.trial117.stdout: available. 2026-02-25T11:55:22.237 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.15.0-170-generic Expected: 5.15.0-171-generic 2026-02-25T11:55:22.238 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2026-02-25T11:55:22.238 DEBUG:teuthology.task.kernel:src is distro, skipping download 2026-02-25T11:55:22.238 INFO:teuthology.task.kernel:Installing distro kernel on mon.a... 2026-02-25T11:55:22.238 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@trial117.front.sepia.ceph.com, path=None, version=distro) 2026-02-25T11:55:22.238 DEBUG:teuthology.orchestra.run.trial117:> sudo apt-get clean 2026-02-25T11:55:22.318 DEBUG:teuthology.orchestra.run.trial117:> sudo apt-get update 2026-02-25T11:55:22.748 INFO:teuthology.orchestra.run.trial117.stdout:Hit:1 https://archive.ubuntu.com/ubuntu jammy InRelease 2026-02-25T11:55:22.815 INFO:teuthology.orchestra.run.trial117.stdout:Hit:2 https://archive.ubuntu.com/ubuntu jammy-updates InRelease 2026-02-25T11:55:22.887 INFO:teuthology.orchestra.run.trial117.stdout:Hit:3 https://archive.ubuntu.com/ubuntu jammy-security InRelease 2026-02-25T11:55:22.960 INFO:teuthology.orchestra.run.trial117.stdout:Hit:4 https://archive.ubuntu.com/ubuntu jammy-backports InRelease 2026-02-25T11:55:23.610 INFO:teuthology.orchestra.run.trial117.stdout:Reading package lists... 2026-02-25T11:55:23.624 DEBUG:teuthology.orchestra.run.trial117:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2026-02-25T11:55:23.657 INFO:teuthology.orchestra.run.trial117.stdout:Reading package lists... 2026-02-25T11:55:23.765 INFO:teuthology.orchestra.run.trial117.stdout:Building dependency tree... 2026-02-25T11:55:23.765 INFO:teuthology.orchestra.run.trial117.stdout:Reading state information... 2026-02-25T11:55:23.851 INFO:teuthology.orchestra.run.trial117.stdout:linux-image-generic is already the newest version (5.15.0.171.160). 2026-02-25T11:55:23.851 INFO:teuthology.orchestra.run.trial117.stdout:linux-image-generic set to manually installed. 2026-02-25T11:55:23.852 INFO:teuthology.orchestra.run.trial117.stdout:The following packages were automatically installed and are no longer required: 2026-02-25T11:55:23.852 INFO:teuthology.orchestra.run.trial117.stdout: kpartx libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 2026-02-25T11:55:23.852 INFO:teuthology.orchestra.run.trial117.stdout: libgfxdr0 libglusterfs0 libiscsi7 libpmemobj1 libsgutils2-2 2026-02-25T11:55:23.852 INFO:teuthology.orchestra.run.trial117.stdout: linux-headers-5.15.0-164 linux-headers-5.15.0-164-generic 2026-02-25T11:55:23.852 INFO:teuthology.orchestra.run.trial117.stdout: linux-image-5.15.0-164-generic linux-modules-5.15.0-164-generic 2026-02-25T11:55:23.852 INFO:teuthology.orchestra.run.trial117.stdout: linux-modules-extra-5.15.0-164-generic sg3-utils sg3-utils-udev 2026-02-25T11:55:23.852 INFO:teuthology.orchestra.run.trial117.stdout:Use 'sudo apt autoremove' to remove them. 2026-02-25T11:55:23.872 INFO:teuthology.orchestra.run.trial117.stdout:0 upgraded, 0 newly installed, 0 to remove and 29 not upgraded. 2026-02-25T11:55:23.874 DEBUG:teuthology.orchestra.run.trial117:> dpkg -s linux-image-generic 2026-02-25T11:55:23.925 INFO:teuthology.orchestra.run.trial117.stdout:Package: linux-image-generic 2026-02-25T11:55:23.926 INFO:teuthology.orchestra.run.trial117.stdout:Status: install ok installed 2026-02-25T11:55:23.926 INFO:teuthology.orchestra.run.trial117.stdout:Priority: optional 2026-02-25T11:55:23.926 INFO:teuthology.orchestra.run.trial117.stdout:Section: kernel 2026-02-25T11:55:23.926 INFO:teuthology.orchestra.run.trial117.stdout:Installed-Size: 22 2026-02-25T11:55:23.926 INFO:teuthology.orchestra.run.trial117.stdout:Maintainer: Ubuntu Kernel Team 2026-02-25T11:55:23.926 INFO:teuthology.orchestra.run.trial117.stdout:Architecture: amd64 2026-02-25T11:55:23.926 INFO:teuthology.orchestra.run.trial117.stdout:Source: linux-meta 2026-02-25T11:55:23.926 INFO:teuthology.orchestra.run.trial117.stdout:Version: 5.15.0.171.160 2026-02-25T11:55:23.926 INFO:teuthology.orchestra.run.trial117.stdout:Provides: spl-modules (= 2.1.5-1ubuntu6~22.04.6), v4l2loopback-modules (= 0.12.7-2ubuntu2~22.04.1), virtualbox-guest-modules (= 5.15.0-171), wireguard-modules (= 1.0.0), zfs-modules (= 2.1.5-1ubuntu6~22.04.6) 2026-02-25T11:55:23.926 INFO:teuthology.orchestra.run.trial117.stdout:Depends: linux-image-5.15.0-171-generic, linux-modules-extra-5.15.0-171-generic, linux-firmware, intel-microcode, amd64-microcode 2026-02-25T11:55:23.926 INFO:teuthology.orchestra.run.trial117.stdout:Recommends: thermald 2026-02-25T11:55:23.926 INFO:teuthology.orchestra.run.trial117.stdout:Description: Generic Linux kernel image 2026-02-25T11:55:23.926 INFO:teuthology.orchestra.run.trial117.stdout: This package will always depend on the latest generic kernel image 2026-02-25T11:55:23.926 INFO:teuthology.orchestra.run.trial117.stdout: available. 2026-02-25T11:55:23.926 DEBUG:teuthology.orchestra.run.trial117:> mktemp 2026-02-25T11:55:23.973 INFO:teuthology.orchestra.run.trial117.stdout:/tmp/tmp.mbCbR79rd0 2026-02-25T11:55:23.973 DEBUG:teuthology.orchestra.run.trial117:> sudo cp /boot/grub/grub.cfg /tmp/tmp.mbCbR79rd0 2026-02-25T11:55:24.025 DEBUG:teuthology.orchestra.run.trial117:> sudo chmod 0666 /tmp/tmp.mbCbR79rd0 2026-02-25T11:55:24.122 DEBUG:teuthology.orchestra.remote:trial117:/tmp/tmp.mbCbR79rd0 is 9KB 2026-02-25T11:55:24.128 DEBUG:teuthology.orchestra.run.trial117:> rm -fr /tmp/tmp.mbCbR79rd0 2026-02-25T11:55:24.131 DEBUG:teuthology.orchestra.run.trial117:> sudo rm -f -- /etc/grub.d/01_ceph_kernel 2026-02-25T11:55:24.185 DEBUG:teuthology.orchestra.run.trial117:> set -ex 2026-02-25T11:55:24.186 DEBUG:teuthology.orchestra.run.trial117:> sudo dd of=/etc/grub.d/01_ceph_kernel 2026-02-25T11:55:24.186 DEBUG:teuthology.orchestra.run.trial117:> sudo chmod 755 /etc/grub.d/01_ceph_kernel 2026-02-25T11:55:24.243 INFO:teuthology.task.kernel:Distro Kernel Version: 5.15.0-171-generic 2026-02-25T11:55:24.244 DEBUG:teuthology.orchestra.run.trial117:> sudo update-grub 2026-02-25T11:55:24.480 INFO:teuthology.orchestra.run.trial117.stderr:Sourcing file `/etc/default/grub' 2026-02-25T11:55:24.499 INFO:teuthology.orchestra.run.trial117.stderr:Sourcing file `/etc/default/grub.d/50-curtin-settings.cfg' 2026-02-25T11:55:24.500 INFO:teuthology.orchestra.run.trial117.stderr:Sourcing file `/etc/default/grub.d/init-select.cfg' 2026-02-25T11:55:24.502 INFO:teuthology.orchestra.run.trial117.stderr:Generating grub configuration file ... 2026-02-25T11:55:24.585 INFO:teuthology.orchestra.run.trial117.stderr:Found linux image: /boot/vmlinuz-5.15.0-171-generic 2026-02-25T11:55:24.591 INFO:teuthology.orchestra.run.trial117.stderr:Found initrd image: /boot/initrd.img-5.15.0-171-generic 2026-02-25T11:55:24.770 INFO:teuthology.orchestra.run.trial117.stderr:Found linux image: /boot/vmlinuz-5.15.0-170-generic 2026-02-25T11:55:24.776 INFO:teuthology.orchestra.run.trial117.stderr:Found initrd image: /boot/initrd.img-5.15.0-170-generic 2026-02-25T11:55:24.819 INFO:teuthology.orchestra.run.trial117.stderr:Found linux image: /boot/vmlinuz-5.15.0-164-generic 2026-02-25T11:55:24.824 INFO:teuthology.orchestra.run.trial117.stderr:Found initrd image: /boot/initrd.img-5.15.0-164-generic 2026-02-25T11:55:24.896 INFO:teuthology.orchestra.run.trial117.stderr:Warning: os-prober will not be executed to detect other bootable partitions. 2026-02-25T11:55:24.896 INFO:teuthology.orchestra.run.trial117.stderr:Systems on them will not be added to the GRUB boot configuration. 2026-02-25T11:55:24.896 INFO:teuthology.orchestra.run.trial117.stderr:Check GRUB_DISABLE_OS_PROBER documentation entry. 2026-02-25T11:55:24.898 INFO:teuthology.orchestra.run.trial117.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2026-02-25T11:55:24.906 INFO:teuthology.orchestra.run.trial117.stderr:done 2026-02-25T11:55:24.908 INFO:teuthology.orchestra.remote:Triggering safe hard reboot via SysRq on trial117 2026-02-25T11:55:24.908 DEBUG:teuthology.orchestra.run.trial117:> sudo bash -c ' 2026-02-25T11:55:24.908 DEBUG:teuthology.orchestra.run.trial117:> echo pci > /sys/kernel/reboot/type && echo 1 > /sys/kernel/reboot/force && sync && echo s > /proc/sysrq-trigger && echo u > /proc/sysrq-trigger && sleep 2 && echo b > /proc/sysrq-trigger 2026-02-25T11:55:24.908 DEBUG:teuthology.orchestra.run.trial117:> ' 2026-02-25T11:55:54.911 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version... 2026-02-25T11:55:54.912 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@trial117.front.sepia.ceph.com' 2026-02-25T11:55:54.912 DEBUG:teuthology.orchestra.connection:{'hostname': 'trial117.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2026-02-25T11:56:09.216 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 10.20.193.117 2026-02-25T11:56:18.224 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@trial117.front.sepia.ceph.com' 2026-02-25T11:56:18.225 DEBUG:teuthology.orchestra.connection:{'hostname': 'trial117.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2026-02-25T11:57:18.232 DEBUG:teuthology.orchestra.remote:timed out 2026-02-25T11:57:30.243 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@trial117.front.sepia.ceph.com' 2026-02-25T11:57:30.244 DEBUG:teuthology.orchestra.connection:{'hostname': 'trial117.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2026-02-25T11:57:30.368 DEBUG:teuthology.orchestra.run.trial117:> true 2026-02-25T11:57:30.757 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@trial117.front.sepia.ceph.com' 2026-02-25T11:57:30.757 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "5.15.0-171-generic"... 2026-02-25T11:57:30.757 DEBUG:teuthology.orchestra.run.trial117:> uname -r 2026-02-25T11:57:30.803 INFO:teuthology.orchestra.run.trial117.stdout:5.15.0-171-generic 2026-02-25T11:57:30.803 DEBUG:teuthology.task.kernel:current kernel version is 5.15.0-171-generic vs 5.15.0-171-generic 2026-02-25T11:57:30.804 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2026-02-25T11:57:30.804 DEBUG:teuthology.task.kernel:Distro of this test job: ubuntu 2026-02-25T11:57:31.805 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2026-02-25T11:57:31.805 DEBUG:teuthology.orchestra.run.trial117:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2026-02-25T11:57:31.817 INFO:teuthology.orchestra.run.trial117.stdout:ttyS1 2026-02-25T11:57:31.838 DEBUG:teuthology.parallel:result is None 2026-02-25T11:57:31.838 INFO:teuthology.run_tasks:Running task internal.base... 2026-02-25T11:57:31.848 INFO:teuthology.task.internal:Creating test directory... 2026-02-25T11:57:31.849 DEBUG:teuthology.orchestra.run.trial117:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2026-02-25T11:57:31.883 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2026-02-25T11:57:31.894 INFO:teuthology.run_tasks:Running task internal.archive... 2026-02-25T11:57:31.907 INFO:teuthology.task.internal:Creating archive directory... 2026-02-25T11:57:31.907 DEBUG:teuthology.orchestra.run.trial117:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2026-02-25T11:57:31.934 INFO:teuthology.run_tasks:Running task internal.coredump... 2026-02-25T11:57:31.944 INFO:teuthology.task.internal:Enabling coredump saving... 2026-02-25T11:57:31.945 DEBUG:teuthology.orchestra.run.trial117:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2026-02-25T11:57:31.982 INFO:teuthology.orchestra.run.trial117.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-25T11:57:31.988 INFO:teuthology.orchestra.run.trial117.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-25T11:57:31.989 INFO:teuthology.run_tasks:Running task internal.sudo... 2026-02-25T11:57:32.004 INFO:teuthology.task.internal:Configuring sudo... 2026-02-25T11:57:32.004 DEBUG:teuthology.orchestra.run.trial117:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2026-02-25T11:57:32.040 INFO:teuthology.run_tasks:Running task internal.syslog... 2026-02-25T11:57:32.059 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2026-02-25T11:57:32.059 DEBUG:teuthology.orchestra.run.trial117:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2026-02-25T11:57:32.088 DEBUG:teuthology.orchestra.run.trial117:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2026-02-25T11:57:32.131 DEBUG:teuthology.orchestra.run.trial117:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2026-02-25T11:57:32.176 DEBUG:teuthology.orchestra.run.trial117:> set -ex 2026-02-25T11:57:32.176 DEBUG:teuthology.orchestra.run.trial117:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2026-02-25T11:57:32.224 DEBUG:teuthology.orchestra.run.trial117:> sudo service rsyslog restart 2026-02-25T11:57:32.279 INFO:teuthology.run_tasks:Running task internal.timer... 2026-02-25T11:57:32.290 INFO:teuthology.task.internal:Starting timer... 2026-02-25T11:57:32.290 INFO:teuthology.run_tasks:Running task pcp... 2026-02-25T11:57:32.308 INFO:teuthology.run_tasks:Running task selinux... 2026-02-25T11:57:32.322 DEBUG:teuthology.task.selinux:Excluding trial117: OS 'ubuntu' does not support SELinux 2026-02-25T11:57:32.322 DEBUG:teuthology.task.selinux:Getting current SELinux state 2026-02-25T11:57:32.322 DEBUG:teuthology.task.selinux:Existing SELinux modes: {} 2026-02-25T11:57:32.322 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2026-02-25T11:57:32.322 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2026-02-25T11:57:32.338 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2026-02-25T11:57:32.350 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2026-02-25T11:57:32.351 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit trial117.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2026-02-25T11:59:52.797 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@trial117.front.sepia.ceph.com')] 2026-02-25T11:59:52.798 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@trial117.front.sepia.ceph.com' 2026-02-25T11:59:52.798 DEBUG:teuthology.orchestra.connection:{'hostname': 'trial117.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2026-02-25T11:59:52.888 DEBUG:teuthology.orchestra.run.trial117:> true 2026-02-25T11:59:53.142 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@trial117.front.sepia.ceph.com' 2026-02-25T11:59:53.143 INFO:teuthology.run_tasks:Running task clock... 2026-02-25T11:59:53.159 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2026-02-25T11:59:53.159 INFO:teuthology.orchestra.run:Running command with timeout 360 2026-02-25T11:59:53.160 DEBUG:teuthology.orchestra.run.trial117:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2026-02-25T11:59:53.202 INFO:teuthology.orchestra.run.trial117.stdout:25 Feb 11:59:53 ntpd[8824]: ntpd 4.2.8p15@1.3728-o Wed Feb 16 17:13:02 UTC 2022 (1): Starting 2026-02-25T11:59:53.202 INFO:teuthology.orchestra.run.trial117.stdout:25 Feb 11:59:53 ntpd[8824]: Command line: ntpd -gq 2026-02-25T11:59:53.202 INFO:teuthology.orchestra.run.trial117.stdout:25 Feb 11:59:53 ntpd[8824]: ---------------------------------------------------- 2026-02-25T11:59:53.203 INFO:teuthology.orchestra.run.trial117.stdout:25 Feb 11:59:53 ntpd[8824]: ntp-4 is maintained by Network Time Foundation, 2026-02-25T11:59:53.203 INFO:teuthology.orchestra.run.trial117.stdout:25 Feb 11:59:53 ntpd[8824]: Inc. (NTF), a non-profit 501(c)(3) public-benefit 2026-02-25T11:59:53.203 INFO:teuthology.orchestra.run.trial117.stdout:25 Feb 11:59:53 ntpd[8824]: corporation. Support and training for ntp-4 are 2026-02-25T11:59:53.203 INFO:teuthology.orchestra.run.trial117.stdout:25 Feb 11:59:53 ntpd[8824]: available at https://www.nwtime.org/support 2026-02-25T11:59:53.203 INFO:teuthology.orchestra.run.trial117.stdout:25 Feb 11:59:53 ntpd[8824]: ---------------------------------------------------- 2026-02-25T11:59:53.203 INFO:teuthology.orchestra.run.trial117.stdout:25 Feb 11:59:53 ntpd[8824]: proto: precision = 0.020 usec (-25) 2026-02-25T11:59:53.203 INFO:teuthology.orchestra.run.trial117.stdout:25 Feb 11:59:53 ntpd[8824]: basedate set to 2022-02-04 2026-02-25T11:59:53.203 INFO:teuthology.orchestra.run.trial117.stdout:25 Feb 11:59:53 ntpd[8824]: gps base set to 2022-02-06 (week 2196) 2026-02-25T11:59:53.204 INFO:teuthology.orchestra.run.trial117.stderr:restrict 0.0.0.0: KOD does nothing without LIMITED. 2026-02-25T11:59:53.205 INFO:teuthology.orchestra.run.trial117.stdout:25 Feb 11:59:53 ntpd[8824]: restrict 0.0.0.0: KOD does nothing without LIMITED. 2026-02-25T11:59:53.205 INFO:teuthology.orchestra.run.trial117.stdout:25 Feb 11:59:53 ntpd[8824]: restrict ::: KOD does nothing without LIMITED. 2026-02-25T11:59:53.205 INFO:teuthology.orchestra.run.trial117.stderr:restrict ::: KOD does nothing without LIMITED. 2026-02-25T11:59:53.207 INFO:teuthology.orchestra.run.trial117.stdout:25 Feb 11:59:53 ntpd[8824]: Listen and drop on 0 v6wildcard [::]:123 2026-02-25T11:59:53.207 INFO:teuthology.orchestra.run.trial117.stdout:25 Feb 11:59:53 ntpd[8824]: Listen and drop on 1 v4wildcard 0.0.0.0:123 2026-02-25T11:59:53.207 INFO:teuthology.orchestra.run.trial117.stdout:25 Feb 11:59:53 ntpd[8824]: Listen normally on 2 lo 127.0.0.1:123 2026-02-25T11:59:53.207 INFO:teuthology.orchestra.run.trial117.stdout:25 Feb 11:59:53 ntpd[8824]: Listen normally on 3 enp1s0f0 10.20.193.117:123 2026-02-25T11:59:53.207 INFO:teuthology.orchestra.run.trial117.stdout:25 Feb 11:59:53 ntpd[8824]: Listen normally on 4 lo [::1]:123 2026-02-25T11:59:53.208 INFO:teuthology.orchestra.run.trial117.stdout:25 Feb 11:59:53 ntpd[8824]: Listen normally on 5 enp1s0f0 [fe80::925a:8ff:fe77:6196%2]:123 2026-02-25T11:59:53.208 INFO:teuthology.orchestra.run.trial117.stdout:25 Feb 11:59:53 ntpd[8824]: Listening on routing socket on fd #22 for interface updates 2026-02-25T11:59:54.202 INFO:teuthology.orchestra.run.trial117.stderr:25 Feb 11:59:54 ntpd[8824]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2026-02-25T11:59:54.202 INFO:teuthology.orchestra.run.trial117.stderr:25 Feb 11:59:54 ntpd[8824]: can't open /var/log/ntpstats/rawstats.20260225: Permission denied 2026-02-25T11:59:54.205 INFO:teuthology.orchestra.run.trial117.stderr:25 Feb 11:59:54 ntpd[8824]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2026-02-25T11:59:54.205 INFO:teuthology.orchestra.run.trial117.stderr:25 Feb 11:59:54 ntpd[8824]: can't open /var/log/ntpstats/rawstats.20260225: Permission denied 2026-02-25T11:59:54.205 INFO:teuthology.orchestra.run.trial117.stderr:25 Feb 11:59:54 ntpd[8824]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2026-02-25T11:59:54.205 INFO:teuthology.orchestra.run.trial117.stderr:25 Feb 11:59:54 ntpd[8824]: can't open /var/log/ntpstats/peerstats.20260225: Permission denied 2026-02-25T11:59:55.201 INFO:teuthology.orchestra.run.trial117.stderr:25 Feb 11:59:55 ntpd[8824]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2026-02-25T11:59:55.201 INFO:teuthology.orchestra.run.trial117.stderr:25 Feb 11:59:55 ntpd[8824]: can't open /var/log/ntpstats/rawstats.20260225: Permission denied 2026-02-25T11:59:55.202 INFO:teuthology.orchestra.run.trial117.stderr:25 Feb 11:59:55 ntpd[8824]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2026-02-25T11:59:55.202 INFO:teuthology.orchestra.run.trial117.stderr:25 Feb 11:59:55 ntpd[8824]: can't open /var/log/ntpstats/rawstats.20260225: Permission denied 2026-02-25T11:59:55.202 INFO:teuthology.orchestra.run.trial117.stderr:25 Feb 11:59:55 ntpd[8824]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2026-02-25T11:59:55.202 INFO:teuthology.orchestra.run.trial117.stderr:25 Feb 11:59:55 ntpd[8824]: can't open /var/log/ntpstats/rawstats.20260225: Permission denied 2026-02-25T11:59:55.205 INFO:teuthology.orchestra.run.trial117.stderr:25 Feb 11:59:55 ntpd[8824]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2026-02-25T11:59:55.205 INFO:teuthology.orchestra.run.trial117.stderr:25 Feb 11:59:55 ntpd[8824]: can't open /var/log/ntpstats/rawstats.20260225: Permission denied 2026-02-25T11:59:55.205 INFO:teuthology.orchestra.run.trial117.stderr:25 Feb 11:59:55 ntpd[8824]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2026-02-25T11:59:55.205 INFO:teuthology.orchestra.run.trial117.stderr:25 Feb 11:59:55 ntpd[8824]: can't open /var/log/ntpstats/peerstats.20260225: Permission denied 2026-02-25T11:59:55.229 INFO:teuthology.orchestra.run.trial117.stderr:25 Feb 11:59:55 ntpd[8824]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2026-02-25T11:59:55.229 INFO:teuthology.orchestra.run.trial117.stderr:25 Feb 11:59:55 ntpd[8824]: can't open /var/log/ntpstats/rawstats.20260225: Permission denied 2026-02-25T11:59:55.229 INFO:teuthology.orchestra.run.trial117.stderr:25 Feb 11:59:55 ntpd[8824]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2026-02-25T11:59:55.230 INFO:teuthology.orchestra.run.trial117.stderr:25 Feb 11:59:55 ntpd[8824]: can't open /var/log/ntpstats/peerstats.20260225: Permission denied 2026-02-25T11:59:55.273 INFO:teuthology.orchestra.run.trial117.stderr:25 Feb 11:59:56 ntpd[8824]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2026-02-25T11:59:55.273 INFO:teuthology.orchestra.run.trial117.stderr:25 Feb 11:59:56 ntpd[8824]: can't open /var/log/ntpstats/rawstats.20260225: Permission denied 2026-02-25T11:59:55.273 INFO:teuthology.orchestra.run.trial117.stderr:25 Feb 11:59:56 ntpd[8824]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2026-02-25T11:59:55.273 INFO:teuthology.orchestra.run.trial117.stderr:25 Feb 11:59:56 ntpd[8824]: can't open /var/log/ntpstats/peerstats.20260225: Permission denied 2026-02-25T11:59:56.201 INFO:teuthology.orchestra.run.trial117.stderr:25 Feb 11:59:56 ntpd[8824]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2026-02-25T11:59:56.202 INFO:teuthology.orchestra.run.trial117.stderr:25 Feb 11:59:56 ntpd[8824]: can't open /var/log/ntpstats/rawstats.20260225: Permission denied 2026-02-25T11:59:56.205 INFO:teuthology.orchestra.run.trial117.stderr:25 Feb 11:59:56 ntpd[8824]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2026-02-25T11:59:56.205 INFO:teuthology.orchestra.run.trial117.stderr:25 Feb 11:59:56 ntpd[8824]: can't open /var/log/ntpstats/rawstats.20260225: Permission denied 2026-02-25T11:59:56.205 INFO:teuthology.orchestra.run.trial117.stderr:25 Feb 11:59:56 ntpd[8824]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2026-02-25T11:59:56.205 INFO:teuthology.orchestra.run.trial117.stderr:25 Feb 11:59:56 ntpd[8824]: can't open /var/log/ntpstats/peerstats.20260225: Permission denied 2026-02-25T11:59:57.201 INFO:teuthology.orchestra.run.trial117.stderr:25 Feb 11:59:57 ntpd[8824]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2026-02-25T11:59:57.201 INFO:teuthology.orchestra.run.trial117.stderr:25 Feb 11:59:57 ntpd[8824]: can't open /var/log/ntpstats/rawstats.20260225: Permission denied 2026-02-25T11:59:57.201 INFO:teuthology.orchestra.run.trial117.stderr:25 Feb 11:59:57 ntpd[8824]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2026-02-25T11:59:57.201 INFO:teuthology.orchestra.run.trial117.stderr:25 Feb 11:59:57 ntpd[8824]: can't open /var/log/ntpstats/rawstats.20260225: Permission denied 2026-02-25T11:59:57.201 INFO:teuthology.orchestra.run.trial117.stderr:25 Feb 11:59:57 ntpd[8824]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2026-02-25T11:59:57.201 INFO:teuthology.orchestra.run.trial117.stderr:25 Feb 11:59:57 ntpd[8824]: can't open /var/log/ntpstats/rawstats.20260225: Permission denied 2026-02-25T11:59:57.204 INFO:teuthology.orchestra.run.trial117.stderr:25 Feb 11:59:57 ntpd[8824]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2026-02-25T11:59:57.204 INFO:teuthology.orchestra.run.trial117.stderr:25 Feb 11:59:57 ntpd[8824]: can't open /var/log/ntpstats/rawstats.20260225: Permission denied 2026-02-25T11:59:57.204 INFO:teuthology.orchestra.run.trial117.stderr:25 Feb 11:59:57 ntpd[8824]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2026-02-25T11:59:57.204 INFO:teuthology.orchestra.run.trial117.stderr:25 Feb 11:59:57 ntpd[8824]: can't open /var/log/ntpstats/peerstats.20260225: Permission denied 2026-02-25T11:59:57.228 INFO:teuthology.orchestra.run.trial117.stderr:25 Feb 11:59:57 ntpd[8824]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2026-02-25T11:59:57.228 INFO:teuthology.orchestra.run.trial117.stderr:25 Feb 11:59:57 ntpd[8824]: can't open /var/log/ntpstats/rawstats.20260225: Permission denied 2026-02-25T11:59:57.229 INFO:teuthology.orchestra.run.trial117.stderr:25 Feb 11:59:57 ntpd[8824]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2026-02-25T11:59:57.229 INFO:teuthology.orchestra.run.trial117.stderr:25 Feb 11:59:57 ntpd[8824]: can't open /var/log/ntpstats/peerstats.20260225: Permission denied 2026-02-25T11:59:57.272 INFO:teuthology.orchestra.run.trial117.stderr:25 Feb 11:59:58 ntpd[8824]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2026-02-25T11:59:57.272 INFO:teuthology.orchestra.run.trial117.stderr:25 Feb 11:59:58 ntpd[8824]: can't open /var/log/ntpstats/rawstats.20260225: Permission denied 2026-02-25T11:59:57.272 INFO:teuthology.orchestra.run.trial117.stderr:25 Feb 11:59:58 ntpd[8824]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2026-02-25T11:59:57.272 INFO:teuthology.orchestra.run.trial117.stderr:25 Feb 11:59:58 ntpd[8824]: can't open /var/log/ntpstats/peerstats.20260225: Permission denied 2026-02-25T11:59:58.201 INFO:teuthology.orchestra.run.trial117.stderr:25 Feb 11:59:58 ntpd[8824]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2026-02-25T11:59:58.202 INFO:teuthology.orchestra.run.trial117.stderr:25 Feb 11:59:58 ntpd[8824]: can't open /var/log/ntpstats/rawstats.20260225: Permission denied 2026-02-25T11:59:58.206 INFO:teuthology.orchestra.run.trial117.stderr:25 Feb 11:59:58 ntpd[8824]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2026-02-25T11:59:58.207 INFO:teuthology.orchestra.run.trial117.stderr:25 Feb 11:59:58 ntpd[8824]: can't open /var/log/ntpstats/rawstats.20260225: Permission denied 2026-02-25T11:59:58.207 INFO:teuthology.orchestra.run.trial117.stderr:25 Feb 11:59:58 ntpd[8824]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2026-02-25T11:59:58.207 INFO:teuthology.orchestra.run.trial117.stderr:25 Feb 11:59:58 ntpd[8824]: can't open /var/log/ntpstats/peerstats.20260225: Permission denied 2026-02-25T11:59:59.202 INFO:teuthology.orchestra.run.trial117.stderr:25 Feb 11:59:59 ntpd[8824]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2026-02-25T11:59:59.202 INFO:teuthology.orchestra.run.trial117.stderr:25 Feb 11:59:59 ntpd[8824]: can't open /var/log/ntpstats/rawstats.20260225: Permission denied 2026-02-25T11:59:59.202 INFO:teuthology.orchestra.run.trial117.stderr:25 Feb 11:59:59 ntpd[8824]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2026-02-25T11:59:59.203 INFO:teuthology.orchestra.run.trial117.stderr:25 Feb 11:59:59 ntpd[8824]: can't open /var/log/ntpstats/rawstats.20260225: Permission denied 2026-02-25T11:59:59.203 INFO:teuthology.orchestra.run.trial117.stderr:25 Feb 11:59:59 ntpd[8824]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2026-02-25T11:59:59.203 INFO:teuthology.orchestra.run.trial117.stderr:25 Feb 11:59:59 ntpd[8824]: can't open /var/log/ntpstats/rawstats.20260225: Permission denied 2026-02-25T11:59:59.205 INFO:teuthology.orchestra.run.trial117.stderr:25 Feb 11:59:59 ntpd[8824]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2026-02-25T11:59:59.205 INFO:teuthology.orchestra.run.trial117.stderr:25 Feb 11:59:59 ntpd[8824]: can't open /var/log/ntpstats/rawstats.20260225: Permission denied 2026-02-25T11:59:59.205 INFO:teuthology.orchestra.run.trial117.stderr:25 Feb 11:59:59 ntpd[8824]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2026-02-25T11:59:59.205 INFO:teuthology.orchestra.run.trial117.stderr:25 Feb 11:59:59 ntpd[8824]: can't open /var/log/ntpstats/peerstats.20260225: Permission denied 2026-02-25T11:59:59.229 INFO:teuthology.orchestra.run.trial117.stderr:25 Feb 11:59:59 ntpd[8824]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2026-02-25T11:59:59.229 INFO:teuthology.orchestra.run.trial117.stderr:25 Feb 11:59:59 ntpd[8824]: can't open /var/log/ntpstats/rawstats.20260225: Permission denied 2026-02-25T11:59:59.229 INFO:teuthology.orchestra.run.trial117.stderr:25 Feb 11:59:59 ntpd[8824]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2026-02-25T11:59:59.229 INFO:teuthology.orchestra.run.trial117.stderr:25 Feb 11:59:59 ntpd[8824]: can't open /var/log/ntpstats/peerstats.20260225: Permission denied 2026-02-25T11:59:59.273 INFO:teuthology.orchestra.run.trial117.stderr:25 Feb 12:00:00 ntpd[8824]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2026-02-25T11:59:59.273 INFO:teuthology.orchestra.run.trial117.stderr:25 Feb 12:00:00 ntpd[8824]: can't open /var/log/ntpstats/rawstats.20260225: Permission denied 2026-02-25T11:59:59.273 INFO:teuthology.orchestra.run.trial117.stderr:25 Feb 12:00:00 ntpd[8824]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2026-02-25T11:59:59.273 INFO:teuthology.orchestra.run.trial117.stderr:25 Feb 12:00:00 ntpd[8824]: can't open /var/log/ntpstats/peerstats.20260225: Permission denied 2026-02-25T12:00:00.202 INFO:teuthology.orchestra.run.trial117.stderr:25 Feb 12:00:00 ntpd[8824]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2026-02-25T12:00:00.202 INFO:teuthology.orchestra.run.trial117.stderr:25 Feb 12:00:00 ntpd[8824]: can't open /var/log/ntpstats/rawstats.20260225: Permission denied 2026-02-25T12:00:00.205 INFO:teuthology.orchestra.run.trial117.stderr:25 Feb 12:00:00 ntpd[8824]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2026-02-25T12:00:00.205 INFO:teuthology.orchestra.run.trial117.stderr:25 Feb 12:00:00 ntpd[8824]: can't open /var/log/ntpstats/rawstats.20260225: Permission denied 2026-02-25T12:00:00.205 INFO:teuthology.orchestra.run.trial117.stderr:25 Feb 12:00:00 ntpd[8824]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2026-02-25T12:00:00.206 INFO:teuthology.orchestra.run.trial117.stderr:25 Feb 12:00:00 ntpd[8824]: can't open /var/log/ntpstats/peerstats.20260225: Permission denied 2026-02-25T12:00:00.206 INFO:teuthology.orchestra.run.trial117.stdout:25 Feb 12:00:00 ntpd[8824]: ntpd: time set -0.736805 s 2026-02-25T12:00:00.206 INFO:teuthology.orchestra.run.trial117.stdout:ntpd: time set -0.736805s 2026-02-25T12:00:00.207 INFO:teuthology.orchestra.run.trial117.stderr:25 Feb 12:00:00 ntpd[8824]: couldn't unlink /var/log/ntpstats/loopstats: Permission denied 2026-02-25T12:00:00.207 INFO:teuthology.orchestra.run.trial117.stderr:25 Feb 12:00:00 ntpd[8824]: can't open /var/log/ntpstats/loopstats.20260225: Permission denied 2026-02-25T12:00:00.254 INFO:teuthology.orchestra.run.trial117.stdout: remote refid st t when poll reach delay offset jitter 2026-02-25T12:00:00.254 INFO:teuthology.orchestra.run.trial117.stdout:============================================================================== 2026-02-25T12:00:00.254 INFO:teuthology.orchestra.run.trial117.stdout: 69-10-208-170.r .INIT. 16 u - 64 0 0.000 +0.000 0.000 2026-02-25T12:00:00.255 INFO:teuthology.orchestra.run.trial117.stdout: B1-66ER.matrix. .INIT. 16 u - 64 0 0.000 +0.000 0.000 2026-02-25T12:00:00.255 INFO:teuthology.orchestra.run.trial117.stdout: 38.45.64.130 .INIT. 16 u - 64 0 0.000 +0.000 0.000 2026-02-25T12:00:00.255 INFO:teuthology.orchestra.run.trial117.stdout: 170-187-142-180 .INIT. 16 u - 64 0 0.000 +0.000 0.000 2026-02-25T12:00:00.256 INFO:teuthology.run_tasks:Running task install... 2026-02-25T12:00:00.271 DEBUG:teuthology.task.install:project ceph 2026-02-25T12:00:00.271 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': '506b402a3d3a240d0bfe073b4769eaffdf84497b'}} 2026-02-25T12:00:00.271 DEBUG:teuthology.task.install:config {'flavor': 'default', 'sha1': '506b402a3d3a240d0bfe073b4769eaffdf84497b'} 2026-02-25T12:00:00.271 INFO:teuthology.task.install:Using flavor: default 2026-02-25T12:00:00.281 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-osd', 'ceph-osd-classic', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-osd', 'ceph-osd-classic', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2026-02-25T12:00:00.281 INFO:teuthology.task.install:extra packages: [] 2026-02-25T12:00:00.281 DEBUG:teuthology.orchestra.run.trial117:> sudo apt-key list | grep Ceph 2026-02-25T12:00:00.345 INFO:teuthology.orchestra.run.trial117.stderr:Warning: apt-key is deprecated. Manage keyring files in trusted.gpg.d instead (see apt-key(8)). 2026-02-25T12:00:00.374 INFO:teuthology.orchestra.run.trial117.stdout:uid [ unknown] Ceph automated package build (Ceph automated package build) 2026-02-25T12:00:00.375 INFO:teuthology.orchestra.run.trial117.stdout:uid [ unknown] Ceph.com (release key) 2026-02-25T12:00:00.375 INFO:teuthology.task.install.deb:Installing packages: ceph, cephadm, ceph-mds, ceph-mgr, ceph-osd, ceph-osd-classic, ceph-common, ceph-fuse, ceph-test, ceph-volume, radosgw, python3-rados, python3-rgw, python3-cephfs, python3-rbd, libcephfs2, libcephfs-dev, librados2, librbd1, rbd-fuse on remote deb x86_64 2026-02-25T12:00:00.376 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&sha1=506b402a3d3a240d0bfe073b4769eaffdf84497b 2026-02-25T12:00:00.543 INFO:teuthology.task.install.deb:Pulling from https://1.chacra.ceph.com/r/ceph/wip-rocky10-branch-of-the-day-2026-02-24-1771941190/506b402a3d3a240d0bfe073b4769eaffdf84497b/ubuntu/jammy/flavors/default/ 2026-02-25T12:00:00.544 INFO:teuthology.task.install.deb:Package version is 20.3.0-5580-g506b402a-1jammy 2026-02-25T12:00:00.714 DEBUG:teuthology.orchestra.run.trial117:> set -ex 2026-02-25T12:00:00.714 DEBUG:teuthology.orchestra.run.trial117:> sudo dd of=/etc/apt/sources.list.d/ceph.list 2026-02-25T12:00:00.723 DEBUG:teuthology.orchestra.run.trial117:> sudo apt-get update 2026-02-25T12:00:00.874 INFO:teuthology.orchestra.run.trial117.stdout:Ign:1 https://1.chacra.ceph.com/r/ceph/wip-rocky10-branch-of-the-day-2026-02-24-1771941190/506b402a3d3a240d0bfe073b4769eaffdf84497b/ubuntu/jammy/flavors/default jammy InRelease 2026-02-25T12:00:00.886 INFO:teuthology.orchestra.run.trial117.stdout:Get:2 https://1.chacra.ceph.com/r/ceph/wip-rocky10-branch-of-the-day-2026-02-24-1771941190/506b402a3d3a240d0bfe073b4769eaffdf84497b/ubuntu/jammy/flavors/default jammy Release [7689 B] 2026-02-25T12:00:00.897 INFO:teuthology.orchestra.run.trial117.stdout:Ign:3 https://1.chacra.ceph.com/r/ceph/wip-rocky10-branch-of-the-day-2026-02-24-1771941190/506b402a3d3a240d0bfe073b4769eaffdf84497b/ubuntu/jammy/flavors/default jammy Release.gpg 2026-02-25T12:00:00.909 INFO:teuthology.orchestra.run.trial117.stdout:Get:4 https://1.chacra.ceph.com/r/ceph/wip-rocky10-branch-of-the-day-2026-02-24-1771941190/506b402a3d3a240d0bfe073b4769eaffdf84497b/ubuntu/jammy/flavors/default jammy/main amd64 Packages [19.2 kB] 2026-02-25T12:00:01.297 INFO:teuthology.orchestra.run.trial117.stdout:Hit:5 https://archive.ubuntu.com/ubuntu jammy InRelease 2026-02-25T12:00:01.378 INFO:teuthology.orchestra.run.trial117.stdout:Hit:6 https://archive.ubuntu.com/ubuntu jammy-updates InRelease 2026-02-25T12:00:01.466 INFO:teuthology.orchestra.run.trial117.stdout:Hit:7 https://archive.ubuntu.com/ubuntu jammy-security InRelease 2026-02-25T12:00:01.555 INFO:teuthology.orchestra.run.trial117.stdout:Hit:8 https://archive.ubuntu.com/ubuntu jammy-backports InRelease 2026-02-25T12:00:01.712 INFO:teuthology.orchestra.run.trial117.stdout:Fetched 26.9 kB in 1s (32.8 kB/s) 2026-02-25T12:00:02.239 INFO:teuthology.orchestra.run.trial117.stdout:Reading package lists... 2026-02-25T12:00:02.252 DEBUG:teuthology.orchestra.run.trial117:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" install ceph=20.3.0-5580-g506b402a-1jammy cephadm=20.3.0-5580-g506b402a-1jammy ceph-mds=20.3.0-5580-g506b402a-1jammy ceph-mgr=20.3.0-5580-g506b402a-1jammy ceph-osd=20.3.0-5580-g506b402a-1jammy ceph-osd-classic=20.3.0-5580-g506b402a-1jammy ceph-common=20.3.0-5580-g506b402a-1jammy ceph-fuse=20.3.0-5580-g506b402a-1jammy ceph-test=20.3.0-5580-g506b402a-1jammy ceph-volume=20.3.0-5580-g506b402a-1jammy radosgw=20.3.0-5580-g506b402a-1jammy python3-rados=20.3.0-5580-g506b402a-1jammy python3-rgw=20.3.0-5580-g506b402a-1jammy python3-cephfs=20.3.0-5580-g506b402a-1jammy python3-rbd=20.3.0-5580-g506b402a-1jammy libcephfs2=20.3.0-5580-g506b402a-1jammy libcephfs-dev=20.3.0-5580-g506b402a-1jammy librados2=20.3.0-5580-g506b402a-1jammy librbd1=20.3.0-5580-g506b402a-1jammy rbd-fuse=20.3.0-5580-g506b402a-1jammy 2026-02-25T12:00:02.286 INFO:teuthology.orchestra.run.trial117.stdout:Reading package lists... 2026-02-25T12:00:02.391 INFO:teuthology.orchestra.run.trial117.stdout:Building dependency tree... 2026-02-25T12:00:02.391 INFO:teuthology.orchestra.run.trial117.stdout:Reading state information... 2026-02-25T12:00:02.479 INFO:teuthology.orchestra.run.trial117.stdout:The following packages were automatically installed and are no longer required: 2026-02-25T12:00:02.479 INFO:teuthology.orchestra.run.trial117.stdout: kpartx libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 2026-02-25T12:00:02.480 INFO:teuthology.orchestra.run.trial117.stdout: libgfxdr0 libglusterfs0 libiscsi7 libpmemobj1 libsgutils2-2 2026-02-25T12:00:02.480 INFO:teuthology.orchestra.run.trial117.stdout: linux-headers-5.15.0-164 linux-headers-5.15.0-164-generic 2026-02-25T12:00:02.480 INFO:teuthology.orchestra.run.trial117.stdout: linux-image-5.15.0-164-generic linux-modules-5.15.0-164-generic 2026-02-25T12:00:02.480 INFO:teuthology.orchestra.run.trial117.stdout: linux-modules-extra-5.15.0-164-generic sg3-utils sg3-utils-udev 2026-02-25T12:00:02.480 INFO:teuthology.orchestra.run.trial117.stdout:Use 'sudo apt autoremove' to remove them. 2026-02-25T12:00:02.480 INFO:teuthology.orchestra.run.trial117.stdout:The following additional packages will be installed: 2026-02-25T12:00:02.480 INFO:teuthology.orchestra.run.trial117.stdout: ceph-base ceph-mgr-cephadm ceph-mgr-dashboard ceph-mgr-diskprediction-local 2026-02-25T12:00:02.480 INFO:teuthology.orchestra.run.trial117.stdout: ceph-mgr-k8sevents ceph-mgr-modules-core ceph-mon jq libcephfs-daemon 2026-02-25T12:00:02.480 INFO:teuthology.orchestra.run.trial117.stdout: libcephfs-proxy2 libdouble-conversion3 libfuse2 libjq1 liblttng-ust1 libnbd0 2026-02-25T12:00:02.480 INFO:teuthology.orchestra.run.trial117.stdout: liboath0 libonig5 libpcre2-16-0 libqt5core5a libqt5dbus5 libqt5network5 2026-02-25T12:00:02.480 INFO:teuthology.orchestra.run.trial117.stdout: libradosstriper1 librdkafka1 librgw2 libsqlite3-mod-ceph libthrift-0.16.0 2026-02-25T12:00:02.481 INFO:teuthology.orchestra.run.trial117.stdout: nvme-cli python-asyncssh-doc python3-asyncssh python3-cachetools 2026-02-25T12:00:02.481 INFO:teuthology.orchestra.run.trial117.stdout: python3-ceph-argparse python3-ceph-common python3-cheroot python3-cherrypy3 2026-02-25T12:00:02.481 INFO:teuthology.orchestra.run.trial117.stdout: python3-google-auth python3-iniconfig python3-jaraco.classes 2026-02-25T12:00:02.481 INFO:teuthology.orchestra.run.trial117.stdout: python3-jaraco.collections python3-jaraco.functools python3-jaraco.text 2026-02-25T12:00:02.481 INFO:teuthology.orchestra.run.trial117.stdout: python3-joblib python3-kubernetes python3-natsort python3-pluggy 2026-02-25T12:00:02.481 INFO:teuthology.orchestra.run.trial117.stdout: python3-portend python3-prettytable python3-psutil python3-py 2026-02-25T12:00:02.481 INFO:teuthology.orchestra.run.trial117.stdout: python3-pygments python3-pytest python3-repoze.lru python3-requests-oauthlib 2026-02-25T12:00:02.481 INFO:teuthology.orchestra.run.trial117.stdout: python3-routes python3-rsa python3-simplejson python3-sklearn 2026-02-25T12:00:02.481 INFO:teuthology.orchestra.run.trial117.stdout: python3-sklearn-lib python3-tempora python3-threadpoolctl python3-toml 2026-02-25T12:00:02.481 INFO:teuthology.orchestra.run.trial117.stdout: python3-wcwidth python3-webob python3-websocket python3-zc.lockfile 2026-02-25T12:00:02.481 INFO:teuthology.orchestra.run.trial117.stdout: qttranslations5-l10n socat xmlstarlet 2026-02-25T12:00:02.482 INFO:teuthology.orchestra.run.trial117.stdout:Suggested packages: 2026-02-25T12:00:02.482 INFO:teuthology.orchestra.run.trial117.stdout: python3-influxdb liblua5.3-dev luarocks python-natsort-doc python-psutil-doc 2026-02-25T12:00:02.482 INFO:teuthology.orchestra.run.trial117.stdout: subversion python-pygments-doc ttf-bitstream-vera python3-paste python3-dap 2026-02-25T12:00:02.482 INFO:teuthology.orchestra.run.trial117.stdout: python-sklearn-doc ipython3 python-webob-doc 2026-02-25T12:00:02.482 INFO:teuthology.orchestra.run.trial117.stdout:Recommended packages: 2026-02-25T12:00:02.482 INFO:teuthology.orchestra.run.trial117.stdout: btrfs-tools 2026-02-25T12:00:02.513 INFO:teuthology.orchestra.run.trial117.stdout:The following NEW packages will be installed: 2026-02-25T12:00:02.514 INFO:teuthology.orchestra.run.trial117.stdout: ceph ceph-base ceph-common ceph-fuse ceph-mds ceph-mgr ceph-mgr-cephadm 2026-02-25T12:00:02.514 INFO:teuthology.orchestra.run.trial117.stdout: ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-k8sevents 2026-02-25T12:00:02.514 INFO:teuthology.orchestra.run.trial117.stdout: ceph-mgr-modules-core ceph-mon ceph-osd ceph-osd-classic ceph-test 2026-02-25T12:00:02.514 INFO:teuthology.orchestra.run.trial117.stdout: ceph-volume cephadm jq libcephfs-daemon libcephfs-dev libcephfs-proxy2 2026-02-25T12:00:02.514 INFO:teuthology.orchestra.run.trial117.stdout: libcephfs2 libdouble-conversion3 libfuse2 libjq1 liblttng-ust1 libnbd0 2026-02-25T12:00:02.514 INFO:teuthology.orchestra.run.trial117.stdout: liboath0 libonig5 libpcre2-16-0 libqt5core5a libqt5dbus5 libqt5network5 2026-02-25T12:00:02.514 INFO:teuthology.orchestra.run.trial117.stdout: librados2 libradosstriper1 librbd1 librdkafka1 librgw2 libsqlite3-mod-ceph 2026-02-25T12:00:02.514 INFO:teuthology.orchestra.run.trial117.stdout: libthrift-0.16.0 nvme-cli python-asyncssh-doc python3-asyncssh 2026-02-25T12:00:02.515 INFO:teuthology.orchestra.run.trial117.stdout: python3-cachetools python3-ceph-argparse python3-ceph-common python3-cephfs 2026-02-25T12:00:02.515 INFO:teuthology.orchestra.run.trial117.stdout: python3-cheroot python3-cherrypy3 python3-google-auth python3-iniconfig 2026-02-25T12:00:02.515 INFO:teuthology.orchestra.run.trial117.stdout: python3-jaraco.classes python3-jaraco.collections python3-jaraco.functools 2026-02-25T12:00:02.515 INFO:teuthology.orchestra.run.trial117.stdout: python3-jaraco.text python3-joblib python3-kubernetes python3-natsort 2026-02-25T12:00:02.515 INFO:teuthology.orchestra.run.trial117.stdout: python3-pluggy python3-portend python3-prettytable python3-psutil python3-py 2026-02-25T12:00:02.515 INFO:teuthology.orchestra.run.trial117.stdout: python3-pygments python3-pytest python3-rados python3-rbd python3-repoze.lru 2026-02-25T12:00:02.515 INFO:teuthology.orchestra.run.trial117.stdout: python3-requests-oauthlib python3-rgw python3-routes python3-rsa 2026-02-25T12:00:02.515 INFO:teuthology.orchestra.run.trial117.stdout: python3-simplejson python3-sklearn python3-sklearn-lib python3-tempora 2026-02-25T12:00:02.515 INFO:teuthology.orchestra.run.trial117.stdout: python3-threadpoolctl python3-toml python3-wcwidth python3-webob 2026-02-25T12:00:02.515 INFO:teuthology.orchestra.run.trial117.stdout: python3-websocket python3-zc.lockfile qttranslations5-l10n radosgw rbd-fuse 2026-02-25T12:00:02.516 INFO:teuthology.orchestra.run.trial117.stdout: socat xmlstarlet 2026-02-25T12:00:02.591 INFO:teuthology.orchestra.run.trial117.stdout:0 upgraded, 87 newly installed, 0 to remove and 29 not upgraded. 2026-02-25T12:00:02.591 INFO:teuthology.orchestra.run.trial117.stdout:Need to get 693 MB of archives. 2026-02-25T12:00:02.592 INFO:teuthology.orchestra.run.trial117.stdout:After this operation, 2583 MB of additional disk space will be used. 2026-02-25T12:00:02.592 INFO:teuthology.orchestra.run.trial117.stdout:Get:1 https://1.chacra.ceph.com/r/ceph/wip-rocky10-branch-of-the-day-2026-02-24-1771941190/506b402a3d3a240d0bfe073b4769eaffdf84497b/ubuntu/jammy/flavors/default jammy/main amd64 librados2 amd64 20.3.0-5580-g506b402a-1jammy [3794 kB] 2026-02-25T12:00:02.611 INFO:teuthology.orchestra.run.trial117.stdout:Get:2 https://archive.ubuntu.com/ubuntu jammy/main amd64 liblttng-ust1 amd64 2.13.1-1ubuntu1 [190 kB] 2026-02-25T12:00:02.653 INFO:teuthology.orchestra.run.trial117.stdout:Get:3 https://archive.ubuntu.com/ubuntu jammy/universe amd64 libdouble-conversion3 amd64 3.1.7-4 [39.0 kB] 2026-02-25T12:00:02.654 INFO:teuthology.orchestra.run.trial117.stdout:Get:4 https://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libpcre2-16-0 amd64 10.39-3ubuntu0.1 [203 kB] 2026-02-25T12:00:02.662 INFO:teuthology.orchestra.run.trial117.stdout:Get:5 https://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5core5a amd64 5.15.3+dfsg-2ubuntu0.2 [2006 kB] 2026-02-25T12:00:02.688 INFO:teuthology.orchestra.run.trial117.stdout:Get:6 https://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5dbus5 amd64 5.15.3+dfsg-2ubuntu0.2 [222 kB] 2026-02-25T12:00:02.690 INFO:teuthology.orchestra.run.trial117.stdout:Get:7 https://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5network5 amd64 5.15.3+dfsg-2ubuntu0.2 [731 kB] 2026-02-25T12:00:02.695 INFO:teuthology.orchestra.run.trial117.stdout:Get:8 https://archive.ubuntu.com/ubuntu jammy/universe amd64 libthrift-0.16.0 amd64 0.16.0-2 [267 kB] 2026-02-25T12:00:02.697 INFO:teuthology.orchestra.run.trial117.stdout:Get:9 https://1.chacra.ceph.com/r/ceph/wip-rocky10-branch-of-the-day-2026-02-24-1771941190/506b402a3d3a240d0bfe073b4769eaffdf84497b/ubuntu/jammy/flavors/default jammy/main amd64 librbd1 amd64 20.3.0-5580-g506b402a-1jammy [2912 kB] 2026-02-25T12:00:02.697 INFO:teuthology.orchestra.run.trial117.stdout:Get:10 https://archive.ubuntu.com/ubuntu jammy/universe amd64 libnbd0 amd64 1.10.5-1 [71.3 kB] 2026-02-25T12:00:02.697 INFO:teuthology.orchestra.run.trial117.stdout:Get:11 https://archive.ubuntu.com/ubuntu jammy/main amd64 python3-wcwidth all 0.2.5+dfsg1-1 [21.9 kB] 2026-02-25T12:00:02.698 INFO:teuthology.orchestra.run.trial117.stdout:Get:12 https://archive.ubuntu.com/ubuntu jammy/main amd64 python3-prettytable all 2.5.0-2 [31.3 kB] 2026-02-25T12:00:02.698 INFO:teuthology.orchestra.run.trial117.stdout:Get:13 https://archive.ubuntu.com/ubuntu jammy/universe amd64 librdkafka1 amd64 1.8.0-1build1 [633 kB] 2026-02-25T12:00:02.702 INFO:teuthology.orchestra.run.trial117.stdout:Get:14 https://archive.ubuntu.com/ubuntu jammy-updates/main amd64 liboath0 amd64 2.6.7-3ubuntu0.1 [41.3 kB] 2026-02-25T12:00:02.703 INFO:teuthology.orchestra.run.trial117.stdout:Get:15 https://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.functools all 3.4.0-2 [9030 B] 2026-02-25T12:00:02.703 INFO:teuthology.orchestra.run.trial117.stdout:Get:16 https://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-cheroot all 8.5.2+ds1-1ubuntu3.1 [71.1 kB] 2026-02-25T12:00:02.708 INFO:teuthology.orchestra.run.trial117.stdout:Get:17 https://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.classes all 3.2.1-3 [6452 B] 2026-02-25T12:00:02.708 INFO:teuthology.orchestra.run.trial117.stdout:Get:18 https://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.text all 3.6.0-2 [8716 B] 2026-02-25T12:00:02.708 INFO:teuthology.orchestra.run.trial117.stdout:Get:19 https://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.collections all 3.4.0-2 [11.4 kB] 2026-02-25T12:00:02.708 INFO:teuthology.orchestra.run.trial117.stdout:Get:20 https://archive.ubuntu.com/ubuntu jammy/main amd64 python3-tempora all 4.1.2-1 [14.8 kB] 2026-02-25T12:00:02.717 INFO:teuthology.orchestra.run.trial117.stdout:Get:21 https://archive.ubuntu.com/ubuntu jammy/main amd64 python3-portend all 3.0.0-1 [7240 B] 2026-02-25T12:00:02.717 INFO:teuthology.orchestra.run.trial117.stdout:Get:22 https://archive.ubuntu.com/ubuntu jammy/main amd64 python3-zc.lockfile all 2.0-1 [8980 B] 2026-02-25T12:00:02.717 INFO:teuthology.orchestra.run.trial117.stdout:Get:23 https://archive.ubuntu.com/ubuntu jammy/main amd64 python3-cherrypy3 all 18.6.1-4 [208 kB] 2026-02-25T12:00:02.718 INFO:teuthology.orchestra.run.trial117.stdout:Get:24 https://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-natsort all 8.0.2-1 [35.3 kB] 2026-02-25T12:00:02.719 INFO:teuthology.orchestra.run.trial117.stdout:Get:25 https://archive.ubuntu.com/ubuntu jammy/universe amd64 libfuse2 amd64 2.9.9-5ubuntu3 [90.3 kB] 2026-02-25T12:00:02.720 INFO:teuthology.orchestra.run.trial117.stdout:Get:26 https://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 python3-asyncssh all 2.5.0-1ubuntu0.1 [189 kB] 2026-02-25T12:00:02.725 INFO:teuthology.orchestra.run.trial117.stdout:Get:27 https://archive.ubuntu.com/ubuntu jammy/main amd64 python3-repoze.lru all 0.7-2 [12.1 kB] 2026-02-25T12:00:02.726 INFO:teuthology.orchestra.run.trial117.stdout:Get:28 https://archive.ubuntu.com/ubuntu jammy/main amd64 python3-routes all 2.5.1-1ubuntu1 [89.0 kB] 2026-02-25T12:00:02.727 INFO:teuthology.orchestra.run.trial117.stdout:Get:29 https://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-sklearn-lib amd64 0.23.2-5ubuntu6 [2058 kB] 2026-02-25T12:00:02.736 INFO:teuthology.orchestra.run.trial117.stdout:Get:30 https://1.chacra.ceph.com/r/ceph/wip-rocky10-branch-of-the-day-2026-02-24-1771941190/506b402a3d3a240d0bfe073b4769eaffdf84497b/ubuntu/jammy/flavors/default jammy/main amd64 libcephfs2 amd64 20.3.0-5580-g506b402a-1jammy [882 kB] 2026-02-25T12:00:02.741 INFO:teuthology.orchestra.run.trial117.stdout:Get:31 https://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-joblib all 0.17.0-4ubuntu1 [204 kB] 2026-02-25T12:00:02.742 INFO:teuthology.orchestra.run.trial117.stdout:Get:32 https://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-threadpoolctl all 3.1.0-1 [21.3 kB] 2026-02-25T12:00:02.743 INFO:teuthology.orchestra.run.trial117.stdout:Get:33 https://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-sklearn all 0.23.2-5ubuntu6 [1829 kB] 2026-02-25T12:00:02.750 INFO:teuthology.orchestra.run.trial117.stdout:Get:34 https://1.chacra.ceph.com/r/ceph/wip-rocky10-branch-of-the-day-2026-02-24-1771941190/506b402a3d3a240d0bfe073b4769eaffdf84497b/ubuntu/jammy/flavors/default jammy/main amd64 python3-rados amd64 20.3.0-5580-g506b402a-1jammy [359 kB] 2026-02-25T12:00:02.753 INFO:teuthology.orchestra.run.trial117.stdout:Get:35 https://archive.ubuntu.com/ubuntu jammy/main amd64 python3-cachetools all 5.0.0-1 [9722 B] 2026-02-25T12:00:02.753 INFO:teuthology.orchestra.run.trial117.stdout:Get:36 https://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-rsa all 4.8-1 [28.4 kB] 2026-02-25T12:00:02.753 INFO:teuthology.orchestra.run.trial117.stdout:Get:37 https://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-google-auth all 1.5.1-3 [35.7 kB] 2026-02-25T12:00:02.754 INFO:teuthology.orchestra.run.trial117.stdout:Get:38 https://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-requests-oauthlib all 1.3.0+ds-0.1 [18.7 kB] 2026-02-25T12:00:02.754 INFO:teuthology.orchestra.run.trial117.stdout:Get:39 https://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-websocket all 1.2.3-1 [34.7 kB] 2026-02-25T12:00:02.754 INFO:teuthology.orchestra.run.trial117.stdout:Get:40 https://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-kubernetes all 12.0.1-1ubuntu1 [353 kB] 2026-02-25T12:00:02.755 INFO:teuthology.orchestra.run.trial117.stdout:Get:41 https://archive.ubuntu.com/ubuntu jammy/main amd64 libonig5 amd64 6.9.7.1-2build1 [172 kB] 2026-02-25T12:00:02.756 INFO:teuthology.orchestra.run.trial117.stdout:Get:42 https://1.chacra.ceph.com/r/ceph/wip-rocky10-branch-of-the-day-2026-02-24-1771941190/506b402a3d3a240d0bfe073b4769eaffdf84497b/ubuntu/jammy/flavors/default jammy/main amd64 python3-ceph-argparse all 20.3.0-5580-g506b402a-1jammy [32.8 kB] 2026-02-25T12:00:02.756 INFO:teuthology.orchestra.run.trial117.stdout:Get:43 https://1.chacra.ceph.com/r/ceph/wip-rocky10-branch-of-the-day-2026-02-24-1771941190/506b402a3d3a240d0bfe073b4769eaffdf84497b/ubuntu/jammy/flavors/default jammy/main amd64 python3-cephfs amd64 20.3.0-5580-g506b402a-1jammy [225 kB] 2026-02-25T12:00:02.760 INFO:teuthology.orchestra.run.trial117.stdout:Get:44 https://1.chacra.ceph.com/r/ceph/wip-rocky10-branch-of-the-day-2026-02-24-1771941190/506b402a3d3a240d0bfe073b4769eaffdf84497b/ubuntu/jammy/flavors/default jammy/main amd64 python3-ceph-common all 20.3.0-5580-g506b402a-1jammy [89.0 kB] 2026-02-25T12:00:02.761 INFO:teuthology.orchestra.run.trial117.stdout:Get:45 https://1.chacra.ceph.com/r/ceph/wip-rocky10-branch-of-the-day-2026-02-24-1771941190/506b402a3d3a240d0bfe073b4769eaffdf84497b/ubuntu/jammy/flavors/default jammy/main amd64 python3-rbd amd64 20.3.0-5580-g506b402a-1jammy [339 kB] 2026-02-25T12:00:02.765 INFO:teuthology.orchestra.run.trial117.stdout:Get:46 https://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libjq1 amd64 1.6-2.1ubuntu3.1 [133 kB] 2026-02-25T12:00:02.766 INFO:teuthology.orchestra.run.trial117.stdout:Get:47 https://1.chacra.ceph.com/r/ceph/wip-rocky10-branch-of-the-day-2026-02-24-1771941190/506b402a3d3a240d0bfe073b4769eaffdf84497b/ubuntu/jammy/flavors/default jammy/main amd64 librgw2 amd64 20.3.0-5580-g506b402a-1jammy [9132 kB] 2026-02-25T12:00:02.768 INFO:teuthology.orchestra.run.trial117.stdout:Get:48 https://archive.ubuntu.com/ubuntu jammy-updates/main amd64 jq amd64 1.6-2.1ubuntu3.1 [52.5 kB] 2026-02-25T12:00:02.769 INFO:teuthology.orchestra.run.trial117.stdout:Get:49 https://archive.ubuntu.com/ubuntu jammy/main amd64 socat amd64 1.7.4.1-3ubuntu4 [349 kB] 2026-02-25T12:00:02.775 INFO:teuthology.orchestra.run.trial117.stdout:Get:50 https://archive.ubuntu.com/ubuntu jammy/universe amd64 xmlstarlet amd64 1.6.1-2.1 [265 kB] 2026-02-25T12:00:02.780 INFO:teuthology.orchestra.run.trial117.stdout:Get:51 https://archive.ubuntu.com/ubuntu jammy-updates/main amd64 nvme-cli amd64 1.16-3ubuntu0.3 [474 kB] 2026-02-25T12:00:02.785 INFO:teuthology.orchestra.run.trial117.stdout:Get:52 https://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 python-asyncssh-doc all 2.5.0-1ubuntu0.1 [309 kB] 2026-02-25T12:00:02.790 INFO:teuthology.orchestra.run.trial117.stdout:Get:53 https://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-iniconfig all 1.1.1-2 [6024 B] 2026-02-25T12:00:02.790 INFO:teuthology.orchestra.run.trial117.stdout:Get:54 https://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-pluggy all 0.13.0-7.1 [19.0 kB] 2026-02-25T12:00:02.790 INFO:teuthology.orchestra.run.trial117.stdout:Get:55 https://archive.ubuntu.com/ubuntu jammy/main amd64 python3-psutil amd64 5.9.0-1build1 [158 kB] 2026-02-25T12:00:02.794 INFO:teuthology.orchestra.run.trial117.stdout:Get:56 https://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-py all 1.10.0-1 [71.9 kB] 2026-02-25T12:00:02.796 INFO:teuthology.orchestra.run.trial117.stdout:Get:57 https://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-pygments all 2.11.2+dfsg-2ubuntu0.1 [750 kB] 2026-02-25T12:00:02.808 INFO:teuthology.orchestra.run.trial117.stdout:Get:58 https://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-toml all 0.10.2-1 [16.5 kB] 2026-02-25T12:00:02.808 INFO:teuthology.orchestra.run.trial117.stdout:Get:59 https://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-pytest all 6.2.5-1ubuntu2 [214 kB] 2026-02-25T12:00:02.817 INFO:teuthology.orchestra.run.trial117.stdout:Get:60 https://archive.ubuntu.com/ubuntu jammy/main amd64 python3-simplejson amd64 3.17.6-1build1 [54.7 kB] 2026-02-25T12:00:02.820 INFO:teuthology.orchestra.run.trial117.stdout:Get:61 https://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-webob all 1:1.8.6-1.1ubuntu0.1 [86.7 kB] 2026-02-25T12:00:02.821 INFO:teuthology.orchestra.run.trial117.stdout:Get:62 https://archive.ubuntu.com/ubuntu jammy/universe amd64 qttranslations5-l10n all 5.15.3-1 [1983 kB] 2026-02-25T12:00:02.918 INFO:teuthology.orchestra.run.trial117.stdout:Get:63 https://1.chacra.ceph.com/r/ceph/wip-rocky10-branch-of-the-day-2026-02-24-1771941190/506b402a3d3a240d0bfe073b4769eaffdf84497b/ubuntu/jammy/flavors/default jammy/main amd64 python3-rgw amd64 20.3.0-5580-g506b402a-1jammy [112 kB] 2026-02-25T12:00:02.919 INFO:teuthology.orchestra.run.trial117.stdout:Get:64 https://1.chacra.ceph.com/r/ceph/wip-rocky10-branch-of-the-day-2026-02-24-1771941190/506b402a3d3a240d0bfe073b4769eaffdf84497b/ubuntu/jammy/flavors/default jammy/main amd64 libradosstriper1 amd64 20.3.0-5580-g506b402a-1jammy [261 kB] 2026-02-25T12:00:02.923 INFO:teuthology.orchestra.run.trial117.stdout:Get:65 https://1.chacra.ceph.com/r/ceph/wip-rocky10-branch-of-the-day-2026-02-24-1771941190/506b402a3d3a240d0bfe073b4769eaffdf84497b/ubuntu/jammy/flavors/default jammy/main amd64 ceph-common amd64 20.3.0-5580-g506b402a-1jammy [31.6 MB] 2026-02-25T12:00:03.445 INFO:teuthology.orchestra.run.trial117.stdout:Get:66 https://1.chacra.ceph.com/r/ceph/wip-rocky10-branch-of-the-day-2026-02-24-1771941190/506b402a3d3a240d0bfe073b4769eaffdf84497b/ubuntu/jammy/flavors/default jammy/main amd64 ceph-base amd64 20.3.0-5580-g506b402a-1jammy [4984 kB] 2026-02-25T12:00:03.523 INFO:teuthology.orchestra.run.trial117.stdout:Get:67 https://1.chacra.ceph.com/r/ceph/wip-rocky10-branch-of-the-day-2026-02-24-1771941190/506b402a3d3a240d0bfe073b4769eaffdf84497b/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-modules-core all 20.3.0-5580-g506b402a-1jammy [297 kB] 2026-02-25T12:00:03.526 INFO:teuthology.orchestra.run.trial117.stdout:Get:68 https://1.chacra.ceph.com/r/ceph/wip-rocky10-branch-of-the-day-2026-02-24-1771941190/506b402a3d3a240d0bfe073b4769eaffdf84497b/ubuntu/jammy/flavors/default jammy/main amd64 libsqlite3-mod-ceph amd64 20.3.0-5580-g506b402a-1jammy [127 kB] 2026-02-25T12:00:03.528 INFO:teuthology.orchestra.run.trial117.stdout:Get:69 https://1.chacra.ceph.com/r/ceph/wip-rocky10-branch-of-the-day-2026-02-24-1771941190/506b402a3d3a240d0bfe073b4769eaffdf84497b/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr amd64 20.3.0-5580-g506b402a-1jammy [939 kB] 2026-02-25T12:00:03.544 INFO:teuthology.orchestra.run.trial117.stdout:Get:70 https://1.chacra.ceph.com/r/ceph/wip-rocky10-branch-of-the-day-2026-02-24-1771941190/506b402a3d3a240d0bfe073b4769eaffdf84497b/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mon amd64 20.3.0-5580-g506b402a-1jammy [7540 kB] 2026-02-25T12:00:03.669 INFO:teuthology.orchestra.run.trial117.stdout:Get:71 https://1.chacra.ceph.com/r/ceph/wip-rocky10-branch-of-the-day-2026-02-24-1771941190/506b402a3d3a240d0bfe073b4769eaffdf84497b/ubuntu/jammy/flavors/default jammy/main amd64 ceph-osd-classic amd64 20.3.0-5580-g506b402a-1jammy [468 MB] 2026-02-25T12:00:13.421 INFO:teuthology.orchestra.run.trial117.stdout:Get:72 https://1.chacra.ceph.com/r/ceph/wip-rocky10-branch-of-the-day-2026-02-24-1771941190/506b402a3d3a240d0bfe073b4769eaffdf84497b/ubuntu/jammy/flavors/default jammy/main amd64 ceph-osd amd64 20.3.0-5580-g506b402a-1jammy [180 kB] 2026-02-25T12:00:13.422 INFO:teuthology.orchestra.run.trial117.stdout:Get:73 https://1.chacra.ceph.com/r/ceph/wip-rocky10-branch-of-the-day-2026-02-24-1771941190/506b402a3d3a240d0bfe073b4769eaffdf84497b/ubuntu/jammy/flavors/default jammy/main amd64 ceph amd64 20.3.0-5580-g506b402a-1jammy [14.0 kB] 2026-02-25T12:00:13.422 INFO:teuthology.orchestra.run.trial117.stdout:Get:74 https://1.chacra.ceph.com/r/ceph/wip-rocky10-branch-of-the-day-2026-02-24-1771941190/506b402a3d3a240d0bfe073b4769eaffdf84497b/ubuntu/jammy/flavors/default jammy/main amd64 ceph-fuse amd64 20.3.0-5580-g506b402a-1jammy [1030 kB] 2026-02-25T12:00:13.428 INFO:teuthology.orchestra.run.trial117.stdout:Get:75 https://1.chacra.ceph.com/r/ceph/wip-rocky10-branch-of-the-day-2026-02-24-1771941190/506b402a3d3a240d0bfe073b4769eaffdf84497b/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mds amd64 20.3.0-5580-g506b402a-1jammy [2437 kB] 2026-02-25T12:00:13.440 INFO:teuthology.orchestra.run.trial117.stdout:Get:76 https://1.chacra.ceph.com/r/ceph/wip-rocky10-branch-of-the-day-2026-02-24-1771941190/506b402a3d3a240d0bfe073b4769eaffdf84497b/ubuntu/jammy/flavors/default jammy/main amd64 cephadm amd64 20.3.0-5580-g506b402a-1jammy [1072 kB] 2026-02-25T12:00:13.465 INFO:teuthology.orchestra.run.trial117.stdout:Get:77 https://1.chacra.ceph.com/r/ceph/wip-rocky10-branch-of-the-day-2026-02-24-1771941190/506b402a3d3a240d0bfe073b4769eaffdf84497b/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-cephadm all 20.3.0-5580-g506b402a-1jammy [191 kB] 2026-02-25T12:00:13.468 INFO:teuthology.orchestra.run.trial117.stdout:Get:78 https://1.chacra.ceph.com/r/ceph/wip-rocky10-branch-of-the-day-2026-02-24-1771941190/506b402a3d3a240d0bfe073b4769eaffdf84497b/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-dashboard all 20.3.0-5580-g506b402a-1jammy [49.3 MB] 2026-02-25T12:00:15.057 INFO:teuthology.orchestra.run.trial117.stdout:Get:79 https://1.chacra.ceph.com/r/ceph/wip-rocky10-branch-of-the-day-2026-02-24-1771941190/506b402a3d3a240d0bfe073b4769eaffdf84497b/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-diskprediction-local all 20.3.0-5580-g506b402a-1jammy [8626 kB] 2026-02-25T12:00:15.341 INFO:teuthology.orchestra.run.trial117.stdout:Get:80 https://1.chacra.ceph.com/r/ceph/wip-rocky10-branch-of-the-day-2026-02-24-1771941190/506b402a3d3a240d0bfe073b4769eaffdf84497b/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-k8sevents all 20.3.0-5580-g506b402a-1jammy [14.2 kB] 2026-02-25T12:00:15.342 INFO:teuthology.orchestra.run.trial117.stdout:Get:81 https://1.chacra.ceph.com/r/ceph/wip-rocky10-branch-of-the-day-2026-02-24-1771941190/506b402a3d3a240d0bfe073b4769eaffdf84497b/ubuntu/jammy/flavors/default jammy/main amd64 ceph-test amd64 20.3.0-5580-g506b402a-1jammy [65.6 MB] 2026-02-25T12:00:17.479 INFO:teuthology.orchestra.run.trial117.stdout:Get:82 https://1.chacra.ceph.com/r/ceph/wip-rocky10-branch-of-the-day-2026-02-24-1771941190/506b402a3d3a240d0bfe073b4769eaffdf84497b/ubuntu/jammy/flavors/default jammy/main amd64 ceph-volume all 20.3.0-5580-g506b402a-1jammy [136 kB] 2026-02-25T12:00:17.480 INFO:teuthology.orchestra.run.trial117.stdout:Get:83 https://1.chacra.ceph.com/r/ceph/wip-rocky10-branch-of-the-day-2026-02-24-1771941190/506b402a3d3a240d0bfe073b4769eaffdf84497b/ubuntu/jammy/flavors/default jammy/main amd64 libcephfs-daemon amd64 20.3.0-5580-g506b402a-1jammy [47.3 kB] 2026-02-25T12:00:17.480 INFO:teuthology.orchestra.run.trial117.stdout:Get:84 https://1.chacra.ceph.com/r/ceph/wip-rocky10-branch-of-the-day-2026-02-24-1771941190/506b402a3d3a240d0bfe073b4769eaffdf84497b/ubuntu/jammy/flavors/default jammy/main amd64 libcephfs-proxy2 amd64 20.3.0-5580-g506b402a-1jammy [33.1 kB] 2026-02-25T12:00:17.481 INFO:teuthology.orchestra.run.trial117.stdout:Get:85 https://1.chacra.ceph.com/r/ceph/wip-rocky10-branch-of-the-day-2026-02-24-1771941190/506b402a3d3a240d0bfe073b4769eaffdf84497b/ubuntu/jammy/flavors/default jammy/main amd64 libcephfs-dev amd64 20.3.0-5580-g506b402a-1jammy [44.2 kB] 2026-02-25T12:00:17.481 INFO:teuthology.orchestra.run.trial117.stdout:Get:86 https://1.chacra.ceph.com/r/ceph/wip-rocky10-branch-of-the-day-2026-02-24-1771941190/506b402a3d3a240d0bfe073b4769eaffdf84497b/ubuntu/jammy/flavors/default jammy/main amd64 radosgw amd64 20.3.0-5580-g506b402a-1jammy [18.4 MB] 2026-02-25T12:00:18.080 INFO:teuthology.orchestra.run.trial117.stdout:Get:87 https://1.chacra.ceph.com/r/ceph/wip-rocky10-branch-of-the-day-2026-02-24-1771941190/506b402a3d3a240d0bfe073b4769eaffdf84497b/ubuntu/jammy/flavors/default jammy/main amd64 rbd-fuse amd64 20.3.0-5580-g506b402a-1jammy [93.7 kB] 2026-02-25T12:00:18.357 INFO:teuthology.orchestra.run.trial117.stdout:Fetched 693 MB in 16s (44.6 MB/s) 2026-02-25T12:00:18.374 INFO:teuthology.orchestra.run.trial117.stdout:Selecting previously unselected package liblttng-ust1:amd64. 2026-02-25T12:00:18.537 INFO:teuthology.orchestra.run.trial117.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 202926 files and directories currently installed.) 2026-02-25T12:00:18.541 INFO:teuthology.orchestra.run.trial117.stdout:Preparing to unpack .../00-liblttng-ust1_2.13.1-1ubuntu1_amd64.deb ... 2026-02-25T12:00:18.542 INFO:teuthology.orchestra.run.trial117.stdout:Unpacking liblttng-ust1:amd64 (2.13.1-1ubuntu1) ... 2026-02-25T12:00:18.563 INFO:teuthology.orchestra.run.trial117.stdout:Selecting previously unselected package libdouble-conversion3:amd64. 2026-02-25T12:00:18.578 INFO:teuthology.orchestra.run.trial117.stdout:Preparing to unpack .../01-libdouble-conversion3_3.1.7-4_amd64.deb ... 2026-02-25T12:00:18.578 INFO:teuthology.orchestra.run.trial117.stdout:Unpacking libdouble-conversion3:amd64 (3.1.7-4) ... 2026-02-25T12:00:18.595 INFO:teuthology.orchestra.run.trial117.stdout:Selecting previously unselected package libpcre2-16-0:amd64. 2026-02-25T12:00:18.610 INFO:teuthology.orchestra.run.trial117.stdout:Preparing to unpack .../02-libpcre2-16-0_10.39-3ubuntu0.1_amd64.deb ... 2026-02-25T12:00:18.610 INFO:teuthology.orchestra.run.trial117.stdout:Unpacking libpcre2-16-0:amd64 (10.39-3ubuntu0.1) ... 2026-02-25T12:00:18.630 INFO:teuthology.orchestra.run.trial117.stdout:Selecting previously unselected package libqt5core5a:amd64. 2026-02-25T12:00:18.640 INFO:teuthology.orchestra.run.trial117.stdout:Preparing to unpack .../03-libqt5core5a_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2026-02-25T12:00:18.644 INFO:teuthology.orchestra.run.trial117.stdout:Unpacking libqt5core5a:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2026-02-25T12:00:18.676 INFO:teuthology.orchestra.run.trial117.stdout:Selecting previously unselected package libqt5dbus5:amd64. 2026-02-25T12:00:18.686 INFO:teuthology.orchestra.run.trial117.stdout:Preparing to unpack .../04-libqt5dbus5_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2026-02-25T12:00:18.686 INFO:teuthology.orchestra.run.trial117.stdout:Unpacking libqt5dbus5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2026-02-25T12:00:18.704 INFO:teuthology.orchestra.run.trial117.stdout:Selecting previously unselected package libqt5network5:amd64. 2026-02-25T12:00:18.714 INFO:teuthology.orchestra.run.trial117.stdout:Preparing to unpack .../05-libqt5network5_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2026-02-25T12:00:18.715 INFO:teuthology.orchestra.run.trial117.stdout:Unpacking libqt5network5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2026-02-25T12:00:18.735 INFO:teuthology.orchestra.run.trial117.stdout:Selecting previously unselected package libthrift-0.16.0:amd64. 2026-02-25T12:00:18.745 INFO:teuthology.orchestra.run.trial117.stdout:Preparing to unpack .../06-libthrift-0.16.0_0.16.0-2_amd64.deb ... 2026-02-25T12:00:18.746 INFO:teuthology.orchestra.run.trial117.stdout:Unpacking libthrift-0.16.0:amd64 (0.16.0-2) ... 2026-02-25T12:00:18.763 INFO:teuthology.orchestra.run.trial117.stdout:Selecting previously unselected package librados2. 2026-02-25T12:00:18.773 INFO:teuthology.orchestra.run.trial117.stdout:Preparing to unpack .../07-librados2_20.3.0-5580-g506b402a-1jammy_amd64.deb ... 2026-02-25T12:00:18.774 INFO:teuthology.orchestra.run.trial117.stdout:Unpacking librados2 (20.3.0-5580-g506b402a-1jammy) ... 2026-02-25T12:00:18.817 INFO:teuthology.orchestra.run.trial117.stdout:Selecting previously unselected package libnbd0. 2026-02-25T12:00:18.827 INFO:teuthology.orchestra.run.trial117.stdout:Preparing to unpack .../08-libnbd0_1.10.5-1_amd64.deb ... 2026-02-25T12:00:18.828 INFO:teuthology.orchestra.run.trial117.stdout:Unpacking libnbd0 (1.10.5-1) ... 2026-02-25T12:00:18.842 INFO:teuthology.orchestra.run.trial117.stdout:Selecting previously unselected package librbd1. 2026-02-25T12:00:18.852 INFO:teuthology.orchestra.run.trial117.stdout:Preparing to unpack .../09-librbd1_20.3.0-5580-g506b402a-1jammy_amd64.deb ... 2026-02-25T12:00:18.853 INFO:teuthology.orchestra.run.trial117.stdout:Unpacking librbd1 (20.3.0-5580-g506b402a-1jammy) ... 2026-02-25T12:00:18.891 INFO:teuthology.orchestra.run.trial117.stdout:Selecting previously unselected package libcephfs2. 2026-02-25T12:00:18.901 INFO:teuthology.orchestra.run.trial117.stdout:Preparing to unpack .../10-libcephfs2_20.3.0-5580-g506b402a-1jammy_amd64.deb ... 2026-02-25T12:00:18.901 INFO:teuthology.orchestra.run.trial117.stdout:Unpacking libcephfs2 (20.3.0-5580-g506b402a-1jammy) ... 2026-02-25T12:00:18.922 INFO:teuthology.orchestra.run.trial117.stdout:Selecting previously unselected package python3-rados. 2026-02-25T12:00:18.932 INFO:teuthology.orchestra.run.trial117.stdout:Preparing to unpack .../11-python3-rados_20.3.0-5580-g506b402a-1jammy_amd64.deb ... 2026-02-25T12:00:18.933 INFO:teuthology.orchestra.run.trial117.stdout:Unpacking python3-rados (20.3.0-5580-g506b402a-1jammy) ... 2026-02-25T12:00:18.950 INFO:teuthology.orchestra.run.trial117.stdout:Selecting previously unselected package python3-ceph-argparse. 2026-02-25T12:00:18.960 INFO:teuthology.orchestra.run.trial117.stdout:Preparing to unpack .../12-python3-ceph-argparse_20.3.0-5580-g506b402a-1jammy_all.deb ... 2026-02-25T12:00:18.961 INFO:teuthology.orchestra.run.trial117.stdout:Unpacking python3-ceph-argparse (20.3.0-5580-g506b402a-1jammy) ... 2026-02-25T12:00:18.975 INFO:teuthology.orchestra.run.trial117.stdout:Selecting previously unselected package python3-cephfs. 2026-02-25T12:00:18.986 INFO:teuthology.orchestra.run.trial117.stdout:Preparing to unpack .../13-python3-cephfs_20.3.0-5580-g506b402a-1jammy_amd64.deb ... 2026-02-25T12:00:18.986 INFO:teuthology.orchestra.run.trial117.stdout:Unpacking python3-cephfs (20.3.0-5580-g506b402a-1jammy) ... 2026-02-25T12:00:19.003 INFO:teuthology.orchestra.run.trial117.stdout:Selecting previously unselected package python3-ceph-common. 2026-02-25T12:00:19.013 INFO:teuthology.orchestra.run.trial117.stdout:Preparing to unpack .../14-python3-ceph-common_20.3.0-5580-g506b402a-1jammy_all.deb ... 2026-02-25T12:00:19.013 INFO:teuthology.orchestra.run.trial117.stdout:Unpacking python3-ceph-common (20.3.0-5580-g506b402a-1jammy) ... 2026-02-25T12:00:19.032 INFO:teuthology.orchestra.run.trial117.stdout:Selecting previously unselected package python3-wcwidth. 2026-02-25T12:00:19.042 INFO:teuthology.orchestra.run.trial117.stdout:Preparing to unpack .../15-python3-wcwidth_0.2.5+dfsg1-1_all.deb ... 2026-02-25T12:00:19.042 INFO:teuthology.orchestra.run.trial117.stdout:Unpacking python3-wcwidth (0.2.5+dfsg1-1) ... 2026-02-25T12:00:19.058 INFO:teuthology.orchestra.run.trial117.stdout:Selecting previously unselected package python3-prettytable. 2026-02-25T12:00:19.068 INFO:teuthology.orchestra.run.trial117.stdout:Preparing to unpack .../16-python3-prettytable_2.5.0-2_all.deb ... 2026-02-25T12:00:19.069 INFO:teuthology.orchestra.run.trial117.stdout:Unpacking python3-prettytable (2.5.0-2) ... 2026-02-25T12:00:19.084 INFO:teuthology.orchestra.run.trial117.stdout:Selecting previously unselected package python3-rbd. 2026-02-25T12:00:19.095 INFO:teuthology.orchestra.run.trial117.stdout:Preparing to unpack .../17-python3-rbd_20.3.0-5580-g506b402a-1jammy_amd64.deb ... 2026-02-25T12:00:19.095 INFO:teuthology.orchestra.run.trial117.stdout:Unpacking python3-rbd (20.3.0-5580-g506b402a-1jammy) ... 2026-02-25T12:00:19.112 INFO:teuthology.orchestra.run.trial117.stdout:Selecting previously unselected package librdkafka1:amd64. 2026-02-25T12:00:19.123 INFO:teuthology.orchestra.run.trial117.stdout:Preparing to unpack .../18-librdkafka1_1.8.0-1build1_amd64.deb ... 2026-02-25T12:00:19.123 INFO:teuthology.orchestra.run.trial117.stdout:Unpacking librdkafka1:amd64 (1.8.0-1build1) ... 2026-02-25T12:00:19.142 INFO:teuthology.orchestra.run.trial117.stdout:Selecting previously unselected package librgw2. 2026-02-25T12:00:19.152 INFO:teuthology.orchestra.run.trial117.stdout:Preparing to unpack .../19-librgw2_20.3.0-5580-g506b402a-1jammy_amd64.deb ... 2026-02-25T12:00:19.152 INFO:teuthology.orchestra.run.trial117.stdout:Unpacking librgw2 (20.3.0-5580-g506b402a-1jammy) ... 2026-02-25T12:00:19.250 INFO:teuthology.orchestra.run.trial117.stdout:Selecting previously unselected package python3-rgw. 2026-02-25T12:00:19.260 INFO:teuthology.orchestra.run.trial117.stdout:Preparing to unpack .../20-python3-rgw_20.3.0-5580-g506b402a-1jammy_amd64.deb ... 2026-02-25T12:00:19.261 INFO:teuthology.orchestra.run.trial117.stdout:Unpacking python3-rgw (20.3.0-5580-g506b402a-1jammy) ... 2026-02-25T12:00:19.277 INFO:teuthology.orchestra.run.trial117.stdout:Selecting previously unselected package liboath0:amd64. 2026-02-25T12:00:19.286 INFO:teuthology.orchestra.run.trial117.stdout:Preparing to unpack .../21-liboath0_2.6.7-3ubuntu0.1_amd64.deb ... 2026-02-25T12:00:19.287 INFO:teuthology.orchestra.run.trial117.stdout:Unpacking liboath0:amd64 (2.6.7-3ubuntu0.1) ... 2026-02-25T12:00:19.302 INFO:teuthology.orchestra.run.trial117.stdout:Selecting previously unselected package libradosstriper1. 2026-02-25T12:00:19.312 INFO:teuthology.orchestra.run.trial117.stdout:Preparing to unpack .../22-libradosstriper1_20.3.0-5580-g506b402a-1jammy_amd64.deb ... 2026-02-25T12:00:19.313 INFO:teuthology.orchestra.run.trial117.stdout:Unpacking libradosstriper1 (20.3.0-5580-g506b402a-1jammy) ... 2026-02-25T12:00:19.330 INFO:teuthology.orchestra.run.trial117.stdout:Selecting previously unselected package ceph-common. 2026-02-25T12:00:19.340 INFO:teuthology.orchestra.run.trial117.stdout:Preparing to unpack .../23-ceph-common_20.3.0-5580-g506b402a-1jammy_amd64.deb ... 2026-02-25T12:00:19.340 INFO:teuthology.orchestra.run.trial117.stdout:Unpacking ceph-common (20.3.0-5580-g506b402a-1jammy) ... 2026-02-25T12:00:19.597 INFO:teuthology.orchestra.run.trial117.stdout:Selecting previously unselected package ceph-base. 2026-02-25T12:00:19.607 INFO:teuthology.orchestra.run.trial117.stdout:Preparing to unpack .../24-ceph-base_20.3.0-5580-g506b402a-1jammy_amd64.deb ... 2026-02-25T12:00:19.612 INFO:teuthology.orchestra.run.trial117.stdout:Unpacking ceph-base (20.3.0-5580-g506b402a-1jammy) ... 2026-02-25T12:00:19.683 INFO:teuthology.orchestra.run.trial117.stdout:Selecting previously unselected package python3-jaraco.functools. 2026-02-25T12:00:19.693 INFO:teuthology.orchestra.run.trial117.stdout:Preparing to unpack .../25-python3-jaraco.functools_3.4.0-2_all.deb ... 2026-02-25T12:00:19.694 INFO:teuthology.orchestra.run.trial117.stdout:Unpacking python3-jaraco.functools (3.4.0-2) ... 2026-02-25T12:00:19.709 INFO:teuthology.orchestra.run.trial117.stdout:Selecting previously unselected package python3-cheroot. 2026-02-25T12:00:19.718 INFO:teuthology.orchestra.run.trial117.stdout:Preparing to unpack .../26-python3-cheroot_8.5.2+ds1-1ubuntu3.1_all.deb ... 2026-02-25T12:00:19.719 INFO:teuthology.orchestra.run.trial117.stdout:Unpacking python3-cheroot (8.5.2+ds1-1ubuntu3.1) ... 2026-02-25T12:00:19.736 INFO:teuthology.orchestra.run.trial117.stdout:Selecting previously unselected package python3-jaraco.classes. 2026-02-25T12:00:19.747 INFO:teuthology.orchestra.run.trial117.stdout:Preparing to unpack .../27-python3-jaraco.classes_3.2.1-3_all.deb ... 2026-02-25T12:00:19.747 INFO:teuthology.orchestra.run.trial117.stdout:Unpacking python3-jaraco.classes (3.2.1-3) ... 2026-02-25T12:00:19.762 INFO:teuthology.orchestra.run.trial117.stdout:Selecting previously unselected package python3-jaraco.text. 2026-02-25T12:00:19.772 INFO:teuthology.orchestra.run.trial117.stdout:Preparing to unpack .../28-python3-jaraco.text_3.6.0-2_all.deb ... 2026-02-25T12:00:19.772 INFO:teuthology.orchestra.run.trial117.stdout:Unpacking python3-jaraco.text (3.6.0-2) ... 2026-02-25T12:00:19.786 INFO:teuthology.orchestra.run.trial117.stdout:Selecting previously unselected package python3-jaraco.collections. 2026-02-25T12:00:19.796 INFO:teuthology.orchestra.run.trial117.stdout:Preparing to unpack .../29-python3-jaraco.collections_3.4.0-2_all.deb ... 2026-02-25T12:00:19.797 INFO:teuthology.orchestra.run.trial117.stdout:Unpacking python3-jaraco.collections (3.4.0-2) ... 2026-02-25T12:00:19.812 INFO:teuthology.orchestra.run.trial117.stdout:Selecting previously unselected package python3-tempora. 2026-02-25T12:00:19.822 INFO:teuthology.orchestra.run.trial117.stdout:Preparing to unpack .../30-python3-tempora_4.1.2-1_all.deb ... 2026-02-25T12:00:19.822 INFO:teuthology.orchestra.run.trial117.stdout:Unpacking python3-tempora (4.1.2-1) ... 2026-02-25T12:00:19.838 INFO:teuthology.orchestra.run.trial117.stdout:Selecting previously unselected package python3-portend. 2026-02-25T12:00:19.848 INFO:teuthology.orchestra.run.trial117.stdout:Preparing to unpack .../31-python3-portend_3.0.0-1_all.deb ... 2026-02-25T12:00:19.848 INFO:teuthology.orchestra.run.trial117.stdout:Unpacking python3-portend (3.0.0-1) ... 2026-02-25T12:00:19.863 INFO:teuthology.orchestra.run.trial117.stdout:Selecting previously unselected package python3-zc.lockfile. 2026-02-25T12:00:19.873 INFO:teuthology.orchestra.run.trial117.stdout:Preparing to unpack .../32-python3-zc.lockfile_2.0-1_all.deb ... 2026-02-25T12:00:19.873 INFO:teuthology.orchestra.run.trial117.stdout:Unpacking python3-zc.lockfile (2.0-1) ... 2026-02-25T12:00:19.889 INFO:teuthology.orchestra.run.trial117.stdout:Selecting previously unselected package python3-cherrypy3. 2026-02-25T12:00:19.898 INFO:teuthology.orchestra.run.trial117.stdout:Preparing to unpack .../33-python3-cherrypy3_18.6.1-4_all.deb ... 2026-02-25T12:00:19.899 INFO:teuthology.orchestra.run.trial117.stdout:Unpacking python3-cherrypy3 (18.6.1-4) ... 2026-02-25T12:00:19.923 INFO:teuthology.orchestra.run.trial117.stdout:Selecting previously unselected package python3-natsort. 2026-02-25T12:00:19.934 INFO:teuthology.orchestra.run.trial117.stdout:Preparing to unpack .../34-python3-natsort_8.0.2-1_all.deb ... 2026-02-25T12:00:19.934 INFO:teuthology.orchestra.run.trial117.stdout:Unpacking python3-natsort (8.0.2-1) ... 2026-02-25T12:00:19.950 INFO:teuthology.orchestra.run.trial117.stdout:Selecting previously unselected package ceph-mgr-modules-core. 2026-02-25T12:00:19.960 INFO:teuthology.orchestra.run.trial117.stdout:Preparing to unpack .../35-ceph-mgr-modules-core_20.3.0-5580-g506b402a-1jammy_all.deb ... 2026-02-25T12:00:19.961 INFO:teuthology.orchestra.run.trial117.stdout:Unpacking ceph-mgr-modules-core (20.3.0-5580-g506b402a-1jammy) ... 2026-02-25T12:00:19.990 INFO:teuthology.orchestra.run.trial117.stdout:Selecting previously unselected package libsqlite3-mod-ceph. 2026-02-25T12:00:20.000 INFO:teuthology.orchestra.run.trial117.stdout:Preparing to unpack .../36-libsqlite3-mod-ceph_20.3.0-5580-g506b402a-1jammy_amd64.deb ... 2026-02-25T12:00:20.000 INFO:teuthology.orchestra.run.trial117.stdout:Unpacking libsqlite3-mod-ceph (20.3.0-5580-g506b402a-1jammy) ... 2026-02-25T12:00:20.016 INFO:teuthology.orchestra.run.trial117.stdout:Selecting previously unselected package ceph-mgr. 2026-02-25T12:00:20.026 INFO:teuthology.orchestra.run.trial117.stdout:Preparing to unpack .../37-ceph-mgr_20.3.0-5580-g506b402a-1jammy_amd64.deb ... 2026-02-25T12:00:20.026 INFO:teuthology.orchestra.run.trial117.stdout:Unpacking ceph-mgr (20.3.0-5580-g506b402a-1jammy) ... 2026-02-25T12:00:20.048 INFO:teuthology.orchestra.run.trial117.stdout:Selecting previously unselected package ceph-mon. 2026-02-25T12:00:20.057 INFO:teuthology.orchestra.run.trial117.stdout:Preparing to unpack .../38-ceph-mon_20.3.0-5580-g506b402a-1jammy_amd64.deb ... 2026-02-25T12:00:20.058 INFO:teuthology.orchestra.run.trial117.stdout:Unpacking ceph-mon (20.3.0-5580-g506b402a-1jammy) ... 2026-02-25T12:00:20.122 INFO:teuthology.orchestra.run.trial117.stdout:Selecting previously unselected package libfuse2:amd64. 2026-02-25T12:00:20.132 INFO:teuthology.orchestra.run.trial117.stdout:Preparing to unpack .../39-libfuse2_2.9.9-5ubuntu3_amd64.deb ... 2026-02-25T12:00:20.133 INFO:teuthology.orchestra.run.trial117.stdout:Unpacking libfuse2:amd64 (2.9.9-5ubuntu3) ... 2026-02-25T12:00:20.149 INFO:teuthology.orchestra.run.trial117.stdout:Selecting previously unselected package ceph-osd-classic. 2026-02-25T12:00:20.159 INFO:teuthology.orchestra.run.trial117.stdout:Preparing to unpack .../40-ceph-osd-classic_20.3.0-5580-g506b402a-1jammy_amd64.deb ... 2026-02-25T12:00:20.159 INFO:teuthology.orchestra.run.trial117.stdout:Unpacking ceph-osd-classic (20.3.0-5580-g506b402a-1jammy) ... 2026-02-25T12:00:24.099 INFO:teuthology.orchestra.run.trial117.stdout:Selecting previously unselected package ceph-osd. 2026-02-25T12:00:24.109 INFO:teuthology.orchestra.run.trial117.stdout:Preparing to unpack .../41-ceph-osd_20.3.0-5580-g506b402a-1jammy_amd64.deb ... 2026-02-25T12:00:24.110 INFO:teuthology.orchestra.run.trial117.stdout:Unpacking ceph-osd (20.3.0-5580-g506b402a-1jammy) ... 2026-02-25T12:00:24.130 INFO:teuthology.orchestra.run.trial117.stdout:Selecting previously unselected package ceph. 2026-02-25T12:00:24.145 INFO:teuthology.orchestra.run.trial117.stdout:Preparing to unpack .../42-ceph_20.3.0-5580-g506b402a-1jammy_amd64.deb ... 2026-02-25T12:00:24.145 INFO:teuthology.orchestra.run.trial117.stdout:Unpacking ceph (20.3.0-5580-g506b402a-1jammy) ... 2026-02-25T12:00:24.163 INFO:teuthology.orchestra.run.trial117.stdout:Selecting previously unselected package ceph-fuse. 2026-02-25T12:00:24.178 INFO:teuthology.orchestra.run.trial117.stdout:Preparing to unpack .../43-ceph-fuse_20.3.0-5580-g506b402a-1jammy_amd64.deb ... 2026-02-25T12:00:24.179 INFO:teuthology.orchestra.run.trial117.stdout:Unpacking ceph-fuse (20.3.0-5580-g506b402a-1jammy) ... 2026-02-25T12:00:24.203 INFO:teuthology.orchestra.run.trial117.stdout:Selecting previously unselected package ceph-mds. 2026-02-25T12:00:24.214 INFO:teuthology.orchestra.run.trial117.stdout:Preparing to unpack .../44-ceph-mds_20.3.0-5580-g506b402a-1jammy_amd64.deb ... 2026-02-25T12:00:24.214 INFO:teuthology.orchestra.run.trial117.stdout:Unpacking ceph-mds (20.3.0-5580-g506b402a-1jammy) ... 2026-02-25T12:00:24.246 INFO:teuthology.orchestra.run.trial117.stdout:Selecting previously unselected package cephadm. 2026-02-25T12:00:24.257 INFO:teuthology.orchestra.run.trial117.stdout:Preparing to unpack .../45-cephadm_20.3.0-5580-g506b402a-1jammy_amd64.deb ... 2026-02-25T12:00:24.257 INFO:teuthology.orchestra.run.trial117.stdout:Unpacking cephadm (20.3.0-5580-g506b402a-1jammy) ... 2026-02-25T12:00:24.274 INFO:teuthology.orchestra.run.trial117.stdout:Selecting previously unselected package python3-asyncssh. 2026-02-25T12:00:24.284 INFO:teuthology.orchestra.run.trial117.stdout:Preparing to unpack .../46-python3-asyncssh_2.5.0-1ubuntu0.1_all.deb ... 2026-02-25T12:00:24.284 INFO:teuthology.orchestra.run.trial117.stdout:Unpacking python3-asyncssh (2.5.0-1ubuntu0.1) ... 2026-02-25T12:00:24.306 INFO:teuthology.orchestra.run.trial117.stdout:Selecting previously unselected package ceph-mgr-cephadm. 2026-02-25T12:00:24.316 INFO:teuthology.orchestra.run.trial117.stdout:Preparing to unpack .../47-ceph-mgr-cephadm_20.3.0-5580-g506b402a-1jammy_all.deb ... 2026-02-25T12:00:24.317 INFO:teuthology.orchestra.run.trial117.stdout:Unpacking ceph-mgr-cephadm (20.3.0-5580-g506b402a-1jammy) ... 2026-02-25T12:00:24.338 INFO:teuthology.orchestra.run.trial117.stdout:Selecting previously unselected package python3-repoze.lru. 2026-02-25T12:00:24.348 INFO:teuthology.orchestra.run.trial117.stdout:Preparing to unpack .../48-python3-repoze.lru_0.7-2_all.deb ... 2026-02-25T12:00:24.349 INFO:teuthology.orchestra.run.trial117.stdout:Unpacking python3-repoze.lru (0.7-2) ... 2026-02-25T12:00:24.364 INFO:teuthology.orchestra.run.trial117.stdout:Selecting previously unselected package python3-routes. 2026-02-25T12:00:24.374 INFO:teuthology.orchestra.run.trial117.stdout:Preparing to unpack .../49-python3-routes_2.5.1-1ubuntu1_all.deb ... 2026-02-25T12:00:24.374 INFO:teuthology.orchestra.run.trial117.stdout:Unpacking python3-routes (2.5.1-1ubuntu1) ... 2026-02-25T12:00:24.395 INFO:teuthology.orchestra.run.trial117.stdout:Selecting previously unselected package ceph-mgr-dashboard. 2026-02-25T12:00:24.405 INFO:teuthology.orchestra.run.trial117.stdout:Preparing to unpack .../50-ceph-mgr-dashboard_20.3.0-5580-g506b402a-1jammy_all.deb ... 2026-02-25T12:00:24.405 INFO:teuthology.orchestra.run.trial117.stdout:Unpacking ceph-mgr-dashboard (20.3.0-5580-g506b402a-1jammy) ... 2026-02-25T12:00:24.952 INFO:teuthology.orchestra.run.trial117.stdout:Selecting previously unselected package python3-sklearn-lib:amd64. 2026-02-25T12:00:24.963 INFO:teuthology.orchestra.run.trial117.stdout:Preparing to unpack .../51-python3-sklearn-lib_0.23.2-5ubuntu6_amd64.deb ... 2026-02-25T12:00:24.963 INFO:teuthology.orchestra.run.trial117.stdout:Unpacking python3-sklearn-lib:amd64 (0.23.2-5ubuntu6) ... 2026-02-25T12:00:25.003 INFO:teuthology.orchestra.run.trial117.stdout:Selecting previously unselected package python3-joblib. 2026-02-25T12:00:25.013 INFO:teuthology.orchestra.run.trial117.stdout:Preparing to unpack .../52-python3-joblib_0.17.0-4ubuntu1_all.deb ... 2026-02-25T12:00:25.014 INFO:teuthology.orchestra.run.trial117.stdout:Unpacking python3-joblib (0.17.0-4ubuntu1) ... 2026-02-25T12:00:25.042 INFO:teuthology.orchestra.run.trial117.stdout:Selecting previously unselected package python3-threadpoolctl. 2026-02-25T12:00:25.052 INFO:teuthology.orchestra.run.trial117.stdout:Preparing to unpack .../53-python3-threadpoolctl_3.1.0-1_all.deb ... 2026-02-25T12:00:25.052 INFO:teuthology.orchestra.run.trial117.stdout:Unpacking python3-threadpoolctl (3.1.0-1) ... 2026-02-25T12:00:25.067 INFO:teuthology.orchestra.run.trial117.stdout:Selecting previously unselected package python3-sklearn. 2026-02-25T12:00:25.078 INFO:teuthology.orchestra.run.trial117.stdout:Preparing to unpack .../54-python3-sklearn_0.23.2-5ubuntu6_all.deb ... 2026-02-25T12:00:25.079 INFO:teuthology.orchestra.run.trial117.stdout:Unpacking python3-sklearn (0.23.2-5ubuntu6) ... 2026-02-25T12:00:25.161 INFO:teuthology.orchestra.run.trial117.stdout:Selecting previously unselected package ceph-mgr-diskprediction-local. 2026-02-25T12:00:25.172 INFO:teuthology.orchestra.run.trial117.stdout:Preparing to unpack .../55-ceph-mgr-diskprediction-local_20.3.0-5580-g506b402a-1jammy_all.deb ... 2026-02-25T12:00:25.172 INFO:teuthology.orchestra.run.trial117.stdout:Unpacking ceph-mgr-diskprediction-local (20.3.0-5580-g506b402a-1jammy) ... 2026-02-25T12:00:25.332 INFO:teuthology.orchestra.run.trial117.stdout:Selecting previously unselected package python3-cachetools. 2026-02-25T12:00:25.343 INFO:teuthology.orchestra.run.trial117.stdout:Preparing to unpack .../56-python3-cachetools_5.0.0-1_all.deb ... 2026-02-25T12:00:25.343 INFO:teuthology.orchestra.run.trial117.stdout:Unpacking python3-cachetools (5.0.0-1) ... 2026-02-25T12:00:25.358 INFO:teuthology.orchestra.run.trial117.stdout:Selecting previously unselected package python3-rsa. 2026-02-25T12:00:25.369 INFO:teuthology.orchestra.run.trial117.stdout:Preparing to unpack .../57-python3-rsa_4.8-1_all.deb ... 2026-02-25T12:00:25.369 INFO:teuthology.orchestra.run.trial117.stdout:Unpacking python3-rsa (4.8-1) ... 2026-02-25T12:00:25.388 INFO:teuthology.orchestra.run.trial117.stdout:Selecting previously unselected package python3-google-auth. 2026-02-25T12:00:25.399 INFO:teuthology.orchestra.run.trial117.stdout:Preparing to unpack .../58-python3-google-auth_1.5.1-3_all.deb ... 2026-02-25T12:00:25.399 INFO:teuthology.orchestra.run.trial117.stdout:Unpacking python3-google-auth (1.5.1-3) ... 2026-02-25T12:00:25.417 INFO:teuthology.orchestra.run.trial117.stdout:Selecting previously unselected package python3-requests-oauthlib. 2026-02-25T12:00:25.427 INFO:teuthology.orchestra.run.trial117.stdout:Preparing to unpack .../59-python3-requests-oauthlib_1.3.0+ds-0.1_all.deb ... 2026-02-25T12:00:25.427 INFO:teuthology.orchestra.run.trial117.stdout:Unpacking python3-requests-oauthlib (1.3.0+ds-0.1) ... 2026-02-25T12:00:25.444 INFO:teuthology.orchestra.run.trial117.stdout:Selecting previously unselected package python3-websocket. 2026-02-25T12:00:25.454 INFO:teuthology.orchestra.run.trial117.stdout:Preparing to unpack .../60-python3-websocket_1.2.3-1_all.deb ... 2026-02-25T12:00:25.454 INFO:teuthology.orchestra.run.trial117.stdout:Unpacking python3-websocket (1.2.3-1) ... 2026-02-25T12:00:25.472 INFO:teuthology.orchestra.run.trial117.stdout:Selecting previously unselected package python3-kubernetes. 2026-02-25T12:00:25.483 INFO:teuthology.orchestra.run.trial117.stdout:Preparing to unpack .../61-python3-kubernetes_12.0.1-1ubuntu1_all.deb ... 2026-02-25T12:00:25.483 INFO:teuthology.orchestra.run.trial117.stdout:Unpacking python3-kubernetes (12.0.1-1ubuntu1) ... 2026-02-25T12:00:25.578 INFO:teuthology.orchestra.run.trial117.stdout:Selecting previously unselected package ceph-mgr-k8sevents. 2026-02-25T12:00:25.590 INFO:teuthology.orchestra.run.trial117.stdout:Preparing to unpack .../62-ceph-mgr-k8sevents_20.3.0-5580-g506b402a-1jammy_all.deb ... 2026-02-25T12:00:25.590 INFO:teuthology.orchestra.run.trial117.stdout:Unpacking ceph-mgr-k8sevents (20.3.0-5580-g506b402a-1jammy) ... 2026-02-25T12:00:25.606 INFO:teuthology.orchestra.run.trial117.stdout:Selecting previously unselected package libonig5:amd64. 2026-02-25T12:00:25.616 INFO:teuthology.orchestra.run.trial117.stdout:Preparing to unpack .../63-libonig5_6.9.7.1-2build1_amd64.deb ... 2026-02-25T12:00:25.616 INFO:teuthology.orchestra.run.trial117.stdout:Unpacking libonig5:amd64 (6.9.7.1-2build1) ... 2026-02-25T12:00:25.631 INFO:teuthology.orchestra.run.trial117.stdout:Selecting previously unselected package libjq1:amd64. 2026-02-25T12:00:25.641 INFO:teuthology.orchestra.run.trial117.stdout:Preparing to unpack .../64-libjq1_1.6-2.1ubuntu3.1_amd64.deb ... 2026-02-25T12:00:25.642 INFO:teuthology.orchestra.run.trial117.stdout:Unpacking libjq1:amd64 (1.6-2.1ubuntu3.1) ... 2026-02-25T12:00:25.658 INFO:teuthology.orchestra.run.trial117.stdout:Selecting previously unselected package jq. 2026-02-25T12:00:25.668 INFO:teuthology.orchestra.run.trial117.stdout:Preparing to unpack .../65-jq_1.6-2.1ubuntu3.1_amd64.deb ... 2026-02-25T12:00:25.668 INFO:teuthology.orchestra.run.trial117.stdout:Unpacking jq (1.6-2.1ubuntu3.1) ... 2026-02-25T12:00:25.683 INFO:teuthology.orchestra.run.trial117.stdout:Selecting previously unselected package socat. 2026-02-25T12:00:25.694 INFO:teuthology.orchestra.run.trial117.stdout:Preparing to unpack .../66-socat_1.7.4.1-3ubuntu4_amd64.deb ... 2026-02-25T12:00:25.694 INFO:teuthology.orchestra.run.trial117.stdout:Unpacking socat (1.7.4.1-3ubuntu4) ... 2026-02-25T12:00:25.714 INFO:teuthology.orchestra.run.trial117.stdout:Selecting previously unselected package xmlstarlet. 2026-02-25T12:00:25.725 INFO:teuthology.orchestra.run.trial117.stdout:Preparing to unpack .../67-xmlstarlet_1.6.1-2.1_amd64.deb ... 2026-02-25T12:00:25.725 INFO:teuthology.orchestra.run.trial117.stdout:Unpacking xmlstarlet (1.6.1-2.1) ... 2026-02-25T12:00:25.763 INFO:teuthology.orchestra.run.trial117.stdout:Selecting previously unselected package ceph-test. 2026-02-25T12:00:25.773 INFO:teuthology.orchestra.run.trial117.stdout:Preparing to unpack .../68-ceph-test_20.3.0-5580-g506b402a-1jammy_amd64.deb ... 2026-02-25T12:00:25.773 INFO:teuthology.orchestra.run.trial117.stdout:Unpacking ceph-test (20.3.0-5580-g506b402a-1jammy) ... 2026-02-25T12:00:26.410 INFO:teuthology.orchestra.run.trial117.stdout:Selecting previously unselected package ceph-volume. 2026-02-25T12:00:26.420 INFO:teuthology.orchestra.run.trial117.stdout:Preparing to unpack .../69-ceph-volume_20.3.0-5580-g506b402a-1jammy_all.deb ... 2026-02-25T12:00:26.421 INFO:teuthology.orchestra.run.trial117.stdout:Unpacking ceph-volume (20.3.0-5580-g506b402a-1jammy) ... 2026-02-25T12:00:26.444 INFO:teuthology.orchestra.run.trial117.stdout:Selecting previously unselected package libcephfs-daemon. 2026-02-25T12:00:26.455 INFO:teuthology.orchestra.run.trial117.stdout:Preparing to unpack .../70-libcephfs-daemon_20.3.0-5580-g506b402a-1jammy_amd64.deb ... 2026-02-25T12:00:26.455 INFO:teuthology.orchestra.run.trial117.stdout:Unpacking libcephfs-daemon (20.3.0-5580-g506b402a-1jammy) ... 2026-02-25T12:00:26.470 INFO:teuthology.orchestra.run.trial117.stdout:Selecting previously unselected package libcephfs-proxy2. 2026-02-25T12:00:26.480 INFO:teuthology.orchestra.run.trial117.stdout:Preparing to unpack .../71-libcephfs-proxy2_20.3.0-5580-g506b402a-1jammy_amd64.deb ... 2026-02-25T12:00:26.481 INFO:teuthology.orchestra.run.trial117.stdout:Unpacking libcephfs-proxy2 (20.3.0-5580-g506b402a-1jammy) ... 2026-02-25T12:00:26.496 INFO:teuthology.orchestra.run.trial117.stdout:Selecting previously unselected package libcephfs-dev. 2026-02-25T12:00:26.506 INFO:teuthology.orchestra.run.trial117.stdout:Preparing to unpack .../72-libcephfs-dev_20.3.0-5580-g506b402a-1jammy_amd64.deb ... 2026-02-25T12:00:26.507 INFO:teuthology.orchestra.run.trial117.stdout:Unpacking libcephfs-dev (20.3.0-5580-g506b402a-1jammy) ... 2026-02-25T12:00:26.523 INFO:teuthology.orchestra.run.trial117.stdout:Selecting previously unselected package nvme-cli. 2026-02-25T12:00:26.533 INFO:teuthology.orchestra.run.trial117.stdout:Preparing to unpack .../73-nvme-cli_1.16-3ubuntu0.3_amd64.deb ... 2026-02-25T12:00:26.534 INFO:teuthology.orchestra.run.trial117.stdout:Unpacking nvme-cli (1.16-3ubuntu0.3) ... 2026-02-25T12:00:26.566 INFO:teuthology.orchestra.run.trial117.stdout:Selecting previously unselected package python-asyncssh-doc. 2026-02-25T12:00:26.576 INFO:teuthology.orchestra.run.trial117.stdout:Preparing to unpack .../74-python-asyncssh-doc_2.5.0-1ubuntu0.1_all.deb ... 2026-02-25T12:00:26.577 INFO:teuthology.orchestra.run.trial117.stdout:Unpacking python-asyncssh-doc (2.5.0-1ubuntu0.1) ... 2026-02-25T12:00:26.607 INFO:teuthology.orchestra.run.trial117.stdout:Selecting previously unselected package python3-iniconfig. 2026-02-25T12:00:26.618 INFO:teuthology.orchestra.run.trial117.stdout:Preparing to unpack .../75-python3-iniconfig_1.1.1-2_all.deb ... 2026-02-25T12:00:26.618 INFO:teuthology.orchestra.run.trial117.stdout:Unpacking python3-iniconfig (1.1.1-2) ... 2026-02-25T12:00:26.633 INFO:teuthology.orchestra.run.trial117.stdout:Selecting previously unselected package python3-pluggy. 2026-02-25T12:00:26.644 INFO:teuthology.orchestra.run.trial117.stdout:Preparing to unpack .../76-python3-pluggy_0.13.0-7.1_all.deb ... 2026-02-25T12:00:26.644 INFO:teuthology.orchestra.run.trial117.stdout:Unpacking python3-pluggy (0.13.0-7.1) ... 2026-02-25T12:00:26.661 INFO:teuthology.orchestra.run.trial117.stdout:Selecting previously unselected package python3-psutil. 2026-02-25T12:00:26.671 INFO:teuthology.orchestra.run.trial117.stdout:Preparing to unpack .../77-python3-psutil_5.9.0-1build1_amd64.deb ... 2026-02-25T12:00:26.672 INFO:teuthology.orchestra.run.trial117.stdout:Unpacking python3-psutil (5.9.0-1build1) ... 2026-02-25T12:00:26.691 INFO:teuthology.orchestra.run.trial117.stdout:Selecting previously unselected package python3-py. 2026-02-25T12:00:26.701 INFO:teuthology.orchestra.run.trial117.stdout:Preparing to unpack .../78-python3-py_1.10.0-1_all.deb ... 2026-02-25T12:00:26.701 INFO:teuthology.orchestra.run.trial117.stdout:Unpacking python3-py (1.10.0-1) ... 2026-02-25T12:00:26.722 INFO:teuthology.orchestra.run.trial117.stdout:Selecting previously unselected package python3-pygments. 2026-02-25T12:00:26.732 INFO:teuthology.orchestra.run.trial117.stdout:Preparing to unpack .../79-python3-pygments_2.11.2+dfsg-2ubuntu0.1_all.deb ... 2026-02-25T12:00:26.733 INFO:teuthology.orchestra.run.trial117.stdout:Unpacking python3-pygments (2.11.2+dfsg-2ubuntu0.1) ... 2026-02-25T12:00:26.776 INFO:teuthology.orchestra.run.trial117.stdout:Selecting previously unselected package python3-toml. 2026-02-25T12:00:26.786 INFO:teuthology.orchestra.run.trial117.stdout:Preparing to unpack .../80-python3-toml_0.10.2-1_all.deb ... 2026-02-25T12:00:26.787 INFO:teuthology.orchestra.run.trial117.stdout:Unpacking python3-toml (0.10.2-1) ... 2026-02-25T12:00:26.803 INFO:teuthology.orchestra.run.trial117.stdout:Selecting previously unselected package python3-pytest. 2026-02-25T12:00:26.813 INFO:teuthology.orchestra.run.trial117.stdout:Preparing to unpack .../81-python3-pytest_6.2.5-1ubuntu2_all.deb ... 2026-02-25T12:00:26.813 INFO:teuthology.orchestra.run.trial117.stdout:Unpacking python3-pytest (6.2.5-1ubuntu2) ... 2026-02-25T12:00:26.835 INFO:teuthology.orchestra.run.trial117.stdout:Selecting previously unselected package python3-simplejson. 2026-02-25T12:00:26.845 INFO:teuthology.orchestra.run.trial117.stdout:Preparing to unpack .../82-python3-simplejson_3.17.6-1build1_amd64.deb ... 2026-02-25T12:00:26.846 INFO:teuthology.orchestra.run.trial117.stdout:Unpacking python3-simplejson (3.17.6-1build1) ... 2026-02-25T12:00:26.871 INFO:teuthology.orchestra.run.trial117.stdout:Selecting previously unselected package python3-webob. 2026-02-25T12:00:26.881 INFO:teuthology.orchestra.run.trial117.stdout:Preparing to unpack .../83-python3-webob_1%3a1.8.6-1.1ubuntu0.1_all.deb ... 2026-02-25T12:00:26.882 INFO:teuthology.orchestra.run.trial117.stdout:Unpacking python3-webob (1:1.8.6-1.1ubuntu0.1) ... 2026-02-25T12:00:26.899 INFO:teuthology.orchestra.run.trial117.stdout:Selecting previously unselected package qttranslations5-l10n. 2026-02-25T12:00:26.909 INFO:teuthology.orchestra.run.trial117.stdout:Preparing to unpack .../84-qttranslations5-l10n_5.15.3-1_all.deb ... 2026-02-25T12:00:26.910 INFO:teuthology.orchestra.run.trial117.stdout:Unpacking qttranslations5-l10n (5.15.3-1) ... 2026-02-25T12:00:26.977 INFO:teuthology.orchestra.run.trial117.stdout:Selecting previously unselected package radosgw. 2026-02-25T12:00:26.988 INFO:teuthology.orchestra.run.trial117.stdout:Preparing to unpack .../85-radosgw_20.3.0-5580-g506b402a-1jammy_amd64.deb ... 2026-02-25T12:00:26.989 INFO:teuthology.orchestra.run.trial117.stdout:Unpacking radosgw (20.3.0-5580-g506b402a-1jammy) ... 2026-02-25T12:00:27.146 INFO:teuthology.orchestra.run.trial117.stdout:Selecting previously unselected package rbd-fuse. 2026-02-25T12:00:27.156 INFO:teuthology.orchestra.run.trial117.stdout:Preparing to unpack .../86-rbd-fuse_20.3.0-5580-g506b402a-1jammy_amd64.deb ... 2026-02-25T12:00:27.156 INFO:teuthology.orchestra.run.trial117.stdout:Unpacking rbd-fuse (20.3.0-5580-g506b402a-1jammy) ... 2026-02-25T12:00:27.186 INFO:teuthology.orchestra.run.trial117.stdout:Setting up python3-iniconfig (1.1.1-2) ... 2026-02-25T12:00:27.253 INFO:teuthology.orchestra.run.trial117.stdout:Setting up libdouble-conversion3:amd64 (3.1.7-4) ... 2026-02-25T12:00:27.254 INFO:teuthology.orchestra.run.trial117.stdout:Setting up nvme-cli (1.16-3ubuntu0.3) ... 2026-02-25T12:00:27.325 INFO:teuthology.orchestra.run.trial117.stdout:Created symlink /etc/systemd/system/default.target.wants/nvmefc-boot-connections.service → /lib/systemd/system/nvmefc-boot-connections.service. 2026-02-25T12:00:27.609 INFO:teuthology.orchestra.run.trial117.stdout:Created symlink /etc/systemd/system/default.target.wants/nvmf-autoconnect.service → /lib/systemd/system/nvmf-autoconnect.service. 2026-02-25T12:00:27.993 INFO:teuthology.orchestra.run.trial117.stdout:nvmf-connect.target is a disabled or a static unit, not starting it. 2026-02-25T12:00:28.002 INFO:teuthology.orchestra.run.trial117.stdout:Setting up cephadm (20.3.0-5580-g506b402a-1jammy) ... 2026-02-25T12:00:28.045 INFO:teuthology.orchestra.run.trial117.stdout:Adding system user cephadm....done 2026-02-25T12:00:28.050 INFO:teuthology.orchestra.run.trial117.stdout:usermod: no changes 2026-02-25T12:00:28.056 INFO:teuthology.orchestra.run.trial117.stdout:Setting up python3-jaraco.classes (3.2.1-3) ... 2026-02-25T12:00:28.129 INFO:teuthology.orchestra.run.trial117.stdout:Setting up python-asyncssh-doc (2.5.0-1ubuntu0.1) ... 2026-02-25T12:00:28.130 INFO:teuthology.orchestra.run.trial117.stdout:Setting up python3-jaraco.functools (3.4.0-2) ... 2026-02-25T12:00:28.202 INFO:teuthology.orchestra.run.trial117.stdout:Setting up python3-repoze.lru (0.7-2) ... 2026-02-25T12:00:28.276 INFO:teuthology.orchestra.run.trial117.stdout:Setting up liboath0:amd64 (2.6.7-3ubuntu0.1) ... 2026-02-25T12:00:28.277 INFO:teuthology.orchestra.run.trial117.stdout:Setting up python3-py (1.10.0-1) ... 2026-02-25T12:00:28.372 INFO:teuthology.orchestra.run.trial117.stdout:Setting up python3-joblib (0.17.0-4ubuntu1) ... 2026-02-25T12:00:28.495 INFO:teuthology.orchestra.run.trial117.stdout:Setting up python3-cachetools (5.0.0-1) ... 2026-02-25T12:00:28.568 INFO:teuthology.orchestra.run.trial117.stdout:Setting up python3-threadpoolctl (3.1.0-1) ... 2026-02-25T12:00:28.642 INFO:teuthology.orchestra.run.trial117.stdout:Setting up python3-ceph-argparse (20.3.0-5580-g506b402a-1jammy) ... 2026-02-25T12:00:28.708 INFO:teuthology.orchestra.run.trial117.stdout:Setting up python3-sklearn-lib:amd64 (0.23.2-5ubuntu6) ... 2026-02-25T12:00:28.709 INFO:teuthology.orchestra.run.trial117.stdout:Setting up libnbd0 (1.10.5-1) ... 2026-02-25T12:00:28.710 INFO:teuthology.orchestra.run.trial117.stdout:Setting up libfuse2:amd64 (2.9.9-5ubuntu3) ... 2026-02-25T12:00:28.711 INFO:teuthology.orchestra.run.trial117.stdout:Setting up libpcre2-16-0:amd64 (10.39-3ubuntu0.1) ... 2026-02-25T12:00:28.711 INFO:teuthology.orchestra.run.trial117.stdout:Setting up python3-psutil (5.9.0-1build1) ... 2026-02-25T12:00:28.830 INFO:teuthology.orchestra.run.trial117.stdout:Setting up python3-natsort (8.0.2-1) ... 2026-02-25T12:00:28.911 INFO:teuthology.orchestra.run.trial117.stdout:Setting up libcephfs-proxy2 (20.3.0-5580-g506b402a-1jammy) ... 2026-02-25T12:00:28.912 INFO:teuthology.orchestra.run.trial117.stdout:Setting up python3-routes (2.5.1-1ubuntu1) ... 2026-02-25T12:00:28.989 INFO:teuthology.orchestra.run.trial117.stdout:Setting up python3-simplejson (3.17.6-1build1) ... 2026-02-25T12:00:29.078 INFO:teuthology.orchestra.run.trial117.stdout:Setting up python3-pygments (2.11.2+dfsg-2ubuntu0.1) ... 2026-02-25T12:00:29.323 INFO:teuthology.orchestra.run.trial117.stdout:Setting up qttranslations5-l10n (5.15.3-1) ... 2026-02-25T12:00:29.323 INFO:teuthology.orchestra.run.trial117.stdout:Setting up python3-wcwidth (0.2.5+dfsg1-1) ... 2026-02-25T12:00:29.416 INFO:teuthology.orchestra.run.trial117.stdout:Setting up python3-asyncssh (2.5.0-1ubuntu0.1) ... 2026-02-25T12:00:29.540 INFO:teuthology.orchestra.run.trial117.stdout:Setting up python3-cheroot (8.5.2+ds1-1ubuntu3.1) ... 2026-02-25T12:00:29.621 INFO:teuthology.orchestra.run.trial117.stdout:Setting up python3-jaraco.text (3.6.0-2) ... 2026-02-25T12:00:29.693 INFO:teuthology.orchestra.run.trial117.stdout:Setting up socat (1.7.4.1-3ubuntu4) ... 2026-02-25T12:00:29.694 INFO:teuthology.orchestra.run.trial117.stdout:Setting up python3-ceph-common (20.3.0-5580-g506b402a-1jammy) ... 2026-02-25T12:00:29.784 INFO:teuthology.orchestra.run.trial117.stdout:Setting up python3-sklearn (0.23.2-5ubuntu6) ... 2026-02-25T12:00:30.278 INFO:teuthology.orchestra.run.trial117.stdout:Setting up libqt5core5a:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2026-02-25T12:00:30.282 INFO:teuthology.orchestra.run.trial117.stdout:Setting up python3-toml (0.10.2-1) ... 2026-02-25T12:00:30.358 INFO:teuthology.orchestra.run.trial117.stdout:Setting up librdkafka1:amd64 (1.8.0-1build1) ... 2026-02-25T12:00:30.359 INFO:teuthology.orchestra.run.trial117.stdout:Setting up xmlstarlet (1.6.1-2.1) ... 2026-02-25T12:00:30.360 INFO:teuthology.orchestra.run.trial117.stdout:Setting up python3-pluggy (0.13.0-7.1) ... 2026-02-25T12:00:30.432 INFO:teuthology.orchestra.run.trial117.stdout:Setting up python3-zc.lockfile (2.0-1) ... 2026-02-25T12:00:30.508 INFO:teuthology.orchestra.run.trial117.stdout:Setting up libqt5dbus5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2026-02-25T12:00:30.509 INFO:teuthology.orchestra.run.trial117.stdout:Setting up python3-rsa (4.8-1) ... 2026-02-25T12:00:30.586 INFO:teuthology.orchestra.run.trial117.stdout:Setting up python3-tempora (4.1.2-1) ... 2026-02-25T12:00:30.658 INFO:teuthology.orchestra.run.trial117.stdout:Setting up python3-prettytable (2.5.0-2) ... 2026-02-25T12:00:30.735 INFO:teuthology.orchestra.run.trial117.stdout:Setting up liblttng-ust1:amd64 (2.13.1-1ubuntu1) ... 2026-02-25T12:00:30.736 INFO:teuthology.orchestra.run.trial117.stdout:Setting up python3-websocket (1.2.3-1) ... 2026-02-25T12:00:30.816 INFO:teuthology.orchestra.run.trial117.stdout:Setting up libonig5:amd64 (6.9.7.1-2build1) ... 2026-02-25T12:00:30.817 INFO:teuthology.orchestra.run.trial117.stdout:Setting up python3-requests-oauthlib (1.3.0+ds-0.1) ... 2026-02-25T12:00:30.893 INFO:teuthology.orchestra.run.trial117.stdout:Setting up python3-webob (1:1.8.6-1.1ubuntu0.1) ... 2026-02-25T12:00:30.976 INFO:teuthology.orchestra.run.trial117.stdout:Setting up python3-jaraco.collections (3.4.0-2) ... 2026-02-25T12:00:31.051 INFO:teuthology.orchestra.run.trial117.stdout:Setting up libjq1:amd64 (1.6-2.1ubuntu3.1) ... 2026-02-25T12:00:31.052 INFO:teuthology.orchestra.run.trial117.stdout:Setting up python3-pytest (6.2.5-1ubuntu2) ... 2026-02-25T12:00:31.182 INFO:teuthology.orchestra.run.trial117.stdout:Setting up python3-portend (3.0.0-1) ... 2026-02-25T12:00:31.253 INFO:teuthology.orchestra.run.trial117.stdout:Setting up libqt5network5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2026-02-25T12:00:31.254 INFO:teuthology.orchestra.run.trial117.stdout:Setting up python3-google-auth (1.5.1-3) ... 2026-02-25T12:00:31.337 INFO:teuthology.orchestra.run.trial117.stdout:Setting up jq (1.6-2.1ubuntu3.1) ... 2026-02-25T12:00:31.338 INFO:teuthology.orchestra.run.trial117.stdout:Setting up python3-cherrypy3 (18.6.1-4) ... 2026-02-25T12:00:31.471 INFO:teuthology.orchestra.run.trial117.stdout:Setting up libthrift-0.16.0:amd64 (0.16.0-2) ... 2026-02-25T12:00:31.472 INFO:teuthology.orchestra.run.trial117.stdout:Setting up librados2 (20.3.0-5580-g506b402a-1jammy) ... 2026-02-25T12:00:31.473 INFO:teuthology.orchestra.run.trial117.stdout:Setting up librgw2 (20.3.0-5580-g506b402a-1jammy) ... 2026-02-25T12:00:31.474 INFO:teuthology.orchestra.run.trial117.stdout:Setting up libsqlite3-mod-ceph (20.3.0-5580-g506b402a-1jammy) ... 2026-02-25T12:00:31.475 INFO:teuthology.orchestra.run.trial117.stdout:Setting up python3-kubernetes (12.0.1-1ubuntu1) ... 2026-02-25T12:00:31.987 INFO:teuthology.orchestra.run.trial117.stdout:Setting up libcephfs2 (20.3.0-5580-g506b402a-1jammy) ... 2026-02-25T12:00:31.988 INFO:teuthology.orchestra.run.trial117.stdout:Setting up libradosstriper1 (20.3.0-5580-g506b402a-1jammy) ... 2026-02-25T12:00:31.989 INFO:teuthology.orchestra.run.trial117.stdout:Setting up librbd1 (20.3.0-5580-g506b402a-1jammy) ... 2026-02-25T12:00:31.990 INFO:teuthology.orchestra.run.trial117.stdout:Setting up ceph-mgr-modules-core (20.3.0-5580-g506b402a-1jammy) ... 2026-02-25T12:00:31.991 INFO:teuthology.orchestra.run.trial117.stdout:Setting up ceph-fuse (20.3.0-5580-g506b402a-1jammy) ... 2026-02-25T12:00:32.061 INFO:teuthology.orchestra.run.trial117.stdout:Created symlink /etc/systemd/system/remote-fs.target.wants/ceph-fuse.target → /lib/systemd/system/ceph-fuse.target. 2026-02-25T12:00:32.062 INFO:teuthology.orchestra.run.trial117.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-fuse.target → /lib/systemd/system/ceph-fuse.target. 2026-02-25T12:00:32.452 INFO:teuthology.orchestra.run.trial117.stdout:Setting up libcephfs-dev (20.3.0-5580-g506b402a-1jammy) ... 2026-02-25T12:00:32.453 INFO:teuthology.orchestra.run.trial117.stdout:Setting up python3-rados (20.3.0-5580-g506b402a-1jammy) ... 2026-02-25T12:00:32.454 INFO:teuthology.orchestra.run.trial117.stdout:Setting up libcephfs-daemon (20.3.0-5580-g506b402a-1jammy) ... 2026-02-25T12:00:32.455 INFO:teuthology.orchestra.run.trial117.stdout:Setting up python3-rbd (20.3.0-5580-g506b402a-1jammy) ... 2026-02-25T12:00:32.456 INFO:teuthology.orchestra.run.trial117.stdout:Setting up rbd-fuse (20.3.0-5580-g506b402a-1jammy) ... 2026-02-25T12:00:32.457 INFO:teuthology.orchestra.run.trial117.stdout:Setting up python3-rgw (20.3.0-5580-g506b402a-1jammy) ... 2026-02-25T12:00:32.458 INFO:teuthology.orchestra.run.trial117.stdout:Setting up python3-cephfs (20.3.0-5580-g506b402a-1jammy) ... 2026-02-25T12:00:32.459 INFO:teuthology.orchestra.run.trial117.stdout:Setting up ceph-common (20.3.0-5580-g506b402a-1jammy) ... 2026-02-25T12:00:32.486 INFO:teuthology.orchestra.run.trial117.stdout:Adding group ceph....done 2026-02-25T12:00:32.522 INFO:teuthology.orchestra.run.trial117.stdout:Adding system user ceph....done 2026-02-25T12:00:32.532 INFO:teuthology.orchestra.run.trial117.stdout:Setting system user ceph properties....done 2026-02-25T12:00:32.538 INFO:teuthology.orchestra.run.trial117.stdout:chown: cannot access '/var/log/ceph/*.log*': No such file or directory 2026-02-25T12:00:32.614 INFO:teuthology.orchestra.run.trial117.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /lib/systemd/system/ceph.target. 2026-02-25T12:00:32.905 INFO:teuthology.orchestra.run.trial117.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/rbdmap.service → /lib/systemd/system/rbdmap.service. 2026-02-25T12:00:33.293 INFO:teuthology.orchestra.run.trial117.stdout:Setting up ceph-test (20.3.0-5580-g506b402a-1jammy) ... 2026-02-25T12:00:33.294 INFO:teuthology.orchestra.run.trial117.stdout:Setting up radosgw (20.3.0-5580-g506b402a-1jammy) ... 2026-02-25T12:00:33.544 INFO:teuthology.orchestra.run.trial117.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /lib/systemd/system/ceph-radosgw.target. 2026-02-25T12:00:33.544 INFO:teuthology.orchestra.run.trial117.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /lib/systemd/system/ceph-radosgw.target. 2026-02-25T12:00:33.928 INFO:teuthology.orchestra.run.trial117.stdout:Setting up ceph-base (20.3.0-5580-g506b402a-1jammy) ... 2026-02-25T12:00:34.045 INFO:teuthology.orchestra.run.trial117.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /lib/systemd/system/ceph-crash.service. 2026-02-25T12:00:34.418 INFO:teuthology.orchestra.run.trial117.stdout:Setting up ceph-mds (20.3.0-5580-g506b402a-1jammy) ... 2026-02-25T12:00:34.478 INFO:teuthology.orchestra.run.trial117.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /lib/systemd/system/ceph-mds.target. 2026-02-25T12:00:34.478 INFO:teuthology.orchestra.run.trial117.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /lib/systemd/system/ceph-mds.target. 2026-02-25T12:00:34.812 INFO:teuthology.orchestra.run.trial117.stdout:Setting up ceph-mgr (20.3.0-5580-g506b402a-1jammy) ... 2026-02-25T12:00:34.892 INFO:teuthology.orchestra.run.trial117.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /lib/systemd/system/ceph-mgr.target. 2026-02-25T12:00:34.892 INFO:teuthology.orchestra.run.trial117.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /lib/systemd/system/ceph-mgr.target. 2026-02-25T12:00:35.247 INFO:teuthology.orchestra.run.trial117.stdout:Setting up ceph-mgr-k8sevents (20.3.0-5580-g506b402a-1jammy) ... 2026-02-25T12:00:35.248 INFO:teuthology.orchestra.run.trial117.stdout:Setting up ceph-osd-classic (20.3.0-5580-g506b402a-1jammy) ... 2026-02-25T12:00:35.265 INFO:teuthology.orchestra.run.trial117.stdout:update-alternatives: using /usr/bin/ceph-osd-classic to provide /usr/bin/ceph-osd (ceph-osd) in auto mode 2026-02-25T12:00:35.275 INFO:teuthology.orchestra.run.trial117.stdout:Setting up ceph-mgr-diskprediction-local (20.3.0-5580-g506b402a-1jammy) ... 2026-02-25T12:00:35.285 INFO:teuthology.orchestra.run.trial117.stdout:Setting up ceph-mon (20.3.0-5580-g506b402a-1jammy) ... 2026-02-25T12:00:35.357 INFO:teuthology.orchestra.run.trial117.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /lib/systemd/system/ceph-mon.target. 2026-02-25T12:00:35.357 INFO:teuthology.orchestra.run.trial117.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /lib/systemd/system/ceph-mon.target. 2026-02-25T12:00:35.735 INFO:teuthology.orchestra.run.trial117.stdout:Setting up ceph-mgr-cephadm (20.3.0-5580-g506b402a-1jammy) ... 2026-02-25T12:00:35.746 INFO:teuthology.orchestra.run.trial117.stdout:Setting up ceph-mgr-dashboard (20.3.0-5580-g506b402a-1jammy) ... 2026-02-25T12:00:35.757 INFO:teuthology.orchestra.run.trial117.stdout:Setting up ceph-osd (20.3.0-5580-g506b402a-1jammy) ... 2026-02-25T12:00:35.833 INFO:teuthology.orchestra.run.trial117.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /lib/systemd/system/ceph-osd.target. 2026-02-25T12:00:35.833 INFO:teuthology.orchestra.run.trial117.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /lib/systemd/system/ceph-osd.target. 2026-02-25T12:00:36.263 INFO:teuthology.orchestra.run.trial117.stdout:Setting up ceph (20.3.0-5580-g506b402a-1jammy) ... 2026-02-25T12:00:36.264 INFO:teuthology.orchestra.run.trial117.stdout:Setting up ceph-volume (20.3.0-5580-g506b402a-1jammy) ... 2026-02-25T12:00:36.369 INFO:teuthology.orchestra.run.trial117.stdout:Processing triggers for man-db (2.10.2-1) ... 2026-02-25T12:00:36.453 INFO:teuthology.orchestra.run.trial117.stdout:Processing triggers for libc-bin (2.35-0ubuntu3.13) ... 2026-02-25T12:00:36.908 INFO:teuthology.orchestra.run.trial117.stdout: 2026-02-25T12:00:36.908 INFO:teuthology.orchestra.run.trial117.stdout:Running kernel seems to be up-to-date. 2026-02-25T12:00:36.908 INFO:teuthology.orchestra.run.trial117.stdout: 2026-02-25T12:00:36.909 INFO:teuthology.orchestra.run.trial117.stdout:The processor microcode seems to be up-to-date. 2026-02-25T12:00:36.909 INFO:teuthology.orchestra.run.trial117.stdout: 2026-02-25T12:00:36.909 INFO:teuthology.orchestra.run.trial117.stdout:No services need to be restarted. 2026-02-25T12:00:36.909 INFO:teuthology.orchestra.run.trial117.stdout: 2026-02-25T12:00:36.909 INFO:teuthology.orchestra.run.trial117.stdout:No containers need to be restarted. 2026-02-25T12:00:36.909 INFO:teuthology.orchestra.run.trial117.stdout: 2026-02-25T12:00:36.909 INFO:teuthology.orchestra.run.trial117.stdout:No user sessions are running outdated binaries. 2026-02-25T12:00:36.909 INFO:teuthology.orchestra.run.trial117.stdout: 2026-02-25T12:00:36.909 INFO:teuthology.orchestra.run.trial117.stdout:No VM guests are running outdated hypervisor (qemu) binaries on this host. 2026-02-25T12:00:37.732 INFO:teuthology.orchestra.run.trial117.stderr:W: --force-yes is deprecated, use one of the options starting with --allow instead. 2026-02-25T12:00:37.739 DEBUG:teuthology.parallel:result is None 2026-02-25T12:00:37.739 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&sha1=506b402a3d3a240d0bfe073b4769eaffdf84497b 2026-02-25T12:00:42.917 DEBUG:teuthology.orchestra.run.trial117:> dpkg-query -W -f '${Version}' ceph 2026-02-25T12:00:42.928 INFO:teuthology.orchestra.run.trial117.stdout:20.3.0-5580-g506b402a-1jammy 2026-02-25T12:00:42.929 INFO:teuthology.packaging:The installed version of ceph is 20.3.0-5580-g506b402a-1jammy 2026-02-25T12:00:42.929 INFO:teuthology.task.install:The correct ceph version 20.3.0-5580-g506b402a-1jammy is installed. 2026-02-25T12:00:42.931 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2026-02-25T12:00:42.932 DEBUG:teuthology.orchestra.run.trial117:> set -ex 2026-02-25T12:00:42.932 DEBUG:teuthology.orchestra.run.trial117:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2026-02-25T12:00:42.978 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2026-02-25T12:00:42.978 DEBUG:teuthology.orchestra.run.trial117:> set -ex 2026-02-25T12:00:42.978 DEBUG:teuthology.orchestra.run.trial117:> sudo dd of=/usr/bin/daemon-helper 2026-02-25T12:00:43.029 DEBUG:teuthology.orchestra.run.trial117:> sudo chmod a=rx -- /usr/bin/daemon-helper 2026-02-25T12:00:43.079 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2026-02-25T12:00:43.079 DEBUG:teuthology.orchestra.run.trial117:> set -ex 2026-02-25T12:00:43.080 DEBUG:teuthology.orchestra.run.trial117:> sudo dd of=/usr/bin/adjust-ulimits 2026-02-25T12:00:43.129 DEBUG:teuthology.orchestra.run.trial117:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2026-02-25T12:00:43.179 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2026-02-25T12:00:43.180 DEBUG:teuthology.orchestra.run.trial117:> set -ex 2026-02-25T12:00:43.180 DEBUG:teuthology.orchestra.run.trial117:> sudo dd of=/usr/bin/stdin-killer 2026-02-25T12:00:43.230 DEBUG:teuthology.orchestra.run.trial117:> sudo chmod a=rx -- /usr/bin/stdin-killer 2026-02-25T12:00:43.280 INFO:teuthology.run_tasks:Running task ceph... 2026-02-25T12:00:43.400 INFO:tasks.ceph:Making ceph log dir writeable by non-root... 2026-02-25T12:00:43.400 DEBUG:teuthology.orchestra.run.trial117:> sudo chmod 777 /var/log/ceph 2026-02-25T12:00:43.408 INFO:tasks.ceph:Disabling ceph logrotate... 2026-02-25T12:00:43.408 DEBUG:teuthology.orchestra.run.trial117:> sudo rm -f -- /etc/logrotate.d/ceph 2026-02-25T12:00:43.459 INFO:tasks.ceph:Creating extra log directories... 2026-02-25T12:00:43.459 DEBUG:teuthology.orchestra.run.trial117:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2026-02-25T12:00:43.519 INFO:tasks.ceph:Creating ceph cluster ceph... 2026-02-25T12:00:43.519 INFO:tasks.ceph:config {'conf': {'osd': {'osd max object name len': 400, 'osd max object namespace len': 64, 'bluestore zero block detection': True, 'debug ms': 1, 'debug osd': 20, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd mclock override recovery settings': True, 'osd mclock profile': 'high_recovery_ops', 'osd mclock skip benchmark': True, 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random'}, 'global': {'mon election default strategy': 3}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 30, 'debug ms': 1, 'debug paxos': 20, 'mon scrub interval': 300}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['but it is still running', 'overall HEALTH_', '\\(OSDMAP_FLAGS\\)', '\\(OSD_', '\\(PG_', '\\(POOL_APP_NOT_ENABLED\\)', '\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)'], 'cpu_profile': set(), 'cluster': 'ceph', 'mon_bind_msgr2': True, 'mon_bind_addrvec': True} 2026-02-25T12:00:43.519 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/mnt/teuthology/nmordech-2026-02-25_11:36:23-rados-wip-rocky10-branch-of-the-day-2026-02-24-1771941190-distro-default-trial/69872', 'branch': 'wip-rocky10-branch-of-the-day-2026-02-24-1771941190', 'description': 'rados/singleton-nomsgr/{all/health-warnings mon_election/connectivity rados supported-random-distro$/{ubuntu_latest}}', 'email': None, 'first_in_suite': False, 'flavor': 'default', 'job_id': '69872', 'kernel': {'branch': 'distro', 'kdb': 1, 'sha1': 'distro'}, 'ktype': 'distro', 'last_in_suite': False, 'machine_type': 'trial', 'name': 'nmordech-2026-02-25_11:36:23-rados-wip-rocky10-branch-of-the-day-2026-02-24-1771941190-distro-default-trial', 'no_nested_subset': False, 'os_type': 'ubuntu', 'os_version': '22.04', 'overrides': {'admin_socket': {'branch': 'wip-rocky10-branch-of-the-day-2026-02-24-1771941190'}, 'ceph': {'conf': {'global': {'mon election default strategy': 3}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 30, 'debug ms': 1, 'debug paxos': 20, 'mon scrub interval': 300}, 'osd': {'bluestore zero block detection': True, 'debug ms': 1, 'debug osd': 20, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd mclock override recovery settings': True, 'osd mclock profile': 'high_recovery_ops', 'osd mclock skip benchmark': True, 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random'}}, 'flavor': 'default', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)'], 'sha1': '506b402a3d3a240d0bfe073b4769eaffdf84497b'}, 'ceph-deploy': {'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {}}}, 'cephadm': {'distro-suffix': 'rockylinux-10'}, 'install': {'ceph': {'flavor': 'default', 'sha1': '506b402a3d3a240d0bfe073b4769eaffdf84497b'}}, 'workunit': {'branch': 'wip-rocky10-branch-of-the-day-2026-02-24-1771941190', 'sha1': '506b402a3d3a240d0bfe073b4769eaffdf84497b'}}, 'owner': 'scheduled_nmordech@soko04.front.sepia.ceph.com', 'priority': 10, 'repo': 'https://git.ceph.com/ceph-ci.git', 'roles': [['mon.a', 'mgr.x', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'osd.8', 'osd.9', 'client.0']], 'seed': 317, 'sha1': '506b402a3d3a240d0bfe073b4769eaffdf84497b', 'sleep_before_teardown': 0, 'subset': '111/120000', 'suite': 'rados', 'suite_branch': 'wip-rocky10-branch-of-the-day-2026-02-24-1771941190', 'suite_path': '/home/teuthworker/src/git.ceph.com_ceph-c_506b402a3d3a240d0bfe073b4769eaffdf84497b/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://git.ceph.com/ceph-ci.git', 'suite_sha1': '506b402a3d3a240d0bfe073b4769eaffdf84497b', 'targets': {'trial117.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBJFfl9t2An17wPEskxXrqDLtD3lvUT6PBH8Egb5x1vc3SHIZuO4Xga9Z9XZIdDLcmbBOFUwlEWs56vvPpXmd+3E='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'branch': 'distro', 'kdb': 1, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': None}, {'ceph': {'conf': {'osd': {'osd max object name len': 400, 'osd max object namespace len': 64, 'bluestore zero block detection': True, 'debug ms': 1, 'debug osd': 20, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd mclock override recovery settings': True, 'osd mclock profile': 'high_recovery_ops', 'osd mclock skip benchmark': True, 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random'}, 'global': {'mon election default strategy': 3}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 30, 'debug ms': 1, 'debug paxos': 20, 'mon scrub interval': 300}}, 'log-ignorelist': ['but it is still running', 'overall HEALTH_', '\\(OSDMAP_FLAGS\\)', '\\(OSD_', '\\(PG_', '\\(POOL_APP_NOT_ENABLED\\)', '\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)'], 'pre-mgr-commands': ['sudo ceph config set mgr mgr_pool false --force'], 'flavor': 'default', 'sha1': '506b402a3d3a240d0bfe073b4769eaffdf84497b', 'cluster': 'ceph'}}, {'workunit': {'clients': {'all': ['rados/test_health_warnings.sh']}}}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': []}, 'teuthology_branch': 'main', 'teuthology_sha1': 'fa17720d0088c3ac28e473468bfc79eeaff5cd38', 'timestamp': '2026-02-25_11:36:23', 'tube': 'trial', 'user': 'nmordech', 'verbose': False, 'worker_log': '/home/teuthworker/mnt/teuthology/worker_logs/dispatcher.trial.904699'} 2026-02-25T12:00:43.520 DEBUG:teuthology.orchestra.run.trial117:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2026-02-25T12:00:43.557 DEBUG:teuthology.orchestra.run.trial117:> sudo install -d -m0777 -- /var/run/ceph 2026-02-25T12:00:43.610 DEBUG:teuthology.orchestra.run.trial117:> set -ex 2026-02-25T12:00:43.610 DEBUG:teuthology.orchestra.run.trial117:> dd if=/scratch_devs of=/dev/stdout 2026-02-25T12:00:43.657 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2026-02-25T12:00:43.657 DEBUG:teuthology.orchestra.run.trial117:> stat /dev/vg_nvme/lv_1 2026-02-25T12:00:43.705 INFO:teuthology.orchestra.run.trial117.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2026-02-25T12:00:43.705 INFO:teuthology.orchestra.run.trial117.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2026-02-25T12:00:43.705 INFO:teuthology.orchestra.run.trial117.stdout:Device: 5h/5d Inode: 892 Links: 1 2026-02-25T12:00:43.705 INFO:teuthology.orchestra.run.trial117.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2026-02-25T12:00:43.705 INFO:teuthology.orchestra.run.trial117.stdout:Access: 2026-02-25 11:59:41.218586576 +0000 2026-02-25T12:00:43.706 INFO:teuthology.orchestra.run.trial117.stdout:Modify: 2026-02-25 11:59:41.070589091 +0000 2026-02-25T12:00:43.706 INFO:teuthology.orchestra.run.trial117.stdout:Change: 2026-02-25 11:59:41.070589091 +0000 2026-02-25T12:00:43.706 INFO:teuthology.orchestra.run.trial117.stdout: Birth: - 2026-02-25T12:00:43.706 DEBUG:teuthology.orchestra.run.trial117:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2026-02-25T12:00:43.756 INFO:teuthology.orchestra.run.trial117.stderr:1+0 records in 2026-02-25T12:00:43.756 INFO:teuthology.orchestra.run.trial117.stderr:1+0 records out 2026-02-25T12:00:43.756 INFO:teuthology.orchestra.run.trial117.stderr:512 bytes copied, 0.000191857 s, 2.7 MB/s 2026-02-25T12:00:43.758 DEBUG:teuthology.orchestra.run.trial117:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2026-02-25T12:00:43.806 DEBUG:teuthology.orchestra.run.trial117:> stat /dev/vg_nvme/lv_2 2026-02-25T12:00:43.852 INFO:teuthology.orchestra.run.trial117.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2026-02-25T12:00:43.853 INFO:teuthology.orchestra.run.trial117.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2026-02-25T12:00:43.853 INFO:teuthology.orchestra.run.trial117.stdout:Device: 5h/5d Inode: 917 Links: 1 2026-02-25T12:00:43.853 INFO:teuthology.orchestra.run.trial117.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2026-02-25T12:00:43.853 INFO:teuthology.orchestra.run.trial117.stdout:Access: 2026-02-25 11:59:41.386583718 +0000 2026-02-25T12:00:43.853 INFO:teuthology.orchestra.run.trial117.stdout:Modify: 2026-02-25 11:59:41.382583788 +0000 2026-02-25T12:00:43.853 INFO:teuthology.orchestra.run.trial117.stdout:Change: 2026-02-25 11:59:41.382583788 +0000 2026-02-25T12:00:43.853 INFO:teuthology.orchestra.run.trial117.stdout: Birth: - 2026-02-25T12:00:43.853 DEBUG:teuthology.orchestra.run.trial117:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2026-02-25T12:00:43.903 INFO:teuthology.orchestra.run.trial117.stderr:1+0 records in 2026-02-25T12:00:43.904 INFO:teuthology.orchestra.run.trial117.stderr:1+0 records out 2026-02-25T12:00:43.904 INFO:teuthology.orchestra.run.trial117.stderr:512 bytes copied, 0.000178191 s, 2.9 MB/s 2026-02-25T12:00:43.905 DEBUG:teuthology.orchestra.run.trial117:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2026-02-25T12:00:43.954 DEBUG:teuthology.orchestra.run.trial117:> stat /dev/vg_nvme/lv_3 2026-02-25T12:00:44.000 INFO:teuthology.orchestra.run.trial117.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2026-02-25T12:00:44.000 INFO:teuthology.orchestra.run.trial117.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2026-02-25T12:00:44.000 INFO:teuthology.orchestra.run.trial117.stdout:Device: 5h/5d Inode: 938 Links: 1 2026-02-25T12:00:44.000 INFO:teuthology.orchestra.run.trial117.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2026-02-25T12:00:44.000 INFO:teuthology.orchestra.run.trial117.stdout:Access: 2026-02-25 11:59:41.918574676 +0000 2026-02-25T12:00:44.000 INFO:teuthology.orchestra.run.trial117.stdout:Modify: 2026-02-25 11:59:41.730577872 +0000 2026-02-25T12:00:44.000 INFO:teuthology.orchestra.run.trial117.stdout:Change: 2026-02-25 11:59:41.730577872 +0000 2026-02-25T12:00:44.000 INFO:teuthology.orchestra.run.trial117.stdout: Birth: - 2026-02-25T12:00:44.000 DEBUG:teuthology.orchestra.run.trial117:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2026-02-25T12:00:44.051 INFO:teuthology.orchestra.run.trial117.stderr:1+0 records in 2026-02-25T12:00:44.052 INFO:teuthology.orchestra.run.trial117.stderr:1+0 records out 2026-02-25T12:00:44.052 INFO:teuthology.orchestra.run.trial117.stderr:512 bytes copied, 0.000186857 s, 2.7 MB/s 2026-02-25T12:00:44.053 DEBUG:teuthology.orchestra.run.trial117:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2026-02-25T12:00:44.104 DEBUG:teuthology.orchestra.run.trial117:> stat /dev/vg_nvme/lv_4 2026-02-25T12:00:44.151 INFO:teuthology.orchestra.run.trial117.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2026-02-25T12:00:44.151 INFO:teuthology.orchestra.run.trial117.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2026-02-25T12:00:44.151 INFO:teuthology.orchestra.run.trial117.stdout:Device: 5h/5d Inode: 963 Links: 1 2026-02-25T12:00:44.151 INFO:teuthology.orchestra.run.trial117.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2026-02-25T12:00:44.151 INFO:teuthology.orchestra.run.trial117.stdout:Access: 2026-02-25 11:59:42.266568760 +0000 2026-02-25T12:00:44.151 INFO:teuthology.orchestra.run.trial117.stdout:Modify: 2026-02-25 11:59:42.086571818 +0000 2026-02-25T12:00:44.151 INFO:teuthology.orchestra.run.trial117.stdout:Change: 2026-02-25 11:59:42.086571818 +0000 2026-02-25T12:00:44.151 INFO:teuthology.orchestra.run.trial117.stdout: Birth: - 2026-02-25T12:00:44.152 DEBUG:teuthology.orchestra.run.trial117:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2026-02-25T12:00:44.203 INFO:teuthology.orchestra.run.trial117.stderr:1+0 records in 2026-02-25T12:00:44.203 INFO:teuthology.orchestra.run.trial117.stderr:1+0 records out 2026-02-25T12:00:44.203 INFO:teuthology.orchestra.run.trial117.stderr:512 bytes copied, 0.000176989 s, 2.9 MB/s 2026-02-25T12:00:44.203 DEBUG:teuthology.orchestra.run.trial117:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2026-02-25T12:00:44.249 INFO:tasks.ceph:osd dev map: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2026-02-25T12:00:44.249 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@trial117.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'}} 2026-02-25T12:00:44.249 INFO:tasks.ceph:Generating config... 2026-02-25T12:00:44.250 INFO:tasks.ceph:[osd] osd max object name len = 400 2026-02-25T12:00:44.250 INFO:tasks.ceph:[osd] osd max object namespace len = 64 2026-02-25T12:00:44.250 INFO:tasks.ceph:[osd] bluestore zero block detection = True 2026-02-25T12:00:44.250 INFO:tasks.ceph:[osd] debug ms = 1 2026-02-25T12:00:44.250 INFO:tasks.ceph:[osd] debug osd = 20 2026-02-25T12:00:44.250 INFO:tasks.ceph:[osd] osd debug verify cached snaps = True 2026-02-25T12:00:44.250 INFO:tasks.ceph:[osd] osd debug verify missing on start = True 2026-02-25T12:00:44.250 INFO:tasks.ceph:[osd] osd mclock override recovery settings = True 2026-02-25T12:00:44.250 INFO:tasks.ceph:[osd] osd mclock profile = high_recovery_ops 2026-02-25T12:00:44.250 INFO:tasks.ceph:[osd] osd mclock skip benchmark = True 2026-02-25T12:00:44.250 INFO:tasks.ceph:[osd] osd op queue = debug_random 2026-02-25T12:00:44.250 INFO:tasks.ceph:[osd] osd op queue cut off = debug_random 2026-02-25T12:00:44.250 INFO:tasks.ceph:[global] mon election default strategy = 3 2026-02-25T12:00:44.250 INFO:tasks.ceph:[mgr] debug mgr = 20 2026-02-25T12:00:44.250 INFO:tasks.ceph:[mgr] debug ms = 1 2026-02-25T12:00:44.250 INFO:tasks.ceph:[mon] debug mon = 30 2026-02-25T12:00:44.250 INFO:tasks.ceph:[mon] debug ms = 1 2026-02-25T12:00:44.250 INFO:tasks.ceph:[mon] debug paxos = 20 2026-02-25T12:00:44.250 INFO:tasks.ceph:[mon] mon scrub interval = 300 2026-02-25T12:00:44.250 INFO:tasks.ceph:Setting up mon.a... 2026-02-25T12:00:44.250 DEBUG:teuthology.orchestra.run.trial117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/ceph.keyring 2026-02-25T12:00:44.312 INFO:teuthology.orchestra.run.trial117.stdout:creating /etc/ceph/ceph.keyring 2026-02-25T12:00:44.316 DEBUG:teuthology.orchestra.run.trial117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/ceph.keyring 2026-02-25T12:00:44.375 DEBUG:teuthology.orchestra.run.trial117:> sudo chmod 0644 /etc/ceph/ceph.keyring 2026-02-25T12:00:44.424 DEBUG:tasks.ceph:Ceph mon addresses: [('mon.a', '10.20.193.117')] 2026-02-25T12:00:44.425 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'cephx', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=isa technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '10.20.193.117', 'mon election default strategy': 3}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd mclock profile': 'high_recovery_ops', 'osd mclock skip benchmark': True, 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd debug trim objects': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'osd max object name len': 400, 'osd max object namespace len': 64, 'bluestore zero block detection': True, 'debug ms': 1, 'debug osd': 20, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd mclock override recovery settings': True, 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random'}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false'}, 'mon': {'debug ms': 1, 'debug mon': 30, 'debug paxos': 20, 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'mon_down_mkfs_grace': '2m', 'mon_warn_on_filestore_osds': 'false', 'mon scrub interval': 300}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok'}, 'mon.a': {}} 2026-02-25T12:00:44.426 DEBUG:teuthology.orchestra.run.trial117:> set -ex 2026-02-25T12:00:44.426 DEBUG:teuthology.orchestra.run.trial117:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2026-02-25T12:00:44.468 DEBUG:teuthology.orchestra.run.trial117:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --add a 10.20.193.117 --print /home/ubuntu/cephtest/ceph.monmap 2026-02-25T12:00:44.525 INFO:teuthology.orchestra.run.trial117.stdout:monmaptool: monmap file /home/ubuntu/cephtest/ceph.monmap 2026-02-25T12:00:44.525 INFO:teuthology.orchestra.run.trial117.stdout:monmaptool: generated fsid 43e63318-add1-4399-a1fc-80188594353b 2026-02-25T12:00:44.525 INFO:teuthology.orchestra.run.trial117.stdout:setting min_mon_release = tentacle 2026-02-25T12:00:44.525 INFO:teuthology.orchestra.run.trial117.stdout:epoch 0 2026-02-25T12:00:44.525 INFO:teuthology.orchestra.run.trial117.stdout:fsid 43e63318-add1-4399-a1fc-80188594353b 2026-02-25T12:00:44.525 INFO:teuthology.orchestra.run.trial117.stdout:last_changed 2026-02-25T12:00:44.524101+0000 2026-02-25T12:00:44.525 INFO:teuthology.orchestra.run.trial117.stdout:created 2026-02-25T12:00:44.524101+0000 2026-02-25T12:00:44.526 INFO:teuthology.orchestra.run.trial117.stdout:min_mon_release 20 (tentacle) 2026-02-25T12:00:44.526 INFO:teuthology.orchestra.run.trial117.stdout:election_strategy: 3 2026-02-25T12:00:44.526 INFO:teuthology.orchestra.run.trial117.stdout:0: [v2:10.20.193.117:3300/0,v1:10.20.193.117:6789/0] mon.a 2026-02-25T12:00:44.526 INFO:teuthology.orchestra.run.trial117.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/ceph.monmap (1 monitors) 2026-02-25T12:00:44.527 DEBUG:teuthology.orchestra.run.trial117:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2026-02-25T12:00:44.572 INFO:tasks.ceph:Writing /etc/ceph/ceph.conf for FSID 43e63318-add1-4399-a1fc-80188594353b... 2026-02-25T12:00:44.573 DEBUG:teuthology.orchestra.run.trial117:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2026-02-25T12:00:44.632 INFO:teuthology.orchestra.run.trial117.stdout:[global] 2026-02-25T12:00:44.632 INFO:teuthology.orchestra.run.trial117.stdout: chdir = "" 2026-02-25T12:00:44.632 INFO:teuthology.orchestra.run.trial117.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2026-02-25T12:00:44.632 INFO:teuthology.orchestra.run.trial117.stdout: auth supported = cephx 2026-02-25T12:00:44.632 INFO:teuthology.orchestra.run.trial117.stdout: 2026-02-25T12:00:44.632 INFO:teuthology.orchestra.run.trial117.stdout: filestore xattr use omap = true 2026-02-25T12:00:44.632 INFO:teuthology.orchestra.run.trial117.stdout: 2026-02-25T12:00:44.632 INFO:teuthology.orchestra.run.trial117.stdout: mon clock drift allowed = 1.000 2026-02-25T12:00:44.632 INFO:teuthology.orchestra.run.trial117.stdout: 2026-02-25T12:00:44.632 INFO:teuthology.orchestra.run.trial117.stdout: osd crush chooseleaf type = 0 2026-02-25T12:00:44.632 INFO:teuthology.orchestra.run.trial117.stdout: auth debug = true 2026-02-25T12:00:44.632 INFO:teuthology.orchestra.run.trial117.stdout: 2026-02-25T12:00:44.632 INFO:teuthology.orchestra.run.trial117.stdout: ms die on old message = true 2026-02-25T12:00:44.632 INFO:teuthology.orchestra.run.trial117.stdout: ms die on bug = true 2026-02-25T12:00:44.632 INFO:teuthology.orchestra.run.trial117.stdout: 2026-02-25T12:00:44.632 INFO:teuthology.orchestra.run.trial117.stdout: mon max pg per osd = 10000 # >= luminous 2026-02-25T12:00:44.632 INFO:teuthology.orchestra.run.trial117.stdout: mon pg warn max object skew = 0 2026-02-25T12:00:44.632 INFO:teuthology.orchestra.run.trial117.stdout: 2026-02-25T12:00:44.632 INFO:teuthology.orchestra.run.trial117.stdout: # disable pg_autoscaler by default for new pools 2026-02-25T12:00:44.632 INFO:teuthology.orchestra.run.trial117.stdout: osd_pool_default_pg_autoscale_mode = off 2026-02-25T12:00:44.632 INFO:teuthology.orchestra.run.trial117.stdout: 2026-02-25T12:00:44.632 INFO:teuthology.orchestra.run.trial117.stdout: osd pool default size = 2 2026-02-25T12:00:44.632 INFO:teuthology.orchestra.run.trial117.stdout: 2026-02-25T12:00:44.633 INFO:teuthology.orchestra.run.trial117.stdout: mon osd allow primary affinity = true 2026-02-25T12:00:44.633 INFO:teuthology.orchestra.run.trial117.stdout: mon osd allow pg remap = true 2026-02-25T12:00:44.633 INFO:teuthology.orchestra.run.trial117.stdout: mon warn on legacy crush tunables = false 2026-02-25T12:00:44.633 INFO:teuthology.orchestra.run.trial117.stdout: mon warn on crush straw calc version zero = false 2026-02-25T12:00:44.633 INFO:teuthology.orchestra.run.trial117.stdout: mon warn on no sortbitwise = false 2026-02-25T12:00:44.633 INFO:teuthology.orchestra.run.trial117.stdout: mon warn on osd down out interval zero = false 2026-02-25T12:00:44.633 INFO:teuthology.orchestra.run.trial117.stdout: mon warn on too few osds = false 2026-02-25T12:00:44.633 INFO:teuthology.orchestra.run.trial117.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2026-02-25T12:00:44.633 INFO:teuthology.orchestra.run.trial117.stdout: mon_warn_on_pool_no_redundancy = false 2026-02-25T12:00:44.633 INFO:teuthology.orchestra.run.trial117.stdout: mon_allow_pool_size_one = true 2026-02-25T12:00:44.633 INFO:teuthology.orchestra.run.trial117.stdout: 2026-02-25T12:00:44.633 INFO:teuthology.orchestra.run.trial117.stdout: osd pool default erasure code profile = plugin=isa technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2026-02-25T12:00:44.633 INFO:teuthology.orchestra.run.trial117.stdout: 2026-02-25T12:00:44.633 INFO:teuthology.orchestra.run.trial117.stdout: osd default data pool replay window = 5 2026-02-25T12:00:44.633 INFO:teuthology.orchestra.run.trial117.stdout: 2026-02-25T12:00:44.633 INFO:teuthology.orchestra.run.trial117.stdout: mon allow pool delete = true 2026-02-25T12:00:44.633 INFO:teuthology.orchestra.run.trial117.stdout: 2026-02-25T12:00:44.633 INFO:teuthology.orchestra.run.trial117.stdout: mon cluster log file level = debug 2026-02-25T12:00:44.633 INFO:teuthology.orchestra.run.trial117.stdout: debug asserts on shutdown = true 2026-02-25T12:00:44.633 INFO:teuthology.orchestra.run.trial117.stdout: mon health detail to clog = false 2026-02-25T12:00:44.633 INFO:teuthology.orchestra.run.trial117.stdout: mon host = 10.20.193.117 2026-02-25T12:00:44.633 INFO:teuthology.orchestra.run.trial117.stdout: mon election default strategy = 3 2026-02-25T12:00:44.633 INFO:teuthology.orchestra.run.trial117.stdout: fsid = 43e63318-add1-4399-a1fc-80188594353b 2026-02-25T12:00:44.633 INFO:teuthology.orchestra.run.trial117.stdout: 2026-02-25T12:00:44.633 INFO:teuthology.orchestra.run.trial117.stdout:[osd] 2026-02-25T12:00:44.633 INFO:teuthology.orchestra.run.trial117.stdout: osd journal size = 100 2026-02-25T12:00:44.633 INFO:teuthology.orchestra.run.trial117.stdout: 2026-02-25T12:00:44.633 INFO:teuthology.orchestra.run.trial117.stdout: osd scrub load threshold = 5.0 2026-02-25T12:00:44.633 INFO:teuthology.orchestra.run.trial117.stdout: osd scrub max interval = 600 2026-02-25T12:00:44.633 INFO:teuthology.orchestra.run.trial117.stdout: osd mclock profile = high_recovery_ops 2026-02-25T12:00:44.634 INFO:teuthology.orchestra.run.trial117.stdout: osd mclock skip benchmark = True 2026-02-25T12:00:44.634 INFO:teuthology.orchestra.run.trial117.stdout: 2026-02-25T12:00:44.634 INFO:teuthology.orchestra.run.trial117.stdout: osd recover clone overlap = true 2026-02-25T12:00:44.634 INFO:teuthology.orchestra.run.trial117.stdout: osd recovery max chunk = 1048576 2026-02-25T12:00:44.634 INFO:teuthology.orchestra.run.trial117.stdout: 2026-02-25T12:00:44.634 INFO:teuthology.orchestra.run.trial117.stdout: osd debug shutdown = true 2026-02-25T12:00:44.634 INFO:teuthology.orchestra.run.trial117.stdout: osd debug op order = true 2026-02-25T12:00:44.634 INFO:teuthology.orchestra.run.trial117.stdout: osd debug verify stray on activate = true 2026-02-25T12:00:44.634 INFO:teuthology.orchestra.run.trial117.stdout: osd debug trim objects = true 2026-02-25T12:00:44.634 INFO:teuthology.orchestra.run.trial117.stdout: 2026-02-25T12:00:44.634 INFO:teuthology.orchestra.run.trial117.stdout: osd open classes on start = true 2026-02-25T12:00:44.634 INFO:teuthology.orchestra.run.trial117.stdout: osd debug pg log writeout = true 2026-02-25T12:00:44.634 INFO:teuthology.orchestra.run.trial117.stdout: 2026-02-25T12:00:44.634 INFO:teuthology.orchestra.run.trial117.stdout: osd deep scrub update digest min age = 30 2026-02-25T12:00:44.634 INFO:teuthology.orchestra.run.trial117.stdout: 2026-02-25T12:00:44.634 INFO:teuthology.orchestra.run.trial117.stdout: osd map max advance = 10 2026-02-25T12:00:44.634 INFO:teuthology.orchestra.run.trial117.stdout: 2026-02-25T12:00:44.634 INFO:teuthology.orchestra.run.trial117.stdout: journal zero on create = true 2026-02-25T12:00:44.634 INFO:teuthology.orchestra.run.trial117.stdout: 2026-02-25T12:00:44.634 INFO:teuthology.orchestra.run.trial117.stdout: filestore ondisk finisher threads = 3 2026-02-25T12:00:44.634 INFO:teuthology.orchestra.run.trial117.stdout: filestore apply finisher threads = 3 2026-02-25T12:00:44.634 INFO:teuthology.orchestra.run.trial117.stdout: 2026-02-25T12:00:44.634 INFO:teuthology.orchestra.run.trial117.stdout: bdev debug aio = true 2026-02-25T12:00:44.634 INFO:teuthology.orchestra.run.trial117.stdout: osd debug misdirected ops = true 2026-02-25T12:00:44.634 INFO:teuthology.orchestra.run.trial117.stdout: osd max object name len = 400 2026-02-25T12:00:44.634 INFO:teuthology.orchestra.run.trial117.stdout: osd max object namespace len = 64 2026-02-25T12:00:44.634 INFO:teuthology.orchestra.run.trial117.stdout: bluestore zero block detection = True 2026-02-25T12:00:44.634 INFO:teuthology.orchestra.run.trial117.stdout: debug ms = 1 2026-02-25T12:00:44.634 INFO:teuthology.orchestra.run.trial117.stdout: debug osd = 20 2026-02-25T12:00:44.634 INFO:teuthology.orchestra.run.trial117.stdout: osd debug verify cached snaps = True 2026-02-25T12:00:44.634 INFO:teuthology.orchestra.run.trial117.stdout: osd debug verify missing on start = True 2026-02-25T12:00:44.634 INFO:teuthology.orchestra.run.trial117.stdout: osd mclock override recovery settings = True 2026-02-25T12:00:44.635 INFO:teuthology.orchestra.run.trial117.stdout: osd op queue = debug_random 2026-02-25T12:00:44.635 INFO:teuthology.orchestra.run.trial117.stdout: osd op queue cut off = debug_random 2026-02-25T12:00:44.635 INFO:teuthology.orchestra.run.trial117.stdout: 2026-02-25T12:00:44.635 INFO:teuthology.orchestra.run.trial117.stdout:[mgr] 2026-02-25T12:00:44.635 INFO:teuthology.orchestra.run.trial117.stdout: debug ms = 1 2026-02-25T12:00:44.635 INFO:teuthology.orchestra.run.trial117.stdout: debug mgr = 20 2026-02-25T12:00:44.635 INFO:teuthology.orchestra.run.trial117.stdout: debug mon = 20 2026-02-25T12:00:44.635 INFO:teuthology.orchestra.run.trial117.stdout: debug auth = 20 2026-02-25T12:00:44.635 INFO:teuthology.orchestra.run.trial117.stdout: mon reweight min pgs per osd = 4 2026-02-25T12:00:44.635 INFO:teuthology.orchestra.run.trial117.stdout: mon reweight min bytes per osd = 10 2026-02-25T12:00:44.635 INFO:teuthology.orchestra.run.trial117.stdout: mgr/telemetry/nag = false 2026-02-25T12:00:44.635 INFO:teuthology.orchestra.run.trial117.stdout: 2026-02-25T12:00:44.635 INFO:teuthology.orchestra.run.trial117.stdout:[mon] 2026-02-25T12:00:44.635 INFO:teuthology.orchestra.run.trial117.stdout: debug ms = 1 2026-02-25T12:00:44.635 INFO:teuthology.orchestra.run.trial117.stdout: debug mon = 30 2026-02-25T12:00:44.635 INFO:teuthology.orchestra.run.trial117.stdout: debug paxos = 20 2026-02-25T12:00:44.635 INFO:teuthology.orchestra.run.trial117.stdout: debug auth = 20 2026-02-25T12:00:44.635 INFO:teuthology.orchestra.run.trial117.stdout: mon data avail warn = 5 2026-02-25T12:00:44.635 INFO:teuthology.orchestra.run.trial117.stdout: mon mgr mkfs grace = 240 2026-02-25T12:00:44.635 INFO:teuthology.orchestra.run.trial117.stdout: mon reweight min pgs per osd = 4 2026-02-25T12:00:44.635 INFO:teuthology.orchestra.run.trial117.stdout: mon osd reporter subtree level = osd 2026-02-25T12:00:44.635 INFO:teuthology.orchestra.run.trial117.stdout: mon osd prime pg temp = true 2026-02-25T12:00:44.635 INFO:teuthology.orchestra.run.trial117.stdout: mon reweight min bytes per osd = 10 2026-02-25T12:00:44.635 INFO:teuthology.orchestra.run.trial117.stdout: 2026-02-25T12:00:44.635 INFO:teuthology.orchestra.run.trial117.stdout: # rotate auth tickets quickly to exercise renewal paths 2026-02-25T12:00:44.635 INFO:teuthology.orchestra.run.trial117.stdout: auth mon ticket ttl = 660 # 11m 2026-02-25T12:00:44.635 INFO:teuthology.orchestra.run.trial117.stdout: auth service ticket ttl = 240 # 4m 2026-02-25T12:00:44.635 INFO:teuthology.orchestra.run.trial117.stdout: 2026-02-25T12:00:44.635 INFO:teuthology.orchestra.run.trial117.stdout: # don't complain about insecure global_id in the test suite 2026-02-25T12:00:44.635 INFO:teuthology.orchestra.run.trial117.stdout: mon_warn_on_insecure_global_id_reclaim = false 2026-02-25T12:00:44.635 INFO:teuthology.orchestra.run.trial117.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2026-02-25T12:00:44.636 INFO:teuthology.orchestra.run.trial117.stdout: 2026-02-25T12:00:44.636 INFO:teuthology.orchestra.run.trial117.stdout: # 1m isn't quite enough 2026-02-25T12:00:44.636 INFO:teuthology.orchestra.run.trial117.stdout: mon_down_mkfs_grace = 2m 2026-02-25T12:00:44.636 INFO:teuthology.orchestra.run.trial117.stdout: 2026-02-25T12:00:44.636 INFO:teuthology.orchestra.run.trial117.stdout: mon_warn_on_filestore_osds = false 2026-02-25T12:00:44.636 INFO:teuthology.orchestra.run.trial117.stdout: mon scrub interval = 300 2026-02-25T12:00:44.636 INFO:teuthology.orchestra.run.trial117.stdout: 2026-02-25T12:00:44.636 INFO:teuthology.orchestra.run.trial117.stdout:[client] 2026-02-25T12:00:44.636 INFO:teuthology.orchestra.run.trial117.stdout: rgw cache enabled = true 2026-02-25T12:00:44.636 INFO:teuthology.orchestra.run.trial117.stdout: rgw enable ops log = true 2026-02-25T12:00:44.636 INFO:teuthology.orchestra.run.trial117.stdout: rgw enable usage log = true 2026-02-25T12:00:44.636 INFO:teuthology.orchestra.run.trial117.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2026-02-25T12:00:44.636 INFO:teuthology.orchestra.run.trial117.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2026-02-25T12:00:44.636 INFO:teuthology.orchestra.run.trial117.stdout:[mon.a] 2026-02-25T12:00:44.640 INFO:tasks.ceph:Creating admin key on mon.a... 2026-02-25T12:00:44.640 DEBUG:teuthology.orchestra.run.trial117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/ceph.keyring 2026-02-25T12:00:44.707 INFO:tasks.ceph:Copying monmap to all nodes... 2026-02-25T12:00:44.708 DEBUG:teuthology.orchestra.run.trial117:> set -ex 2026-02-25T12:00:44.708 DEBUG:teuthology.orchestra.run.trial117:> dd if=/etc/ceph/ceph.keyring of=/dev/stdout 2026-02-25T12:00:44.753 DEBUG:teuthology.orchestra.run.trial117:> set -ex 2026-02-25T12:00:44.753 DEBUG:teuthology.orchestra.run.trial117:> dd if=/home/ubuntu/cephtest/ceph.monmap of=/dev/stdout 2026-02-25T12:00:44.801 INFO:tasks.ceph:Sending monmap to node ubuntu@trial117.front.sepia.ceph.com 2026-02-25T12:00:44.801 DEBUG:teuthology.orchestra.run.trial117:> set -ex 2026-02-25T12:00:44.801 DEBUG:teuthology.orchestra.run.trial117:> sudo dd of=/etc/ceph/ceph.keyring 2026-02-25T12:00:44.801 DEBUG:teuthology.orchestra.run.trial117:> sudo chmod 0644 /etc/ceph/ceph.keyring 2026-02-25T12:00:44.858 DEBUG:teuthology.orchestra.run.trial117:> set -ex 2026-02-25T12:00:44.858 DEBUG:teuthology.orchestra.run.trial117:> dd of=/home/ubuntu/cephtest/ceph.monmap 2026-02-25T12:00:44.905 INFO:tasks.ceph:Setting up mon nodes... 2026-02-25T12:00:44.905 INFO:tasks.ceph:Setting up mgr nodes... 2026-02-25T12:00:44.905 DEBUG:teuthology.orchestra.run.trial117:> sudo mkdir -p /var/lib/ceph/mgr/ceph-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/ceph-x/keyring 2026-02-25T12:00:44.970 INFO:teuthology.orchestra.run.trial117.stdout:creating /var/lib/ceph/mgr/ceph-x/keyring 2026-02-25T12:00:44.972 INFO:tasks.ceph:Setting up mds nodes... 2026-02-25T12:00:44.972 INFO:tasks.ceph_client:Setting up client nodes... 2026-02-25T12:00:44.973 DEBUG:teuthology.orchestra.run.trial117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/ceph.client.0.keyring && sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2026-02-25T12:00:45.029 INFO:teuthology.orchestra.run.trial117.stdout:creating /etc/ceph/ceph.client.0.keyring 2026-02-25T12:00:45.036 INFO:tasks.ceph:Running mkfs on osd nodes... 2026-02-25T12:00:45.036 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@trial117.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'}} 2026-02-25T12:00:45.037 DEBUG:teuthology.orchestra.run.trial117:> sudo mkdir -p /var/lib/ceph/osd/ceph-0 2026-02-25T12:00:45.084 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2026-02-25T12:00:45.084 INFO:tasks.ceph:role: osd.0 2026-02-25T12:00:45.084 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@trial117.front.sepia.ceph.com 2026-02-25T12:00:45.084 DEBUG:teuthology.orchestra.run.trial117:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2026-02-25T12:00:45.133 INFO:teuthology.orchestra.run.trial117.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=45875200 blks 2026-02-25T12:00:45.133 INFO:teuthology.orchestra.run.trial117.stdout: = sectsz=4096 attr=2, projid32bit=1 2026-02-25T12:00:45.133 INFO:teuthology.orchestra.run.trial117.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2026-02-25T12:00:45.134 INFO:teuthology.orchestra.run.trial117.stdout: = reflink=1 bigtime=0 inobtcount=0 2026-02-25T12:00:45.134 INFO:teuthology.orchestra.run.trial117.stdout:data = bsize=4096 blocks=183500800, imaxpct=25 2026-02-25T12:00:45.134 INFO:teuthology.orchestra.run.trial117.stdout: = sunit=0 swidth=0 blks 2026-02-25T12:00:45.134 INFO:teuthology.orchestra.run.trial117.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2026-02-25T12:00:45.134 INFO:teuthology.orchestra.run.trial117.stdout:log =internal log bsize=4096 blocks=89600, version=2 2026-02-25T12:00:45.134 INFO:teuthology.orchestra.run.trial117.stdout: = sectsz=4096 sunit=1 blks, lazy-count=1 2026-02-25T12:00:45.134 INFO:teuthology.orchestra.run.trial117.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2026-02-25T12:00:45.706 INFO:teuthology.orchestra.run.trial117.stdout:Discarding blocks...Done. 2026-02-25T12:00:45.708 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@trial117.front.sepia.ceph.com -o noatime 2026-02-25T12:00:45.708 DEBUG:teuthology.orchestra.run.trial117:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-0 2026-02-25T12:00:45.764 DEBUG:teuthology.orchestra.run.trial117:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-0 2026-02-25T12:00:45.815 DEBUG:teuthology.orchestra.run:got remote process result: 1 2026-02-25T12:00:45.815 INFO:teuthology.orchestra.run.trial117.stderr:sudo: /sbin/restorecon: command not found 2026-02-25T12:00:45.816 DEBUG:teuthology.orchestra.run.trial117:> sudo mkdir -p /var/lib/ceph/osd/ceph-1 2026-02-25T12:00:45.865 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2026-02-25T12:00:45.865 INFO:tasks.ceph:role: osd.1 2026-02-25T12:00:45.865 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@trial117.front.sepia.ceph.com 2026-02-25T12:00:45.865 DEBUG:teuthology.orchestra.run.trial117:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2026-02-25T12:00:45.918 INFO:teuthology.orchestra.run.trial117.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=45875200 blks 2026-02-25T12:00:45.918 INFO:teuthology.orchestra.run.trial117.stdout: = sectsz=4096 attr=2, projid32bit=1 2026-02-25T12:00:45.918 INFO:teuthology.orchestra.run.trial117.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2026-02-25T12:00:45.918 INFO:teuthology.orchestra.run.trial117.stdout: = reflink=1 bigtime=0 inobtcount=0 2026-02-25T12:00:45.918 INFO:teuthology.orchestra.run.trial117.stdout:data = bsize=4096 blocks=183500800, imaxpct=25 2026-02-25T12:00:45.918 INFO:teuthology.orchestra.run.trial117.stdout: = sunit=0 swidth=0 blks 2026-02-25T12:00:45.918 INFO:teuthology.orchestra.run.trial117.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2026-02-25T12:00:45.918 INFO:teuthology.orchestra.run.trial117.stdout:log =internal log bsize=4096 blocks=89600, version=2 2026-02-25T12:00:45.918 INFO:teuthology.orchestra.run.trial117.stdout: = sectsz=4096 sunit=1 blks, lazy-count=1 2026-02-25T12:00:45.918 INFO:teuthology.orchestra.run.trial117.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2026-02-25T12:00:46.480 INFO:teuthology.orchestra.run.trial117.stdout:Discarding blocks...Done. 2026-02-25T12:00:46.481 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@trial117.front.sepia.ceph.com -o noatime 2026-02-25T12:00:46.482 DEBUG:teuthology.orchestra.run.trial117:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-1 2026-02-25T12:00:46.541 DEBUG:teuthology.orchestra.run.trial117:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-1 2026-02-25T12:00:46.590 DEBUG:teuthology.orchestra.run:got remote process result: 1 2026-02-25T12:00:46.591 INFO:teuthology.orchestra.run.trial117.stderr:sudo: /sbin/restorecon: command not found 2026-02-25T12:00:46.592 DEBUG:teuthology.orchestra.run.trial117:> sudo mkdir -p /var/lib/ceph/osd/ceph-2 2026-02-25T12:00:46.642 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2026-02-25T12:00:46.642 INFO:tasks.ceph:role: osd.2 2026-02-25T12:00:46.642 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@trial117.front.sepia.ceph.com 2026-02-25T12:00:46.642 DEBUG:teuthology.orchestra.run.trial117:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2026-02-25T12:00:46.694 INFO:teuthology.orchestra.run.trial117.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=45875200 blks 2026-02-25T12:00:46.694 INFO:teuthology.orchestra.run.trial117.stdout: = sectsz=4096 attr=2, projid32bit=1 2026-02-25T12:00:46.694 INFO:teuthology.orchestra.run.trial117.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2026-02-25T12:00:46.695 INFO:teuthology.orchestra.run.trial117.stdout: = reflink=1 bigtime=0 inobtcount=0 2026-02-25T12:00:46.695 INFO:teuthology.orchestra.run.trial117.stdout:data = bsize=4096 blocks=183500800, imaxpct=25 2026-02-25T12:00:46.695 INFO:teuthology.orchestra.run.trial117.stdout: = sunit=0 swidth=0 blks 2026-02-25T12:00:46.695 INFO:teuthology.orchestra.run.trial117.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2026-02-25T12:00:46.695 INFO:teuthology.orchestra.run.trial117.stdout:log =internal log bsize=4096 blocks=89600, version=2 2026-02-25T12:00:46.695 INFO:teuthology.orchestra.run.trial117.stdout: = sectsz=4096 sunit=1 blks, lazy-count=1 2026-02-25T12:00:46.695 INFO:teuthology.orchestra.run.trial117.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2026-02-25T12:00:47.255 INFO:teuthology.orchestra.run.trial117.stdout:Discarding blocks...Done. 2026-02-25T12:00:47.257 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@trial117.front.sepia.ceph.com -o noatime 2026-02-25T12:00:47.257 DEBUG:teuthology.orchestra.run.trial117:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-2 2026-02-25T12:00:47.317 DEBUG:teuthology.orchestra.run.trial117:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-2 2026-02-25T12:00:47.369 INFO:teuthology.orchestra.run.trial117.stderr:sudo: /sbin/restorecon: command not found 2026-02-25T12:00:47.370 DEBUG:teuthology.orchestra.run:got remote process result: 1 2026-02-25T12:00:47.370 DEBUG:teuthology.orchestra.run.trial117:> sudo mkdir -p /var/lib/ceph/osd/ceph-3 2026-02-25T12:00:47.432 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2026-02-25T12:00:47.432 INFO:tasks.ceph:role: osd.3 2026-02-25T12:00:47.432 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@trial117.front.sepia.ceph.com 2026-02-25T12:00:47.432 DEBUG:teuthology.orchestra.run.trial117:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2026-02-25T12:00:47.490 INFO:teuthology.orchestra.run.trial117.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=45875200 blks 2026-02-25T12:00:47.490 INFO:teuthology.orchestra.run.trial117.stdout: = sectsz=4096 attr=2, projid32bit=1 2026-02-25T12:00:47.490 INFO:teuthology.orchestra.run.trial117.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2026-02-25T12:00:47.490 INFO:teuthology.orchestra.run.trial117.stdout: = reflink=1 bigtime=0 inobtcount=0 2026-02-25T12:00:47.490 INFO:teuthology.orchestra.run.trial117.stdout:data = bsize=4096 blocks=183500800, imaxpct=25 2026-02-25T12:00:47.490 INFO:teuthology.orchestra.run.trial117.stdout: = sunit=0 swidth=0 blks 2026-02-25T12:00:47.490 INFO:teuthology.orchestra.run.trial117.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2026-02-25T12:00:47.490 INFO:teuthology.orchestra.run.trial117.stdout:log =internal log bsize=4096 blocks=89600, version=2 2026-02-25T12:00:47.490 INFO:teuthology.orchestra.run.trial117.stdout: = sectsz=4096 sunit=1 blks, lazy-count=1 2026-02-25T12:00:47.490 INFO:teuthology.orchestra.run.trial117.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2026-02-25T12:00:48.050 INFO:teuthology.orchestra.run.trial117.stdout:Discarding blocks...Done. 2026-02-25T12:00:48.052 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@trial117.front.sepia.ceph.com -o noatime 2026-02-25T12:00:48.052 DEBUG:teuthology.orchestra.run.trial117:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-3 2026-02-25T12:00:48.106 DEBUG:teuthology.orchestra.run.trial117:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-3 2026-02-25T12:00:48.154 INFO:teuthology.orchestra.run.trial117.stderr:sudo: /sbin/restorecon: command not found 2026-02-25T12:00:48.154 DEBUG:teuthology.orchestra.run:got remote process result: 1 2026-02-25T12:00:48.154 DEBUG:teuthology.orchestra.run.trial117:> sudo mkdir -p /var/lib/ceph/osd/ceph-4 2026-02-25T12:00:48.204 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2026-02-25T12:00:48.204 INFO:tasks.ceph:role: osd.4 2026-02-25T12:00:48.204 DEBUG:teuthology.orchestra.run.trial117:> sudo mkdir -p /var/lib/ceph/osd/ceph-5 2026-02-25T12:00:48.253 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2026-02-25T12:00:48.253 INFO:tasks.ceph:role: osd.5 2026-02-25T12:00:48.254 DEBUG:teuthology.orchestra.run.trial117:> sudo mkdir -p /var/lib/ceph/osd/ceph-6 2026-02-25T12:00:48.304 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2026-02-25T12:00:48.304 INFO:tasks.ceph:role: osd.6 2026-02-25T12:00:48.304 DEBUG:teuthology.orchestra.run.trial117:> sudo mkdir -p /var/lib/ceph/osd/ceph-7 2026-02-25T12:00:48.352 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2026-02-25T12:00:48.353 INFO:tasks.ceph:role: osd.7 2026-02-25T12:00:48.353 DEBUG:teuthology.orchestra.run.trial117:> sudo mkdir -p /var/lib/ceph/osd/ceph-8 2026-02-25T12:00:48.400 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2026-02-25T12:00:48.400 INFO:tasks.ceph:role: osd.8 2026-02-25T12:00:48.400 DEBUG:teuthology.orchestra.run.trial117:> sudo mkdir -p /var/lib/ceph/osd/ceph-9 2026-02-25T12:00:48.448 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2026-02-25T12:00:48.448 INFO:tasks.ceph:role: osd.9 2026-02-25T12:00:48.449 DEBUG:teuthology.orchestra.run.trial117:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/ceph.monmap 2026-02-25T12:00:48.510 INFO:teuthology.orchestra.run.trial117.stderr:2026-02-25T12:00:48.507+0000 7f3441235a40 -1 auth: error reading file: /var/lib/ceph/osd/ceph-0/keyring: can't open /var/lib/ceph/osd/ceph-0/keyring: (2) No such file or directory 2026-02-25T12:00:48.510 INFO:teuthology.orchestra.run.trial117.stderr:2026-02-25T12:00:48.507+0000 7f3441235a40 -1 created new key in keyring /var/lib/ceph/osd/ceph-0/keyring 2026-02-25T12:00:48.510 INFO:teuthology.orchestra.run.trial117.stderr:2026-02-25T12:00:48.507+0000 7f3441235a40 -1 bdev(0x55c8571ed800 /var/lib/ceph/osd/ceph-0/block) open stat got: (1) Operation not permitted 2026-02-25T12:00:48.510 INFO:teuthology.orchestra.run.trial117.stderr:2026-02-25T12:00:48.507+0000 7f3441235a40 -1 bluestore(/var/lib/ceph/osd/ceph-0) _read_fsid unparsable uuid 2026-02-25T12:00:49.114 DEBUG:teuthology.orchestra.run.trial117:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2026-02-25T12:00:49.165 DEBUG:teuthology.orchestra.run.trial117:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/ceph.monmap 2026-02-25T12:00:49.229 INFO:teuthology.orchestra.run.trial117.stderr:2026-02-25T12:00:49.227+0000 7f3e2fdefa40 -1 auth: error reading file: /var/lib/ceph/osd/ceph-1/keyring: can't open /var/lib/ceph/osd/ceph-1/keyring: (2) No such file or directory 2026-02-25T12:00:49.229 INFO:teuthology.orchestra.run.trial117.stderr:2026-02-25T12:00:49.227+0000 7f3e2fdefa40 -1 created new key in keyring /var/lib/ceph/osd/ceph-1/keyring 2026-02-25T12:00:49.229 INFO:teuthology.orchestra.run.trial117.stderr:2026-02-25T12:00:49.227+0000 7f3e2fdefa40 -1 bdev(0x5647e70c9800 /var/lib/ceph/osd/ceph-1/block) open stat got: (1) Operation not permitted 2026-02-25T12:00:49.229 INFO:teuthology.orchestra.run.trial117.stderr:2026-02-25T12:00:49.227+0000 7f3e2fdefa40 -1 bluestore(/var/lib/ceph/osd/ceph-1) _read_fsid unparsable uuid 2026-02-25T12:00:49.841 DEBUG:teuthology.orchestra.run.trial117:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2026-02-25T12:00:49.893 DEBUG:teuthology.orchestra.run.trial117:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/ceph.monmap 2026-02-25T12:00:49.957 INFO:teuthology.orchestra.run.trial117.stderr:2026-02-25T12:00:49.951+0000 7f4933b2ea40 -1 auth: error reading file: /var/lib/ceph/osd/ceph-2/keyring: can't open /var/lib/ceph/osd/ceph-2/keyring: (2) No such file or directory 2026-02-25T12:00:49.957 INFO:teuthology.orchestra.run.trial117.stderr:2026-02-25T12:00:49.955+0000 7f4933b2ea40 -1 created new key in keyring /var/lib/ceph/osd/ceph-2/keyring 2026-02-25T12:00:49.957 INFO:teuthology.orchestra.run.trial117.stderr:2026-02-25T12:00:49.955+0000 7f4933b2ea40 -1 bdev(0x55c979829800 /var/lib/ceph/osd/ceph-2/block) open stat got: (1) Operation not permitted 2026-02-25T12:00:49.957 INFO:teuthology.orchestra.run.trial117.stderr:2026-02-25T12:00:49.955+0000 7f4933b2ea40 -1 bluestore(/var/lib/ceph/osd/ceph-2) _read_fsid unparsable uuid 2026-02-25T12:00:50.610 DEBUG:teuthology.orchestra.run.trial117:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2026-02-25T12:00:50.662 DEBUG:teuthology.orchestra.run.trial117:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 3 --monmap /home/ubuntu/cephtest/ceph.monmap 2026-02-25T12:00:50.726 INFO:teuthology.orchestra.run.trial117.stderr:2026-02-25T12:00:50.723+0000 7f3424f29a40 -1 auth: error reading file: /var/lib/ceph/osd/ceph-3/keyring: can't open /var/lib/ceph/osd/ceph-3/keyring: (2) No such file or directory 2026-02-25T12:00:50.726 INFO:teuthology.orchestra.run.trial117.stderr:2026-02-25T12:00:50.723+0000 7f3424f29a40 -1 created new key in keyring /var/lib/ceph/osd/ceph-3/keyring 2026-02-25T12:00:50.727 INFO:teuthology.orchestra.run.trial117.stderr:2026-02-25T12:00:50.723+0000 7f3424f29a40 -1 bdev(0x55b20129f800 /var/lib/ceph/osd/ceph-3/block) open stat got: (1) Operation not permitted 2026-02-25T12:00:50.727 INFO:teuthology.orchestra.run.trial117.stderr:2026-02-25T12:00:50.723+0000 7f3424f29a40 -1 bluestore(/var/lib/ceph/osd/ceph-3) _read_fsid unparsable uuid 2026-02-25T12:00:51.373 DEBUG:teuthology.orchestra.run.trial117:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2026-02-25T12:00:51.425 DEBUG:teuthology.orchestra.run.trial117:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 4 --monmap /home/ubuntu/cephtest/ceph.monmap 2026-02-25T12:00:51.489 INFO:teuthology.orchestra.run.trial117.stderr:2026-02-25T12:00:51.487+0000 7fbac7dfea40 -1 auth: error reading file: /var/lib/ceph/osd/ceph-4/keyring: can't open /var/lib/ceph/osd/ceph-4/keyring: (2) No such file or directory 2026-02-25T12:00:51.489 INFO:teuthology.orchestra.run.trial117.stderr:2026-02-25T12:00:51.487+0000 7fbac7dfea40 -1 created new key in keyring /var/lib/ceph/osd/ceph-4/keyring 2026-02-25T12:00:51.489 INFO:teuthology.orchestra.run.trial117.stderr:2026-02-25T12:00:51.487+0000 7fbac7dfea40 -1 bdev(0x555686661800 /var/lib/ceph/osd/ceph-4/block) open stat got: (1) Operation not permitted 2026-02-25T12:00:51.489 INFO:teuthology.orchestra.run.trial117.stderr:2026-02-25T12:00:51.487+0000 7fbac7dfea40 -1 bluestore(/var/lib/ceph/osd/ceph-4) _read_fsid unparsable uuid 2026-02-25T12:00:52.140 DEBUG:teuthology.orchestra.run.trial117:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2026-02-25T12:00:52.193 DEBUG:teuthology.orchestra.run.trial117:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 5 --monmap /home/ubuntu/cephtest/ceph.monmap 2026-02-25T12:00:52.257 INFO:teuthology.orchestra.run.trial117.stderr:2026-02-25T12:00:52.255+0000 7ff62640aa40 -1 auth: error reading file: /var/lib/ceph/osd/ceph-5/keyring: can't open /var/lib/ceph/osd/ceph-5/keyring: (2) No such file or directory 2026-02-25T12:00:52.257 INFO:teuthology.orchestra.run.trial117.stderr:2026-02-25T12:00:52.255+0000 7ff62640aa40 -1 created new key in keyring /var/lib/ceph/osd/ceph-5/keyring 2026-02-25T12:00:52.257 INFO:teuthology.orchestra.run.trial117.stderr:2026-02-25T12:00:52.255+0000 7ff62640aa40 -1 bdev(0x558b22593800 /var/lib/ceph/osd/ceph-5/block) open stat got: (1) Operation not permitted 2026-02-25T12:00:52.257 INFO:teuthology.orchestra.run.trial117.stderr:2026-02-25T12:00:52.255+0000 7ff62640aa40 -1 bluestore(/var/lib/ceph/osd/ceph-5) _read_fsid unparsable uuid 2026-02-25T12:00:52.885 DEBUG:teuthology.orchestra.run.trial117:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2026-02-25T12:00:52.937 DEBUG:teuthology.orchestra.run.trial117:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 6 --monmap /home/ubuntu/cephtest/ceph.monmap 2026-02-25T12:00:53.001 INFO:teuthology.orchestra.run.trial117.stderr:2026-02-25T12:00:52.999+0000 7fe6db995a40 -1 auth: error reading file: /var/lib/ceph/osd/ceph-6/keyring: can't open /var/lib/ceph/osd/ceph-6/keyring: (2) No such file or directory 2026-02-25T12:00:53.001 INFO:teuthology.orchestra.run.trial117.stderr:2026-02-25T12:00:52.999+0000 7fe6db995a40 -1 created new key in keyring /var/lib/ceph/osd/ceph-6/keyring 2026-02-25T12:00:53.002 INFO:teuthology.orchestra.run.trial117.stderr:2026-02-25T12:00:52.999+0000 7fe6db995a40 -1 bdev(0x55ef67afb800 /var/lib/ceph/osd/ceph-6/block) open stat got: (1) Operation not permitted 2026-02-25T12:00:53.002 INFO:teuthology.orchestra.run.trial117.stderr:2026-02-25T12:00:52.999+0000 7fe6db995a40 -1 bluestore(/var/lib/ceph/osd/ceph-6) _read_fsid unparsable uuid 2026-02-25T12:00:53.621 DEBUG:teuthology.orchestra.run.trial117:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-6 2026-02-25T12:00:53.674 DEBUG:teuthology.orchestra.run.trial117:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 7 --monmap /home/ubuntu/cephtest/ceph.monmap 2026-02-25T12:00:53.737 INFO:teuthology.orchestra.run.trial117.stderr:2026-02-25T12:00:53.735+0000 7ff7d4b2da40 -1 auth: error reading file: /var/lib/ceph/osd/ceph-7/keyring: can't open /var/lib/ceph/osd/ceph-7/keyring: (2) No such file or directory 2026-02-25T12:00:53.738 INFO:teuthology.orchestra.run.trial117.stderr:2026-02-25T12:00:53.735+0000 7ff7d4b2da40 -1 created new key in keyring /var/lib/ceph/osd/ceph-7/keyring 2026-02-25T12:00:53.738 INFO:teuthology.orchestra.run.trial117.stderr:2026-02-25T12:00:53.735+0000 7ff7d4b2da40 -1 bdev(0x564bf6a35800 /var/lib/ceph/osd/ceph-7/block) open stat got: (1) Operation not permitted 2026-02-25T12:00:53.738 INFO:teuthology.orchestra.run.trial117.stderr:2026-02-25T12:00:53.735+0000 7ff7d4b2da40 -1 bluestore(/var/lib/ceph/osd/ceph-7) _read_fsid unparsable uuid 2026-02-25T12:00:54.484 DEBUG:teuthology.orchestra.run.trial117:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-7 2026-02-25T12:00:54.533 DEBUG:teuthology.orchestra.run.trial117:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 8 --monmap /home/ubuntu/cephtest/ceph.monmap 2026-02-25T12:00:54.597 INFO:teuthology.orchestra.run.trial117.stderr:2026-02-25T12:00:54.595+0000 7fb5bea9fa40 -1 auth: error reading file: /var/lib/ceph/osd/ceph-8/keyring: can't open /var/lib/ceph/osd/ceph-8/keyring: (2) No such file or directory 2026-02-25T12:00:54.597 INFO:teuthology.orchestra.run.trial117.stderr:2026-02-25T12:00:54.595+0000 7fb5bea9fa40 -1 created new key in keyring /var/lib/ceph/osd/ceph-8/keyring 2026-02-25T12:00:54.598 INFO:teuthology.orchestra.run.trial117.stderr:2026-02-25T12:00:54.595+0000 7fb5bea9fa40 -1 bdev(0x55e3c9a33800 /var/lib/ceph/osd/ceph-8/block) open stat got: (1) Operation not permitted 2026-02-25T12:00:54.598 INFO:teuthology.orchestra.run.trial117.stderr:2026-02-25T12:00:54.595+0000 7fb5bea9fa40 -1 bluestore(/var/lib/ceph/osd/ceph-8) _read_fsid unparsable uuid 2026-02-25T12:00:55.261 DEBUG:teuthology.orchestra.run.trial117:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-8 2026-02-25T12:00:55.313 DEBUG:teuthology.orchestra.run.trial117:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 9 --monmap /home/ubuntu/cephtest/ceph.monmap 2026-02-25T12:00:55.377 INFO:teuthology.orchestra.run.trial117.stderr:2026-02-25T12:00:55.375+0000 7f275a758a40 -1 auth: error reading file: /var/lib/ceph/osd/ceph-9/keyring: can't open /var/lib/ceph/osd/ceph-9/keyring: (2) No such file or directory 2026-02-25T12:00:55.377 INFO:teuthology.orchestra.run.trial117.stderr:2026-02-25T12:00:55.375+0000 7f275a758a40 -1 created new key in keyring /var/lib/ceph/osd/ceph-9/keyring 2026-02-25T12:00:55.377 INFO:teuthology.orchestra.run.trial117.stderr:2026-02-25T12:00:55.375+0000 7f275a758a40 -1 bdev(0x5602e6633800 /var/lib/ceph/osd/ceph-9/block) open stat got: (1) Operation not permitted 2026-02-25T12:00:55.377 INFO:teuthology.orchestra.run.trial117.stderr:2026-02-25T12:00:55.375+0000 7f275a758a40 -1 bluestore(/var/lib/ceph/osd/ceph-9) _read_fsid unparsable uuid 2026-02-25T12:00:56.013 DEBUG:teuthology.orchestra.run.trial117:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-9 2026-02-25T12:00:56.069 INFO:tasks.ceph:Reading keys from all nodes... 2026-02-25T12:00:56.070 DEBUG:teuthology.orchestra.run.trial117:> set -ex 2026-02-25T12:00:56.070 DEBUG:teuthology.orchestra.run.trial117:> sudo dd if=/var/lib/ceph/mgr/ceph-x/keyring of=/dev/stdout 2026-02-25T12:00:56.120 DEBUG:teuthology.orchestra.run.trial117:> set -ex 2026-02-25T12:00:56.120 DEBUG:teuthology.orchestra.run.trial117:> sudo dd if=/var/lib/ceph/osd/ceph-0/keyring of=/dev/stdout 2026-02-25T12:00:56.176 DEBUG:teuthology.orchestra.run.trial117:> set -ex 2026-02-25T12:00:56.176 DEBUG:teuthology.orchestra.run.trial117:> sudo dd if=/var/lib/ceph/osd/ceph-1/keyring of=/dev/stdout 2026-02-25T12:00:56.234 DEBUG:teuthology.orchestra.run.trial117:> set -ex 2026-02-25T12:00:56.235 DEBUG:teuthology.orchestra.run.trial117:> sudo dd if=/var/lib/ceph/osd/ceph-2/keyring of=/dev/stdout 2026-02-25T12:00:56.290 DEBUG:teuthology.orchestra.run.trial117:> set -ex 2026-02-25T12:00:56.290 DEBUG:teuthology.orchestra.run.trial117:> sudo dd if=/var/lib/ceph/osd/ceph-3/keyring of=/dev/stdout 2026-02-25T12:00:56.340 DEBUG:teuthology.orchestra.run.trial117:> set -ex 2026-02-25T12:00:56.340 DEBUG:teuthology.orchestra.run.trial117:> sudo dd if=/var/lib/ceph/osd/ceph-4/keyring of=/dev/stdout 2026-02-25T12:00:56.393 DEBUG:teuthology.orchestra.run.trial117:> set -ex 2026-02-25T12:00:56.393 DEBUG:teuthology.orchestra.run.trial117:> sudo dd if=/var/lib/ceph/osd/ceph-5/keyring of=/dev/stdout 2026-02-25T12:00:56.445 DEBUG:teuthology.orchestra.run.trial117:> set -ex 2026-02-25T12:00:56.445 DEBUG:teuthology.orchestra.run.trial117:> sudo dd if=/var/lib/ceph/osd/ceph-6/keyring of=/dev/stdout 2026-02-25T12:00:56.498 DEBUG:teuthology.orchestra.run.trial117:> set -ex 2026-02-25T12:00:56.499 DEBUG:teuthology.orchestra.run.trial117:> sudo dd if=/var/lib/ceph/osd/ceph-7/keyring of=/dev/stdout 2026-02-25T12:00:56.549 DEBUG:teuthology.orchestra.run.trial117:> set -ex 2026-02-25T12:00:56.550 DEBUG:teuthology.orchestra.run.trial117:> sudo dd if=/var/lib/ceph/osd/ceph-8/keyring of=/dev/stdout 2026-02-25T12:00:56.601 DEBUG:teuthology.orchestra.run.trial117:> set -ex 2026-02-25T12:00:56.601 DEBUG:teuthology.orchestra.run.trial117:> sudo dd if=/var/lib/ceph/osd/ceph-9/keyring of=/dev/stdout 2026-02-25T12:00:56.653 DEBUG:teuthology.orchestra.run.trial117:> set -ex 2026-02-25T12:00:56.654 DEBUG:teuthology.orchestra.run.trial117:> dd if=/etc/ceph/ceph.client.0.keyring of=/dev/stdout 2026-02-25T12:00:56.701 INFO:tasks.ceph:Adding keys to all mons... 2026-02-25T12:00:56.701 DEBUG:teuthology.orchestra.run.trial117:> sudo tee -a /etc/ceph/ceph.keyring 2026-02-25T12:00:56.751 INFO:teuthology.orchestra.run.trial117.stdout:[mgr.x] 2026-02-25T12:00:56.751 INFO:teuthology.orchestra.run.trial117.stdout: key = AQBs5J5pfJrJORAAqjfQcifxAODEvOKw0Zz0hg== 2026-02-25T12:00:56.752 INFO:teuthology.orchestra.run.trial117.stdout:[osd.0] 2026-02-25T12:00:56.752 INFO:teuthology.orchestra.run.trial117.stdout: key = AQBw5J5p/xpeHhAAXn8IWR4N8aS8SJF5bRbarw== 2026-02-25T12:00:56.752 INFO:teuthology.orchestra.run.trial117.stdout:[osd.1] 2026-02-25T12:00:56.752 INFO:teuthology.orchestra.run.trial117.stdout: key = AQBx5J5pHHShDRAAd8cbgfeHWQNW2PL7pJIY4A== 2026-02-25T12:00:56.752 INFO:teuthology.orchestra.run.trial117.stdout:[osd.2] 2026-02-25T12:00:56.752 INFO:teuthology.orchestra.run.trial117.stdout: key = AQBx5J5pU0P8OBAAEEZiEu+nEZR097K2XcgDpw== 2026-02-25T12:00:56.752 INFO:teuthology.orchestra.run.trial117.stdout:[osd.3] 2026-02-25T12:00:56.752 INFO:teuthology.orchestra.run.trial117.stdout: key = AQBy5J5pFgo6KxAAQ+5i/a50MuotbRn7HW5KdA== 2026-02-25T12:00:56.752 INFO:teuthology.orchestra.run.trial117.stdout:[osd.4] 2026-02-25T12:00:56.752 INFO:teuthology.orchestra.run.trial117.stdout: key = AQBz5J5pgT0fHRAAt+yJ+MmGmHO0aQnyKu8odA== 2026-02-25T12:00:56.753 INFO:teuthology.orchestra.run.trial117.stdout:[osd.5] 2026-02-25T12:00:56.753 INFO:teuthology.orchestra.run.trial117.stdout: key = AQB05J5pwGpLDxAAQWyV9xTTI6CEc4/gWyDpcw== 2026-02-25T12:00:56.753 INFO:teuthology.orchestra.run.trial117.stdout:[osd.6] 2026-02-25T12:00:56.753 INFO:teuthology.orchestra.run.trial117.stdout: key = AQB15J5p3PsMABAAfOdU18dpLebOL7OnRgpa6g== 2026-02-25T12:00:56.753 INFO:teuthology.orchestra.run.trial117.stdout:[osd.7] 2026-02-25T12:00:56.753 INFO:teuthology.orchestra.run.trial117.stdout: key = AQB15J5p5wbpKxAAyOE4+w6YImgtqa/XIgoaKA== 2026-02-25T12:00:56.753 INFO:teuthology.orchestra.run.trial117.stdout:[osd.8] 2026-02-25T12:00:56.753 INFO:teuthology.orchestra.run.trial117.stdout: key = AQB25J5pUJ2TIxAAHfw8YPmLiopy78x4FjV9uA== 2026-02-25T12:00:56.753 INFO:teuthology.orchestra.run.trial117.stdout:[osd.9] 2026-02-25T12:00:56.753 INFO:teuthology.orchestra.run.trial117.stdout: key = AQB35J5p+5BvFhAAyI62/nQb6NAPLWU+RaEeHg== 2026-02-25T12:00:56.753 INFO:teuthology.orchestra.run.trial117.stdout:[client.0] 2026-02-25T12:00:56.754 INFO:teuthology.orchestra.run.trial117.stdout: key = AQBt5J5p57S4ARAApv6brcU6VrWzN80Ck6js2w== 2026-02-25T12:00:56.755 DEBUG:teuthology.orchestra.run.trial117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2026-02-25T12:00:56.817 DEBUG:teuthology.orchestra.run.trial117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2026-02-25T12:00:56.880 DEBUG:teuthology.orchestra.run.trial117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2026-02-25T12:00:56.945 DEBUG:teuthology.orchestra.run.trial117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2026-02-25T12:00:57.008 DEBUG:teuthology.orchestra.run.trial117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2026-02-25T12:00:57.073 DEBUG:teuthology.orchestra.run.trial117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2026-02-25T12:00:57.136 DEBUG:teuthology.orchestra.run.trial117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2026-02-25T12:00:57.201 DEBUG:teuthology.orchestra.run.trial117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2026-02-25T12:00:57.265 DEBUG:teuthology.orchestra.run.trial117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2026-02-25T12:00:57.329 DEBUG:teuthology.orchestra.run.trial117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.8 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2026-02-25T12:00:57.396 DEBUG:teuthology.orchestra.run.trial117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.9 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2026-02-25T12:00:57.460 DEBUG:teuthology.orchestra.run.trial117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2026-02-25T12:00:57.522 INFO:tasks.ceph:Running mkfs on mon nodes... 2026-02-25T12:00:57.522 DEBUG:teuthology.orchestra.run.trial117:> sudo mkdir -p /var/lib/ceph/mon/ceph-a 2026-02-25T12:00:57.572 DEBUG:teuthology.orchestra.run.trial117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i a --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2026-02-25T12:00:57.637 DEBUG:teuthology.orchestra.run.trial117:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-a 2026-02-25T12:00:57.688 DEBUG:teuthology.orchestra.run.trial117:> rm -- /home/ubuntu/cephtest/ceph.monmap 2026-02-25T12:00:57.732 INFO:tasks.ceph:Starting mon daemons in cluster ceph... 2026-02-25T12:00:57.732 INFO:tasks.ceph.mon.a:Restarting daemon 2026-02-25T12:00:57.732 DEBUG:teuthology.orchestra.run.trial117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2026-02-25T12:00:57.777 INFO:tasks.ceph.mon.a:Started 2026-02-25T12:00:57.778 DEBUG:teuthology.orchestra.run.trial117:> sudo ceph config set mgr mgr_pool false --force 2026-02-25T12:00:57.917 INFO:tasks.ceph:Starting mgr daemons in cluster ceph... 2026-02-25T12:00:57.917 INFO:tasks.ceph.mgr.x:Restarting daemon 2026-02-25T12:00:57.918 DEBUG:teuthology.orchestra.run.trial117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i x 2026-02-25T12:00:57.920 INFO:tasks.ceph.mgr.x:Started 2026-02-25T12:00:57.920 DEBUG:tasks.ceph:set 0 configs 2026-02-25T12:00:57.920 DEBUG:teuthology.orchestra.run.trial117:> sudo ceph --cluster ceph config dump 2026-02-25T12:00:57.977 INFO:tasks.ceph.mgr.x.trial117.stderr:2026-02-25T12:00:57.971+0000 7ff10279b280 -1 mgr[py] Loading module balancer in main interpreter 2026-02-25T12:00:58.014 INFO:teuthology.orchestra.run.trial117.stdout:WHO MASK LEVEL OPTION VALUE RO 2026-02-25T12:00:58.014 INFO:teuthology.orchestra.run.trial117.stdout:mgr dev mgr_pool false * 2026-02-25T12:00:58.018 INFO:tasks.ceph.mgr.x.trial117.stderr:2026-02-25T12:00:58.015+0000 7ff10279b280 -1 mgr[py] Loading module telegraf in main interpreter 2026-02-25T12:00:58.021 INFO:tasks.ceph.mgr.x.trial117.stderr:2026-02-25T12:00:58.019+0000 7ff10279b280 -1 mgr[py] Loading module osd_perf_query in main interpreter 2026-02-25T12:00:58.023 INFO:tasks.ceph:Setting crush tunables to default 2026-02-25T12:00:58.023 DEBUG:teuthology.orchestra.run.trial117:> sudo ceph --cluster ceph osd crush tunables default 2026-02-25T12:00:58.033 INFO:tasks.ceph.mgr.x.trial117.stderr:2026-02-25T12:00:58.031+0000 7ff10279b280 -1 mgr[py] Loading module telemetry in main interpreter 2026-02-25T12:00:58.084 INFO:tasks.ceph.mgr.x.trial117.stderr:2026-02-25T12:00:58.079+0000 7ff10279b280 -1 mgr[py] Loading module smb in main interpreter 2026-02-25T12:00:58.122 INFO:teuthology.orchestra.run.trial117.stderr:adjusted tunables profile to default 2026-02-25T12:00:58.131 INFO:tasks.ceph:check_enable_crimson: False 2026-02-25T12:00:58.131 INFO:tasks.ceph:Starting osd daemons in cluster ceph... 2026-02-25T12:00:58.132 DEBUG:teuthology.orchestra.run.trial117:> set -ex 2026-02-25T12:00:58.132 DEBUG:teuthology.orchestra.run.trial117:> sudo dd if=/var/lib/ceph/osd/ceph-0/fsid of=/dev/stdout 2026-02-25T12:00:58.152 INFO:tasks.ceph.mgr.x.trial117.stderr:2026-02-25T12:00:58.147+0000 7ff10279b280 -1 mgr[py] Loading module test_orchestrator in main interpreter 2026-02-25T12:00:58.154 INFO:tasks.ceph.mgr.x.trial117.stderr:2026-02-25T12:00:58.151+0000 7ff10279b280 -1 mgr[py] Loading module status in main interpreter 2026-02-25T12:00:58.155 INFO:tasks.ceph.mgr.x.trial117.stderr:2026-02-25T12:00:58.151+0000 7ff10279b280 -1 mgr[py] Loading module nfs in main interpreter 2026-02-25T12:00:58.160 DEBUG:teuthology.orchestra.run.trial117:> set -ex 2026-02-25T12:00:58.161 DEBUG:teuthology.orchestra.run.trial117:> sudo dd if=/var/lib/ceph/osd/ceph-1/fsid of=/dev/stdout 2026-02-25T12:00:58.165 INFO:tasks.ceph.mgr.x.trial117.stderr:2026-02-25T12:00:58.163+0000 7ff10279b280 -1 mgr[py] Loading module localpool in main interpreter 2026-02-25T12:00:58.165 INFO:tasks.ceph.mgr.x.trial117.stderr:2026-02-25T12:00:58.163+0000 7ff10279b280 -1 mgr[py] Loading module prometheus in main interpreter 2026-02-25T12:00:58.173 DEBUG:teuthology.orchestra.run.trial117:> set -ex 2026-02-25T12:00:58.173 DEBUG:teuthology.orchestra.run.trial117:> sudo dd if=/var/lib/ceph/osd/ceph-2/fsid of=/dev/stdout 2026-02-25T12:00:58.223 DEBUG:teuthology.orchestra.run.trial117:> set -ex 2026-02-25T12:00:58.224 DEBUG:teuthology.orchestra.run.trial117:> sudo dd if=/var/lib/ceph/osd/ceph-3/fsid of=/dev/stdout 2026-02-25T12:00:58.250 INFO:tasks.ceph.mgr.x.trial117.stderr:2026-02-25T12:00:58.247+0000 7ff10279b280 -1 mgr[py] Loading module cephadm in main interpreter 2026-02-25T12:00:58.258 DEBUG:teuthology.orchestra.run.trial117:> set -ex 2026-02-25T12:00:58.258 DEBUG:teuthology.orchestra.run.trial117:> sudo dd if=/var/lib/ceph/osd/ceph-4/fsid of=/dev/stdout 2026-02-25T12:00:58.310 DEBUG:teuthology.orchestra.run.trial117:> set -ex 2026-02-25T12:00:58.310 DEBUG:teuthology.orchestra.run.trial117:> sudo dd if=/var/lib/ceph/osd/ceph-5/fsid of=/dev/stdout 2026-02-25T12:00:58.360 DEBUG:teuthology.orchestra.run.trial117:> set -ex 2026-02-25T12:00:58.361 DEBUG:teuthology.orchestra.run.trial117:> sudo dd if=/var/lib/ceph/osd/ceph-6/fsid of=/dev/stdout 2026-02-25T12:00:58.383 INFO:tasks.ceph.mgr.x.trial117.stderr:2026-02-25T12:00:58.379+0000 7ff10279b280 -1 mgr[py] Loading module osd_support in main interpreter 2026-02-25T12:00:58.384 INFO:tasks.ceph.mgr.x.trial117.stderr:2026-02-25T12:00:58.379+0000 7ff10279b280 -1 mgr[py] Loading module stats in main interpreter 2026-02-25T12:00:58.387 INFO:tasks.ceph.mgr.x.trial117.stderr:2026-02-25T12:00:58.383+0000 7ff10279b280 -1 mgr[py] Loading module insights in main interpreter 2026-02-25T12:00:58.389 INFO:tasks.ceph.mgr.x.trial117.stderr:2026-02-25T12:00:58.387+0000 7ff10279b280 -1 mgr[py] Loading module crash in main interpreter 2026-02-25T12:00:58.390 DEBUG:teuthology.orchestra.run.trial117:> set -ex 2026-02-25T12:00:58.390 DEBUG:teuthology.orchestra.run.trial117:> sudo dd if=/var/lib/ceph/osd/ceph-7/fsid of=/dev/stdout 2026-02-25T12:00:58.391 INFO:tasks.ceph.mgr.x.trial117.stderr:2026-02-25T12:00:58.387+0000 7ff10279b280 -1 mgr[py] Loading module selftest in main interpreter 2026-02-25T12:00:58.394 INFO:tasks.ceph.mgr.x.trial117.stderr:2026-02-25T12:00:58.391+0000 7ff10279b280 -1 mgr[py] Loading module snap_schedule in main interpreter 2026-02-25T12:00:58.398 DEBUG:teuthology.orchestra.run.trial117:> set -ex 2026-02-25T12:00:58.398 DEBUG:teuthology.orchestra.run.trial117:> sudo dd if=/var/lib/ceph/osd/ceph-8/fsid of=/dev/stdout 2026-02-25T12:00:58.401 INFO:tasks.ceph.mgr.x.trial117.stderr:2026-02-25T12:00:58.399+0000 7ff10279b280 -1 mgr[py] Loading module rbd_support in main interpreter 2026-02-25T12:00:58.408 DEBUG:teuthology.orchestra.run.trial117:> set -ex 2026-02-25T12:00:58.408 DEBUG:teuthology.orchestra.run.trial117:> sudo dd if=/var/lib/ceph/osd/ceph-9/fsid of=/dev/stdout 2026-02-25T12:00:58.418 INFO:tasks.ceph.mgr.x.trial117.stderr:2026-02-25T12:00:58.415+0000 7ff10279b280 -1 mgr[py] Loading module mirroring in main interpreter 2026-02-25T12:00:58.425 DEBUG:teuthology.orchestra.run.trial117:> sudo ceph --cluster ceph osd new a879bc90-f2aa-4692-98ac-d45e56987af0 0 2026-02-25T12:00:58.428 INFO:tasks.ceph.mgr.x.trial117.stderr:2026-02-25T12:00:58.423+0000 7ff10279b280 -1 mgr[py] Loading module orchestrator in main interpreter 2026-02-25T12:00:58.428 INFO:tasks.ceph.mgr.x.trial117.stderr:2026-02-25T12:00:58.423+0000 7ff10279b280 -1 mgr[py] Loading module dashboard in main interpreter 2026-02-25T12:00:58.446 INFO:tasks.ceph.mgr.x.trial117.stderr:ERROR:root:Module 'jmespath' is not installed. 2026-02-25T12:00:58.468 INFO:tasks.ceph.mgr.x.trial117.stderr:ERROR:dashboard.controllers.nvmeof:Failed to import NVMeoFClient and related components: cannot import name 'NVMeoFClient' from 'dashboard.services.nvmeof_client' (/usr/share/ceph/mgr/dashboard/services/nvmeof_client.py) 2026-02-25T12:00:58.474 INFO:tasks.ceph.mgr.x.trial117.stderr:ERROR:sso:Module 'jmespath' is not installed. 2026-02-25T12:00:58.519 INFO:tasks.ceph.mgr.x.trial117.stderr:ERROR:root:Module 'xmltodict' is not installed. 2026-02-25T12:00:58.525 INFO:teuthology.orchestra.run.trial117.stdout:0 2026-02-25T12:00:58.525 INFO:tasks.ceph.mgr.x.trial117.stderr:2026-02-25T12:00:58.523+0000 7ff10279b280 -1 mgr[py] Loading module influx in main interpreter 2026-02-25T12:00:58.527 INFO:tasks.ceph.mgr.x.trial117.stderr:2026-02-25T12:00:58.523+0000 7ff10279b280 -1 mgr[py] Loading module alerts in main interpreter 2026-02-25T12:00:58.529 INFO:tasks.ceph.mgr.x.trial117.stderr:2026-02-25T12:00:58.527+0000 7ff10279b280 -1 mgr[py] Loading module volumes in main interpreter 2026-02-25T12:00:58.535 DEBUG:teuthology.orchestra.run.trial117:> sudo ceph --cluster ceph osd new 78845355-1113-42e3-9453-cbd686dd4cfb 1 2026-02-25T12:00:58.558 INFO:tasks.ceph.mgr.x.trial117.stderr:2026-02-25T12:00:58.555+0000 7ff10279b280 -1 mgr[py] Loading module rgw in main interpreter 2026-02-25T12:00:58.562 INFO:tasks.ceph.mgr.x.trial117.stderr:2026-02-25T12:00:58.559+0000 7ff10279b280 -1 mgr[py] Loading module iostat in main interpreter 2026-02-25T12:00:58.562 INFO:tasks.ceph.mgr.x.trial117.stderr:2026-02-25T12:00:58.559+0000 7ff10279b280 -1 mgr[py] Loading module progress in main interpreter 2026-02-25T12:00:58.565 INFO:tasks.ceph.mgr.x.trial117.stderr:2026-02-25T12:00:58.563+0000 7ff10279b280 -1 mgr[py] Loading module devicehealth in main interpreter 2026-02-25T12:00:58.568 INFO:tasks.ceph.mgr.x.trial117.stderr:2026-02-25T12:00:58.563+0000 7ff10279b280 -1 mgr[py] Loading module diskprediction_local in main interpreter 2026-02-25T12:00:58.682 INFO:teuthology.orchestra.run.trial117.stdout:1 2026-02-25T12:00:58.691 DEBUG:teuthology.orchestra.run.trial117:> sudo ceph --cluster ceph osd new f96856b9-ffb4-4170-918a-dcb7374aebbb 2 2026-02-25T12:00:58.694 INFO:tasks.ceph.mgr.x.trial117.stderr:2026-02-25T12:00:58.691+0000 7ff10279b280 -1 mgr[py] Loading module pg_autoscaler in main interpreter 2026-02-25T12:00:58.765 INFO:tasks.ceph.mgr.x.trial117.stderr:DEBUG:mgr_util:locking with 5 timeout 2026-02-25T12:00:58.766 INFO:tasks.ceph.mgr.x.trial117.stderr:DEBUG:mgr_util:locked 2026-02-25T12:00:58.766 INFO:tasks.ceph.mgr.x.trial117.stderr:DEBUG:mgr_util:locking with 5 timeout 2026-02-25T12:00:58.766 INFO:tasks.ceph.mgr.x.trial117.stderr:DEBUG:mgr_util:locking with 5 timeout 2026-02-25T12:00:58.766 INFO:tasks.ceph.mgr.x.trial117.stderr:DEBUG:mgr_util:locked 2026-02-25T12:00:58.766 INFO:tasks.ceph.mgr.x.trial117.stderr:DEBUG:mgr_util:locking with 5 timeout 2026-02-25T12:00:58.766 INFO:tasks.ceph.mgr.x.trial117.stderr:DEBUG:mgr_util:locking with 5 timeout 2026-02-25T12:00:58.767 INFO:tasks.ceph.mgr.x.trial117.stderr:DEBUG:mgr_util:locked 2026-02-25T12:00:58.767 INFO:tasks.ceph.mgr.x.trial117.stderr:DEBUG:mgr_util:locking with 5 timeout 2026-02-25T12:00:58.767 INFO:tasks.ceph.mgr.x.trial117.stderr:DEBUG:mgr_util:locking with 5 timeout 2026-02-25T12:00:58.767 INFO:tasks.ceph.mgr.x.trial117.stderr:DEBUG:mgr_util:locking with 5 timeout 2026-02-25T12:00:58.768 INFO:tasks.ceph.mgr.x.trial117.stderr:DEBUG:mgr_util:locking with 5 timeout 2026-02-25T12:00:58.768 INFO:tasks.ceph.mgr.x.trial117.stderr:DEBUG:mgr_util:locked 2026-02-25T12:00:58.768 INFO:tasks.ceph.mgr.x.trial117.stderr:DEBUG:mgr_util:locked 2026-02-25T12:00:58.768 INFO:tasks.ceph.mgr.x.trial117.stderr:DEBUG:mgr_util:locking with 5 timeout 2026-02-25T12:00:58.768 INFO:tasks.ceph.mgr.x.trial117.stderr:DEBUG:mgr_util:locked 2026-02-25T12:00:58.768 INFO:tasks.ceph.mgr.x.trial117.stderr:DEBUG:mgr_util:locked 2026-02-25T12:00:58.768 INFO:tasks.ceph.mgr.x.trial117.stderr:DEBUG:mgr_util:locked 2026-02-25T12:00:58.768 INFO:tasks.ceph.mgr.x.trial117.stderr:DEBUG:mgr_util:locked 2026-02-25T12:00:58.768 INFO:tasks.ceph.mgr.x.trial117.stderr:DEBUG:mgr_util:locked 2026-02-25T12:00:58.789 INFO:teuthology.orchestra.run.trial117.stdout:2 2026-02-25T12:00:58.800 DEBUG:teuthology.orchestra.run.trial117:> sudo ceph --cluster ceph osd new 5d49c0b4-c47f-466c-bb15-52ef5e96fd02 3 2026-02-25T12:00:58.898 INFO:teuthology.orchestra.run.trial117.stdout:3 2026-02-25T12:00:58.907 DEBUG:teuthology.orchestra.run.trial117:> sudo ceph --cluster ceph osd new 8327850a-7ec7-4203-ae90-0a54b25d3a35 4 2026-02-25T12:00:59.007 INFO:teuthology.orchestra.run.trial117.stdout:4 2026-02-25T12:00:59.018 DEBUG:teuthology.orchestra.run.trial117:> sudo ceph --cluster ceph osd new 04975d9f-b1e5-465b-ae33-7a342cf7f976 5 2026-02-25T12:00:59.116 INFO:teuthology.orchestra.run.trial117.stdout:5 2026-02-25T12:00:59.127 DEBUG:teuthology.orchestra.run.trial117:> sudo ceph --cluster ceph osd new 6d6dfec7-2127-44bb-ad90-d3924e3d0902 6 2026-02-25T12:00:59.222 INFO:teuthology.orchestra.run.trial117.stdout:6 2026-02-25T12:00:59.232 DEBUG:teuthology.orchestra.run.trial117:> sudo ceph --cluster ceph osd new 617a797b-c363-4bde-a19e-ce36bedaba09 7 2026-02-25T12:00:59.329 INFO:teuthology.orchestra.run.trial117.stdout:7 2026-02-25T12:00:59.340 DEBUG:teuthology.orchestra.run.trial117:> sudo ceph --cluster ceph osd new 8a6d937f-608e-4cf8-9d91-fc6104b9ea99 8 2026-02-25T12:00:59.439 INFO:teuthology.orchestra.run.trial117.stdout:8 2026-02-25T12:00:59.448 DEBUG:teuthology.orchestra.run.trial117:> sudo ceph --cluster ceph osd new 5dba0073-60fa-4ce0-baab-a34df860f7b1 9 2026-02-25T12:00:59.547 INFO:teuthology.orchestra.run.trial117.stdout:9 2026-02-25T12:00:59.556 INFO:tasks.ceph.osd.0:Restarting daemon 2026-02-25T12:00:59.556 DEBUG:teuthology.orchestra.run.trial117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 0 2026-02-25T12:00:59.557 INFO:tasks.ceph.osd.0:Started 2026-02-25T12:00:59.558 INFO:tasks.ceph.osd.1:Restarting daemon 2026-02-25T12:00:59.558 DEBUG:teuthology.orchestra.run.trial117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 1 2026-02-25T12:00:59.559 INFO:tasks.ceph.osd.1:Started 2026-02-25T12:00:59.559 INFO:tasks.ceph.osd.2:Restarting daemon 2026-02-25T12:00:59.559 DEBUG:teuthology.orchestra.run.trial117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 2 2026-02-25T12:00:59.561 INFO:tasks.ceph.osd.2:Started 2026-02-25T12:00:59.561 INFO:tasks.ceph.osd.3:Restarting daemon 2026-02-25T12:00:59.561 DEBUG:teuthology.orchestra.run.trial117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 3 2026-02-25T12:00:59.562 INFO:tasks.ceph.osd.3:Started 2026-02-25T12:00:59.563 INFO:tasks.ceph.osd.4:Restarting daemon 2026-02-25T12:00:59.563 DEBUG:teuthology.orchestra.run.trial117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 4 2026-02-25T12:00:59.564 INFO:tasks.ceph.osd.4:Started 2026-02-25T12:00:59.564 INFO:tasks.ceph.osd.5:Restarting daemon 2026-02-25T12:00:59.564 DEBUG:teuthology.orchestra.run.trial117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 5 2026-02-25T12:00:59.566 INFO:tasks.ceph.osd.5:Started 2026-02-25T12:00:59.566 INFO:tasks.ceph.osd.6:Restarting daemon 2026-02-25T12:00:59.566 DEBUG:teuthology.orchestra.run.trial117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 6 2026-02-25T12:00:59.567 INFO:tasks.ceph.osd.6:Started 2026-02-25T12:00:59.567 INFO:tasks.ceph.osd.7:Restarting daemon 2026-02-25T12:00:59.567 DEBUG:teuthology.orchestra.run.trial117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 7 2026-02-25T12:00:59.569 INFO:tasks.ceph.osd.7:Started 2026-02-25T12:00:59.569 INFO:tasks.ceph.osd.8:Restarting daemon 2026-02-25T12:00:59.569 DEBUG:teuthology.orchestra.run.trial117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 8 2026-02-25T12:00:59.570 INFO:tasks.ceph.osd.8:Started 2026-02-25T12:00:59.570 INFO:tasks.ceph.osd.9:Restarting daemon 2026-02-25T12:00:59.570 DEBUG:teuthology.orchestra.run.trial117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 9 2026-02-25T12:00:59.571 INFO:tasks.ceph.osd.9:Started 2026-02-25T12:00:59.571 DEBUG:teuthology.orchestra.run.trial117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2026-02-25T12:00:59.673 INFO:teuthology.orchestra.run.trial117.stdout: 2026-02-25T12:00:59.673 INFO:teuthology.orchestra.run.trial117.stdout:{"epoch":12,"fsid":"43e63318-add1-4399-a1fc-80188594353b","created":"2026-02-25T12:00:57.865568+0000","modified":"2026-02-25T12:00:59.546090+0000","last_up_change":"0.000000","last_in_change":"2026-02-25T12:00:59.546090+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":10,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"a879bc90-f2aa-4692-98ac-d45e56987af0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"78845355-1113-42e3-9453-cbd686dd4cfb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"f96856b9-ffb4-4170-918a-dcb7374aebbb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"5d49c0b4-c47f-466c-bb15-52ef5e96fd02","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"8327850a-7ec7-4203-ae90-0a54b25d3a35","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"04975d9f-b1e5-465b-ae33-7a342cf7f976","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"6d6dfec7-2127-44bb-ad90-d3924e3d0902","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"617a797b-c363-4bde-a19e-ce36bedaba09","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":8,"uuid":"8a6d937f-608e-4cf8-9d91-fc6104b9ea99","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":9,"uuid":"5dba0073-60fa-4ce0-baab-a34df860f7b1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-25T12:00:59.683 INFO:tasks.ceph.ceph_manager.ceph:[] 2026-02-25T12:00:59.683 INFO:tasks.ceph:Waiting for OSDs to come up 2026-02-25T12:00:59.728 INFO:tasks.ceph.mgr.x.trial117.stderr:2026-02-25T12:00:59.723+0000 7ff02c929640 -1 mgr.server handle_report got status from non-daemon mon.a 2026-02-25T12:01:00.002 INFO:tasks.ceph.osd.6.trial117.stderr:2026-02-25T12:00:59.999+0000 7fe19e5c7a40 -1 Falling back to public interface 2026-02-25T12:01:00.009 INFO:tasks.ceph.osd.0.trial117.stderr:2026-02-25T12:01:00.007+0000 7f8a1ddc9a40 -1 Falling back to public interface 2026-02-25T12:01:00.034 INFO:tasks.ceph.osd.3.trial117.stderr:2026-02-25T12:01:00.031+0000 7fb68e5d2a40 -1 Falling back to public interface 2026-02-25T12:01:00.037 INFO:tasks.ceph.osd.1.trial117.stderr:2026-02-25T12:01:00.035+0000 7fbcd0ad6a40 -1 Falling back to public interface 2026-02-25T12:01:00.041 INFO:tasks.ceph.osd.9.trial117.stderr:2026-02-25T12:01:00.039+0000 7f65013cfa40 -1 Falling back to public interface 2026-02-25T12:01:00.058 INFO:tasks.ceph.osd.4.trial117.stderr:2026-02-25T12:01:00.055+0000 7efcafcc6a40 -1 Falling back to public interface 2026-02-25T12:01:00.058 INFO:tasks.ceph.osd.7.trial117.stderr:2026-02-25T12:01:00.055+0000 7f0399e5ba40 -1 Falling back to public interface 2026-02-25T12:01:00.077 INFO:tasks.ceph.osd.2.trial117.stderr:2026-02-25T12:01:00.075+0000 7f0b5a9bea40 -1 Falling back to public interface 2026-02-25T12:01:00.093 INFO:tasks.ceph.osd.5.trial117.stderr:2026-02-25T12:01:00.091+0000 7f0fe6ac4a40 -1 Falling back to public interface 2026-02-25T12:01:00.126 INFO:tasks.ceph.osd.8.trial117.stderr:2026-02-25T12:01:00.123+0000 7fddc6c06a40 -1 Falling back to public interface 2026-02-25T12:01:00.689 DEBUG:teuthology.orchestra.run.trial117:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2026-02-25T12:01:00.774 INFO:tasks.ceph.osd.0.trial117.stderr:2026-02-25T12:01:00.771+0000 7f8a1ddc9a40 -1 osd.0 0 log_to_monitors true 2026-02-25T12:01:00.804 INFO:tasks.ceph.osd.6.trial117.stderr:2026-02-25T12:01:00.799+0000 7fe19e5c7a40 -1 osd.6 0 log_to_monitors true 2026-02-25T12:01:00.829 INFO:tasks.ceph.osd.1.trial117.stderr:2026-02-25T12:01:00.827+0000 7fbcd0ad6a40 -1 osd.1 0 log_to_monitors true 2026-02-25T12:01:00.831 INFO:tasks.ceph.osd.3.trial117.stderr:2026-02-25T12:01:00.827+0000 7fb68e5d2a40 -1 osd.3 0 log_to_monitors true 2026-02-25T12:01:00.840 INFO:teuthology.misc.health.trial117.stdout: 2026-02-25T12:01:00.840 INFO:teuthology.misc.health.trial117.stdout:{"epoch":12,"fsid":"43e63318-add1-4399-a1fc-80188594353b","created":"2026-02-25T12:00:57.865568+0000","modified":"2026-02-25T12:00:59.546090+0000","last_up_change":"0.000000","last_in_change":"2026-02-25T12:00:59.546090+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":10,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"a879bc90-f2aa-4692-98ac-d45e56987af0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"78845355-1113-42e3-9453-cbd686dd4cfb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"f96856b9-ffb4-4170-918a-dcb7374aebbb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"5d49c0b4-c47f-466c-bb15-52ef5e96fd02","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"8327850a-7ec7-4203-ae90-0a54b25d3a35","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"04975d9f-b1e5-465b-ae33-7a342cf7f976","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"6d6dfec7-2127-44bb-ad90-d3924e3d0902","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"617a797b-c363-4bde-a19e-ce36bedaba09","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":8,"uuid":"8a6d937f-608e-4cf8-9d91-fc6104b9ea99","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":9,"uuid":"5dba0073-60fa-4ce0-baab-a34df860f7b1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-25T12:01:00.853 DEBUG:teuthology.misc:0 of 10 OSDs are up 2026-02-25T12:01:00.860 INFO:tasks.ceph.osd.4.trial117.stderr:2026-02-25T12:01:00.855+0000 7efcafcc6a40 -1 osd.4 0 log_to_monitors true 2026-02-25T12:01:00.860 INFO:tasks.ceph.osd.5.trial117.stderr:2026-02-25T12:01:00.855+0000 7f0fe6ac4a40 -1 osd.5 0 log_to_monitors true 2026-02-25T12:01:00.876 INFO:tasks.ceph.osd.7.trial117.stderr:2026-02-25T12:01:00.871+0000 7f0399e5ba40 -1 osd.7 0 log_to_monitors true 2026-02-25T12:01:00.921 INFO:tasks.ceph.osd.9.trial117.stderr:2026-02-25T12:01:00.919+0000 7f65013cfa40 -1 osd.9 0 log_to_monitors true 2026-02-25T12:01:00.945 INFO:tasks.ceph.osd.8.trial117.stderr:2026-02-25T12:01:00.943+0000 7fddc6c06a40 -1 osd.8 0 log_to_monitors true 2026-02-25T12:01:00.952 INFO:tasks.ceph.osd.2.trial117.stderr:2026-02-25T12:01:00.947+0000 7f0b5a9bea40 -1 osd.2 0 log_to_monitors true 2026-02-25T12:01:01.879 INFO:tasks.ceph.osd.4.trial117.stderr:2026-02-25T12:01:01.875+0000 7efcabc70640 -1 osd.4 0 waiting for initial osdmap 2026-02-25T12:01:01.879 INFO:tasks.ceph.osd.5.trial117.stderr:2026-02-25T12:01:01.875+0000 7f0fe2a6e640 -1 osd.5 0 waiting for initial osdmap 2026-02-25T12:01:01.880 INFO:tasks.ceph.osd.0.trial117.stderr:2026-02-25T12:01:01.875+0000 7f8a19d73640 -1 osd.0 0 waiting for initial osdmap 2026-02-25T12:01:01.880 INFO:tasks.ceph.osd.6.trial117.stderr:2026-02-25T12:01:01.879+0000 7fe19a571640 -1 osd.6 0 waiting for initial osdmap 2026-02-25T12:01:01.881 INFO:tasks.ceph.osd.1.trial117.stderr:2026-02-25T12:01:01.879+0000 7fbccca80640 -1 osd.1 0 waiting for initial osdmap 2026-02-25T12:01:01.881 INFO:tasks.ceph.osd.3.trial117.stderr:2026-02-25T12:01:01.879+0000 7fb68a57c640 -1 osd.3 0 waiting for initial osdmap 2026-02-25T12:01:01.882 INFO:tasks.ceph.osd.4.trial117.stderr:2026-02-25T12:01:01.879+0000 7efca333f640 -1 osd.4 14 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2026-02-25T12:01:01.882 INFO:tasks.ceph.osd.5.trial117.stderr:2026-02-25T12:01:01.879+0000 7f0fda13d640 -1 osd.5 14 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2026-02-25T12:01:01.884 INFO:tasks.ceph.osd.1.trial117.stderr:2026-02-25T12:01:01.879+0000 7fbcc414f640 -1 osd.1 14 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2026-02-25T12:01:01.884 INFO:tasks.ceph.osd.0.trial117.stderr:2026-02-25T12:01:01.879+0000 7f8a11442640 -1 osd.0 14 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2026-02-25T12:01:01.884 INFO:tasks.ceph.osd.6.trial117.stderr:2026-02-25T12:01:01.879+0000 7fe191c40640 -1 osd.6 14 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2026-02-25T12:01:01.884 INFO:tasks.ceph.osd.3.trial117.stderr:2026-02-25T12:01:01.879+0000 7fb685c53640 -1 osd.3 14 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2026-02-25T12:01:02.884 INFO:tasks.ceph.osd.9.trial117.stderr:2026-02-25T12:01:02.879+0000 7f64fd379640 -1 osd.9 0 waiting for initial osdmap 2026-02-25T12:01:02.885 INFO:tasks.ceph.osd.7.trial117.stderr:2026-02-25T12:01:02.879+0000 7f0395e05640 -1 osd.7 0 waiting for initial osdmap 2026-02-25T12:01:02.885 INFO:tasks.ceph.osd.8.trial117.stderr:2026-02-25T12:01:02.883+0000 7fddc2bb0640 -1 osd.8 0 waiting for initial osdmap 2026-02-25T12:01:02.886 INFO:tasks.ceph.osd.2.trial117.stderr:2026-02-25T12:01:02.883+0000 7f0b56968640 -1 osd.2 0 waiting for initial osdmap 2026-02-25T12:01:02.886 INFO:tasks.ceph.osd.9.trial117.stderr:2026-02-25T12:01:02.883+0000 7f64f8a50640 -1 osd.9 15 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2026-02-25T12:01:02.887 INFO:tasks.ceph.osd.7.trial117.stderr:2026-02-25T12:01:02.883+0000 7f03914dc640 -1 osd.7 15 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2026-02-25T12:01:02.888 INFO:tasks.ceph.osd.2.trial117.stderr:2026-02-25T12:01:02.883+0000 7f0b4e037640 -1 osd.2 15 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2026-02-25T12:01:02.888 INFO:tasks.ceph.osd.8.trial117.stderr:2026-02-25T12:01:02.883+0000 7fddbe287640 -1 osd.8 15 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2026-02-25T12:01:07.860 DEBUG:teuthology.orchestra.run.trial117:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2026-02-25T12:01:08.011 INFO:teuthology.misc.health.trial117.stdout: 2026-02-25T12:01:08.011 INFO:teuthology.misc.health.trial117.stdout:{"epoch":16,"fsid":"43e63318-add1-4399-a1fc-80188594353b","created":"2026-02-25T12:00:57.865568+0000","modified":"2026-02-25T12:01:03.880119+0000","last_up_change":"2026-02-25T12:01:03.880119+0000","last_in_change":"2026-02-25T12:00:59.546090+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":10,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"a879bc90-f2aa-4692-98ac-d45e56987af0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.117:6810","nonce":2144854447},{"type":"v1","addr":"10.20.193.117:6811","nonce":2144854447}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.117:6812","nonce":2144854447},{"type":"v1","addr":"10.20.193.117:6813","nonce":2144854447}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.117:6816","nonce":2144854447},{"type":"v1","addr":"10.20.193.117:6817","nonce":2144854447}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.117:6814","nonce":2144854447},{"type":"v1","addr":"10.20.193.117:6815","nonce":2144854447}]},"public_addr":"10.20.193.117:6811/2144854447","cluster_addr":"10.20.193.117:6813/2144854447","heartbeat_back_addr":"10.20.193.117:6817/2144854447","heartbeat_front_addr":"10.20.193.117:6815/2144854447","state":["exists","up"]},{"osd":1,"uuid":"78845355-1113-42e3-9453-cbd686dd4cfb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.117:6826","nonce":491871890},{"type":"v1","addr":"10.20.193.117:6827","nonce":491871890}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.117:6828","nonce":491871890},{"type":"v1","addr":"10.20.193.117:6829","nonce":491871890}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.117:6832","nonce":491871890},{"type":"v1","addr":"10.20.193.117:6833","nonce":491871890}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.117:6830","nonce":491871890},{"type":"v1","addr":"10.20.193.117:6831","nonce":491871890}]},"public_addr":"10.20.193.117:6827/491871890","cluster_addr":"10.20.193.117:6829/491871890","heartbeat_back_addr":"10.20.193.117:6833/491871890","heartbeat_front_addr":"10.20.193.117:6831/491871890","state":["exists","up"]},{"osd":2,"uuid":"f96856b9-ffb4-4170-918a-dcb7374aebbb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.117:6858","nonce":3832000820},{"type":"v1","addr":"10.20.193.117:6859","nonce":3832000820}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.117:6860","nonce":3832000820},{"type":"v1","addr":"10.20.193.117:6861","nonce":3832000820}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.117:6864","nonce":3832000820},{"type":"v1","addr":"10.20.193.117:6865","nonce":3832000820}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.117:6862","nonce":3832000820},{"type":"v1","addr":"10.20.193.117:6863","nonce":3832000820}]},"public_addr":"10.20.193.117:6859/3832000820","cluster_addr":"10.20.193.117:6861/3832000820","heartbeat_back_addr":"10.20.193.117:6865/3832000820","heartbeat_front_addr":"10.20.193.117:6863/3832000820","state":["exists","up"]},{"osd":3,"uuid":"5d49c0b4-c47f-466c-bb15-52ef5e96fd02","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.117:6818","nonce":2181619050},{"type":"v1","addr":"10.20.193.117:6819","nonce":2181619050}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.117:6820","nonce":2181619050},{"type":"v1","addr":"10.20.193.117:6821","nonce":2181619050}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.117:6824","nonce":2181619050},{"type":"v1","addr":"10.20.193.117:6825","nonce":2181619050}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.117:6822","nonce":2181619050},{"type":"v1","addr":"10.20.193.117:6823","nonce":2181619050}]},"public_addr":"10.20.193.117:6819/2181619050","cluster_addr":"10.20.193.117:6821/2181619050","heartbeat_back_addr":"10.20.193.117:6825/2181619050","heartbeat_front_addr":"10.20.193.117:6823/2181619050","state":["exists","up"]},{"osd":4,"uuid":"8327850a-7ec7-4203-ae90-0a54b25d3a35","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.117:6843","nonce":2433940125},{"type":"v1","addr":"10.20.193.117:6845","nonce":2433940125}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.117:6847","nonce":2433940125},{"type":"v1","addr":"10.20.193.117:6849","nonce":2433940125}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.117:6855","nonce":2433940125},{"type":"v1","addr":"10.20.193.117:6857","nonce":2433940125}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.117:6851","nonce":2433940125},{"type":"v1","addr":"10.20.193.117:6853","nonce":2433940125}]},"public_addr":"10.20.193.117:6845/2433940125","cluster_addr":"10.20.193.117:6849/2433940125","heartbeat_back_addr":"10.20.193.117:6857/2433940125","heartbeat_front_addr":"10.20.193.117:6853/2433940125","state":["exists","up"]},{"osd":5,"uuid":"04975d9f-b1e5-465b-ae33-7a342cf7f976","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.117:6866","nonce":3722132383},{"type":"v1","addr":"10.20.193.117:6867","nonce":3722132383}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.117:6868","nonce":3722132383},{"type":"v1","addr":"10.20.193.117:6869","nonce":3722132383}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.117:6872","nonce":3722132383},{"type":"v1","addr":"10.20.193.117:6873","nonce":3722132383}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.117:6870","nonce":3722132383},{"type":"v1","addr":"10.20.193.117:6871","nonce":3722132383}]},"public_addr":"10.20.193.117:6867/3722132383","cluster_addr":"10.20.193.117:6869/3722132383","heartbeat_back_addr":"10.20.193.117:6873/3722132383","heartbeat_front_addr":"10.20.193.117:6871/3722132383","state":["exists","up"]},{"osd":6,"uuid":"6d6dfec7-2127-44bb-ad90-d3924e3d0902","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.117:6802","nonce":3926541475},{"type":"v1","addr":"10.20.193.117:6803","nonce":3926541475}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.117:6804","nonce":3926541475},{"type":"v1","addr":"10.20.193.117:6805","nonce":3926541475}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.117:6808","nonce":3926541475},{"type":"v1","addr":"10.20.193.117:6809","nonce":3926541475}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.117:6806","nonce":3926541475},{"type":"v1","addr":"10.20.193.117:6807","nonce":3926541475}]},"public_addr":"10.20.193.117:6803/3926541475","cluster_addr":"10.20.193.117:6805/3926541475","heartbeat_back_addr":"10.20.193.117:6809/3926541475","heartbeat_front_addr":"10.20.193.117:6807/3926541475","state":["exists","up"]},{"osd":7,"uuid":"617a797b-c363-4bde-a19e-ce36bedaba09","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.117:6842","nonce":1628495216},{"type":"v1","addr":"10.20.193.117:6844","nonce":1628495216}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.117:6846","nonce":1628495216},{"type":"v1","addr":"10.20.193.117:6848","nonce":1628495216}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.117:6854","nonce":1628495216},{"type":"v1","addr":"10.20.193.117:6856","nonce":1628495216}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.117:6850","nonce":1628495216},{"type":"v1","addr":"10.20.193.117:6852","nonce":1628495216}]},"public_addr":"10.20.193.117:6844/1628495216","cluster_addr":"10.20.193.117:6848/1628495216","heartbeat_back_addr":"10.20.193.117:6856/1628495216","heartbeat_front_addr":"10.20.193.117:6852/1628495216","state":["exists","up"]},{"osd":8,"uuid":"8a6d937f-608e-4cf8-9d91-fc6104b9ea99","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.117:6874","nonce":3404023726},{"type":"v1","addr":"10.20.193.117:6875","nonce":3404023726}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.117:6876","nonce":3404023726},{"type":"v1","addr":"10.20.193.117:6877","nonce":3404023726}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.117:6880","nonce":3404023726},{"type":"v1","addr":"10.20.193.117:6881","nonce":3404023726}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.117:6878","nonce":3404023726},{"type":"v1","addr":"10.20.193.117:6879","nonce":3404023726}]},"public_addr":"10.20.193.117:6875/3404023726","cluster_addr":"10.20.193.117:6877/3404023726","heartbeat_back_addr":"10.20.193.117:6881/3404023726","heartbeat_front_addr":"10.20.193.117:6879/3404023726","state":["exists","up"]},{"osd":9,"uuid":"5dba0073-60fa-4ce0-baab-a34df860f7b1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.117:6834","nonce":387956821},{"type":"v1","addr":"10.20.193.117:6835","nonce":387956821}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.117:6836","nonce":387956821},{"type":"v1","addr":"10.20.193.117:6837","nonce":387956821}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.117:6840","nonce":387956821},{"type":"v1","addr":"10.20.193.117:6841","nonce":387956821}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.117:6838","nonce":387956821},{"type":"v1","addr":"10.20.193.117:6839","nonce":387956821}]},"public_addr":"10.20.193.117:6835/387956821","cluster_addr":"10.20.193.117:6837/387956821","heartbeat_back_addr":"10.20.193.117:6841/387956821","heartbeat_front_addr":"10.20.193.117:6839/387956821","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-25T12:01:08.021 DEBUG:teuthology.misc:10 of 10 OSDs are up 2026-02-25T12:01:08.021 INFO:tasks.ceph:Creating RBD pool 2026-02-25T12:01:08.021 DEBUG:teuthology.orchestra.run.trial117:> sudo ceph --cluster ceph osd pool create rbd 8 2026-02-25T12:01:08.888 INFO:teuthology.orchestra.run.trial117.stderr:pool 'rbd' created 2026-02-25T12:01:08.902 DEBUG:teuthology.orchestra.run.trial117:> rbd --cluster ceph pool init rbd 2026-02-25T12:01:11.901 INFO:tasks.ceph:Starting mds daemons in cluster ceph... 2026-02-25T12:01:11.902 DEBUG:teuthology.orchestra.run.trial117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config log 1 --format=json 2026-02-25T12:01:11.902 INFO:tasks.daemonwatchdog.daemon_watchdog:watchdog starting 2026-02-25T12:01:12.060 INFO:teuthology.orchestra.run.trial117.stdout: 2026-02-25T12:01:12.070 INFO:teuthology.orchestra.run.trial117.stdout:[{"version":2,"timestamp":"2026-02-25T12:00:57.906287+0000","name":"","changes":[{"name":"mgr/mgr_pool","new_value":"false"}]}] 2026-02-25T12:01:12.071 INFO:tasks.ceph_manager:config epoch is 2 2026-02-25T12:01:12.071 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2026-02-25T12:01:12.071 INFO:tasks.ceph.ceph_manager.ceph:waiting for mgr available 2026-02-25T12:01:12.071 DEBUG:teuthology.orchestra.run.trial117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr dump --format=json 2026-02-25T12:01:12.242 INFO:teuthology.orchestra.run.trial117.stdout: 2026-02-25T12:01:12.252 INFO:teuthology.orchestra.run.trial117.stdout:{"epoch":4,"flags":0,"active_gid":4100,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.117:6800","nonce":1123870964},{"type":"v1","addr":"10.20.193.117:6801","nonce":1123870964}]},"active_addr":"10.20.193.117:6801/1123870964","active_change":"2026-02-25T12:00:58.721093+0000","active_mgr_features":4544132024016699391,"available":true,"standbys":[],"modules":["iostat","nfs"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to, use commas to separate multiple","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"update_pg_upmap_activity":{"name":"update_pg_upmap_activity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Updates pg_upmap activity stats to be used in `balancer status detail`","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often the agent on each host will gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"The first TCP port the agent will try to bind to. Up to 1000 subsequent ports will be attempted if this port cannot be bound","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this option can facilitate debugging daemons that encounter runtime problems.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"How frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"Fraction of total system memory to divide among autotuned daemons. Converged systems hosting both Ceph and compute should set a lower value; systems with more than 10 GiB of memory per OSD may benefit from a slightly larger value so that the abundance of RAM can be fully utilized","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command persistent logging","long_desc":"","tags":[],"see_also":[]},"certificate_automated_rotation_enabled":{"name":"certificate_automated_rotation_enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"This flag controls whether cephadm automatically rotates certificates upon expiration.","long_desc":"","tags":[],"see_also":[]},"certificate_check_debug_mode":{"name":"certificate_check_debug_mode","type":"bool","level":"dev","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"FOR TESTING ONLY: This flag forces the certificate check instead of waiting for certificate_check_period.","long_desc":"","tags":[],"see_also":[]},"certificate_check_period":{"name":"certificate_check_period","type":"int","level":"advanced","flags":0,"default_value":"1","min":"0","max":"30","enum_allowed":[],"desc":"Specifies how often (in days) the certificate should be checked for validity.","long_desc":"","tags":[],"see_also":[]},"certificate_duration_days":{"name":"certificate_duration_days","type":"int","level":"advanced","flags":0,"default_value":"1095","min":"90","max":"3650","enum_allowed":[],"desc":"Specifies the duration of self-signed certificates generated and signed by the cephadm root CA","long_desc":"","tags":[],"see_also":[]},"certificate_renewal_threshold_days":{"name":"certificate_renewal_threshold_days","type":"int","level":"advanced","flags":0,"default_value":"30","min":"10","max":"90","enum_allowed":[],"desc":"Specifies the lead time in days to initiate certificate renewal before expiration.","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently Podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.28.1","min":"","max":"","enum_allowed":[],"desc":"Alertmanager container image","long_desc":"","tags":[],"see_also":[]},"container_image_alloy":{"name":"container_image_alloy","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/alloy:latest","min":"","max":"","enum_allowed":[],"desc":"Alloy container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"Elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:12.3.1","min":"","max":"","enum_allowed":[],"desc":"Grafana container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"Haproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_nginx":{"name":"container_image_nginx","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nginx:sclorg-nginx-126","min":"","max":"","enum_allowed":[],"desc":"Nginx container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.9.1","min":"","max":"","enum_allowed":[],"desc":"Node exporter container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.7","min":"","max":"","enum_allowed":[],"desc":"Nvmeof container image","long_desc":"","tags":[],"see_also":[]},"container_image_oauth2_proxy":{"name":"container_image_oauth2_proxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/oauth2-proxy/oauth2-proxy:v7.6.0","min":"","max":"","enum_allowed":[],"desc":"Oauth2 proxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v3.6.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba":{"name":"container_image_samba","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-server:devbuilds-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba_metrics":{"name":"container_image_samba_metrics","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-metrics:devbuilds-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba metrics container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"Snmp gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"Seconds for which to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"quay.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"Seconds for which to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"Seconds for which to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"Location of dashboards to include in Grafana","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"How frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether the hardware monitoring daemon be deployed on every host?","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at the debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"Max number of daemons per service per host: guard against mistakes and runaways","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"Max number of OSDs that will be drained simultaneously when OSDs are removed","long_desc":"","tags":[],"see_also":[]},"max_parallel_osd_upgrades":{"name":"max_parallel_osd_upgrades","type":"int","level":"advanced","flags":0,"default_value":"16","min":"","max":"","enum_allowed":[],"desc":"Maximum number of OSD daemons upgraded in parallel.","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"Mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default IP address for RedFish API (OOB management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"Location of alerts to include in Prometheus Alertmanager","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry URL for login purposes. This is not the default registry.","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"Cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Customized SSH config file to distribute to managed hosts","long_desc":"","tags":[],"see_also":[]},"ssh_keepalive_count_max":{"name":"ssh_keepalive_count_max","type":"int","level":"advanced","flags":0,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"How many sequential SSH connection liveness check failures before the host is marked offline","long_desc":"","tags":[],"see_also":[]},"ssh_keepalive_interval":{"name":"ssh_keepalive_interval","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"How often SSH connections are checked for liveness","long_desc":"","tags":[],"see_also":[]},"stray_daemon_check_interval":{"name":"stray_daemon_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"How frequently cephadm should check for the presence of stray daemons","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use the cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest to ensure that all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MANAGED_BY_CLUSTERS":{"name":"MANAGED_BY_CLUSTERS","type":"str","level":"advanced","flags":0,"default_value":"[]","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MULTICLUSTER_CONFIG":{"name":"MULTICLUSTER_CONFIG","type":"str","level":"advanced","flags":0,"default_value":"{}","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROM_ALERT_CREDENTIAL_CACHE_TTL":{"name":"PROM_ALERT_CREDENTIAL_CACHE_TTL","type":"int","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_HOSTNAME_PER_DAEMON":{"name":"RGW_HOSTNAME_PER_DAEMON","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crypto_caller":{"name":"crypto_caller","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sso_oauth2":{"name":"sso_oauth2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"healthcheck_history_max_entries":{"name":"healthcheck_history_max_entries","type":"int","level":"advanced","flags":1,"default_value":"1000","min":"","max":"","enum_allowed":[],"desc":"Maximum number of health check history entries to keep","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"secondary_zone_period_retry_limit":{"name":"secondary_zone_period_retry_limit","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"RGW module period update retry limit for secondary site","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"smb","can_run":true,"error_string":"","module_options":{"internal_store_backend":{"name":"internal_store_backend","type":"str","level":"dev","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"set internal store backend. for develoment and testing only","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"update_orchestration":{"name":"update_orchestration","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically update orchestration when smb resources are changed","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"pause_cloning":{"name":"pause_cloning","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Pause asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"pause_purging":{"name":"pause_purging","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Pause asynchronous subvolume purge threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"squid":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"tentacle":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"force_disabled_modules":{},"last_failure_osd_epoch":0,"active_clients":[{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"10.20.193.117:0","nonce":4260043879}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"10.20.193.117:0","nonce":1804512343}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"10.20.193.117:0","nonce":3053202647}]}]} 2026-02-25T12:01:12.254 INFO:tasks.ceph.ceph_manager.ceph:mgr available! 2026-02-25T12:01:12.254 INFO:tasks.ceph.ceph_manager.ceph:waiting for all up 2026-02-25T12:01:12.254 DEBUG:teuthology.orchestra.run.trial117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2026-02-25T12:01:12.413 INFO:teuthology.orchestra.run.trial117.stdout: 2026-02-25T12:01:12.413 INFO:teuthology.orchestra.run.trial117.stdout:{"epoch":20,"fsid":"43e63318-add1-4399-a1fc-80188594353b","created":"2026-02-25T12:00:57.865568+0000","modified":"2026-02-25T12:01:11.891867+0000","last_up_change":"2026-02-25T12:01:03.880119+0000","last_in_change":"2026-02-25T12:00:59.546090+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":10,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":false,"pools":[{"pool":1,"pool_name":"rbd","create_time":"2026-02-25T12:01:08.174903+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":2,"snap_epoch":20,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.5,"score_stable":2.5,"optimal_score":0.89999997615814209,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1.0000001192092896,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"a879bc90-f2aa-4692-98ac-d45e56987af0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":17,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.117:6810","nonce":2144854447},{"type":"v1","addr":"10.20.193.117:6811","nonce":2144854447}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.117:6812","nonce":2144854447},{"type":"v1","addr":"10.20.193.117:6813","nonce":2144854447}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.117:6816","nonce":2144854447},{"type":"v1","addr":"10.20.193.117:6817","nonce":2144854447}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.117:6814","nonce":2144854447},{"type":"v1","addr":"10.20.193.117:6815","nonce":2144854447}]},"public_addr":"10.20.193.117:6811/2144854447","cluster_addr":"10.20.193.117:6813/2144854447","heartbeat_back_addr":"10.20.193.117:6817/2144854447","heartbeat_front_addr":"10.20.193.117:6815/2144854447","state":["exists","up"]},{"osd":1,"uuid":"78845355-1113-42e3-9453-cbd686dd4cfb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.117:6826","nonce":491871890},{"type":"v1","addr":"10.20.193.117:6827","nonce":491871890}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.117:6828","nonce":491871890},{"type":"v1","addr":"10.20.193.117:6829","nonce":491871890}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.117:6832","nonce":491871890},{"type":"v1","addr":"10.20.193.117:6833","nonce":491871890}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.117:6830","nonce":491871890},{"type":"v1","addr":"10.20.193.117:6831","nonce":491871890}]},"public_addr":"10.20.193.117:6827/491871890","cluster_addr":"10.20.193.117:6829/491871890","heartbeat_back_addr":"10.20.193.117:6833/491871890","heartbeat_front_addr":"10.20.193.117:6831/491871890","state":["exists","up"]},{"osd":2,"uuid":"f96856b9-ffb4-4170-918a-dcb7374aebbb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.117:6858","nonce":3832000820},{"type":"v1","addr":"10.20.193.117:6859","nonce":3832000820}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.117:6860","nonce":3832000820},{"type":"v1","addr":"10.20.193.117:6861","nonce":3832000820}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.117:6864","nonce":3832000820},{"type":"v1","addr":"10.20.193.117:6865","nonce":3832000820}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.117:6862","nonce":3832000820},{"type":"v1","addr":"10.20.193.117:6863","nonce":3832000820}]},"public_addr":"10.20.193.117:6859/3832000820","cluster_addr":"10.20.193.117:6861/3832000820","heartbeat_back_addr":"10.20.193.117:6865/3832000820","heartbeat_front_addr":"10.20.193.117:6863/3832000820","state":["exists","up"]},{"osd":3,"uuid":"5d49c0b4-c47f-466c-bb15-52ef5e96fd02","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":17,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.117:6818","nonce":2181619050},{"type":"v1","addr":"10.20.193.117:6819","nonce":2181619050}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.117:6820","nonce":2181619050},{"type":"v1","addr":"10.20.193.117:6821","nonce":2181619050}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.117:6824","nonce":2181619050},{"type":"v1","addr":"10.20.193.117:6825","nonce":2181619050}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.117:6822","nonce":2181619050},{"type":"v1","addr":"10.20.193.117:6823","nonce":2181619050}]},"public_addr":"10.20.193.117:6819/2181619050","cluster_addr":"10.20.193.117:6821/2181619050","heartbeat_back_addr":"10.20.193.117:6825/2181619050","heartbeat_front_addr":"10.20.193.117:6823/2181619050","state":["exists","up"]},{"osd":4,"uuid":"8327850a-7ec7-4203-ae90-0a54b25d3a35","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":17,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.117:6843","nonce":2433940125},{"type":"v1","addr":"10.20.193.117:6845","nonce":2433940125}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.117:6847","nonce":2433940125},{"type":"v1","addr":"10.20.193.117:6849","nonce":2433940125}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.117:6855","nonce":2433940125},{"type":"v1","addr":"10.20.193.117:6857","nonce":2433940125}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.117:6851","nonce":2433940125},{"type":"v1","addr":"10.20.193.117:6853","nonce":2433940125}]},"public_addr":"10.20.193.117:6845/2433940125","cluster_addr":"10.20.193.117:6849/2433940125","heartbeat_back_addr":"10.20.193.117:6857/2433940125","heartbeat_front_addr":"10.20.193.117:6853/2433940125","state":["exists","up"]},{"osd":5,"uuid":"04975d9f-b1e5-465b-ae33-7a342cf7f976","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":17,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.117:6866","nonce":3722132383},{"type":"v1","addr":"10.20.193.117:6867","nonce":3722132383}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.117:6868","nonce":3722132383},{"type":"v1","addr":"10.20.193.117:6869","nonce":3722132383}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.117:6872","nonce":3722132383},{"type":"v1","addr":"10.20.193.117:6873","nonce":3722132383}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.117:6870","nonce":3722132383},{"type":"v1","addr":"10.20.193.117:6871","nonce":3722132383}]},"public_addr":"10.20.193.117:6867/3722132383","cluster_addr":"10.20.193.117:6869/3722132383","heartbeat_back_addr":"10.20.193.117:6873/3722132383","heartbeat_front_addr":"10.20.193.117:6871/3722132383","state":["exists","up"]},{"osd":6,"uuid":"6d6dfec7-2127-44bb-ad90-d3924e3d0902","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":17,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.117:6802","nonce":3926541475},{"type":"v1","addr":"10.20.193.117:6803","nonce":3926541475}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.117:6804","nonce":3926541475},{"type":"v1","addr":"10.20.193.117:6805","nonce":3926541475}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.117:6808","nonce":3926541475},{"type":"v1","addr":"10.20.193.117:6809","nonce":3926541475}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.117:6806","nonce":3926541475},{"type":"v1","addr":"10.20.193.117:6807","nonce":3926541475}]},"public_addr":"10.20.193.117:6803/3926541475","cluster_addr":"10.20.193.117:6805/3926541475","heartbeat_back_addr":"10.20.193.117:6809/3926541475","heartbeat_front_addr":"10.20.193.117:6807/3926541475","state":["exists","up"]},{"osd":7,"uuid":"617a797b-c363-4bde-a19e-ce36bedaba09","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.117:6842","nonce":1628495216},{"type":"v1","addr":"10.20.193.117:6844","nonce":1628495216}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.117:6846","nonce":1628495216},{"type":"v1","addr":"10.20.193.117:6848","nonce":1628495216}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.117:6854","nonce":1628495216},{"type":"v1","addr":"10.20.193.117:6856","nonce":1628495216}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.117:6850","nonce":1628495216},{"type":"v1","addr":"10.20.193.117:6852","nonce":1628495216}]},"public_addr":"10.20.193.117:6844/1628495216","cluster_addr":"10.20.193.117:6848/1628495216","heartbeat_back_addr":"10.20.193.117:6856/1628495216","heartbeat_front_addr":"10.20.193.117:6852/1628495216","state":["exists","up"]},{"osd":8,"uuid":"8a6d937f-608e-4cf8-9d91-fc6104b9ea99","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":17,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.117:6874","nonce":3404023726},{"type":"v1","addr":"10.20.193.117:6875","nonce":3404023726}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.117:6876","nonce":3404023726},{"type":"v1","addr":"10.20.193.117:6877","nonce":3404023726}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.117:6880","nonce":3404023726},{"type":"v1","addr":"10.20.193.117:6881","nonce":3404023726}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.117:6878","nonce":3404023726},{"type":"v1","addr":"10.20.193.117:6879","nonce":3404023726}]},"public_addr":"10.20.193.117:6875/3404023726","cluster_addr":"10.20.193.117:6877/3404023726","heartbeat_back_addr":"10.20.193.117:6881/3404023726","heartbeat_front_addr":"10.20.193.117:6879/3404023726","state":["exists","up"]},{"osd":9,"uuid":"5dba0073-60fa-4ce0-baab-a34df860f7b1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.117:6834","nonce":387956821},{"type":"v1","addr":"10.20.193.117:6835","nonce":387956821}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.117:6836","nonce":387956821},{"type":"v1","addr":"10.20.193.117:6837","nonce":387956821}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.117:6840","nonce":387956821},{"type":"v1","addr":"10.20.193.117:6841","nonce":387956821}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.117:6838","nonce":387956821},{"type":"v1","addr":"10.20.193.117:6839","nonce":387956821}]},"public_addr":"10.20.193.117:6835/387956821","cluster_addr":"10.20.193.117:6837/387956821","heartbeat_back_addr":"10.20.193.117:6841/387956821","heartbeat_front_addr":"10.20.193.117:6839/387956821","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":1,"snaps":[{"begin":2,"length":1}]}],"new_removed_snaps":[{"pool":1,"snaps":[{"begin":2,"length":1}]}],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-25T12:01:12.423 INFO:tasks.ceph.ceph_manager.ceph:all up! 2026-02-25T12:01:12.423 DEBUG:teuthology.orchestra.run.trial117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2026-02-25T12:01:12.588 INFO:teuthology.orchestra.run.trial117.stdout: 2026-02-25T12:01:12.588 INFO:teuthology.orchestra.run.trial117.stdout:{"epoch":20,"fsid":"43e63318-add1-4399-a1fc-80188594353b","created":"2026-02-25T12:00:57.865568+0000","modified":"2026-02-25T12:01:11.891867+0000","last_up_change":"2026-02-25T12:01:03.880119+0000","last_in_change":"2026-02-25T12:00:59.546090+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":10,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":false,"pools":[{"pool":1,"pool_name":"rbd","create_time":"2026-02-25T12:01:08.174903+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":2,"snap_epoch":20,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.5,"score_stable":2.5,"optimal_score":0.89999997615814209,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1.0000001192092896,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"a879bc90-f2aa-4692-98ac-d45e56987af0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":17,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.117:6810","nonce":2144854447},{"type":"v1","addr":"10.20.193.117:6811","nonce":2144854447}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.117:6812","nonce":2144854447},{"type":"v1","addr":"10.20.193.117:6813","nonce":2144854447}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.117:6816","nonce":2144854447},{"type":"v1","addr":"10.20.193.117:6817","nonce":2144854447}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.117:6814","nonce":2144854447},{"type":"v1","addr":"10.20.193.117:6815","nonce":2144854447}]},"public_addr":"10.20.193.117:6811/2144854447","cluster_addr":"10.20.193.117:6813/2144854447","heartbeat_back_addr":"10.20.193.117:6817/2144854447","heartbeat_front_addr":"10.20.193.117:6815/2144854447","state":["exists","up"]},{"osd":1,"uuid":"78845355-1113-42e3-9453-cbd686dd4cfb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.117:6826","nonce":491871890},{"type":"v1","addr":"10.20.193.117:6827","nonce":491871890}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.117:6828","nonce":491871890},{"type":"v1","addr":"10.20.193.117:6829","nonce":491871890}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.117:6832","nonce":491871890},{"type":"v1","addr":"10.20.193.117:6833","nonce":491871890}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.117:6830","nonce":491871890},{"type":"v1","addr":"10.20.193.117:6831","nonce":491871890}]},"public_addr":"10.20.193.117:6827/491871890","cluster_addr":"10.20.193.117:6829/491871890","heartbeat_back_addr":"10.20.193.117:6833/491871890","heartbeat_front_addr":"10.20.193.117:6831/491871890","state":["exists","up"]},{"osd":2,"uuid":"f96856b9-ffb4-4170-918a-dcb7374aebbb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.117:6858","nonce":3832000820},{"type":"v1","addr":"10.20.193.117:6859","nonce":3832000820}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.117:6860","nonce":3832000820},{"type":"v1","addr":"10.20.193.117:6861","nonce":3832000820}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.117:6864","nonce":3832000820},{"type":"v1","addr":"10.20.193.117:6865","nonce":3832000820}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.117:6862","nonce":3832000820},{"type":"v1","addr":"10.20.193.117:6863","nonce":3832000820}]},"public_addr":"10.20.193.117:6859/3832000820","cluster_addr":"10.20.193.117:6861/3832000820","heartbeat_back_addr":"10.20.193.117:6865/3832000820","heartbeat_front_addr":"10.20.193.117:6863/3832000820","state":["exists","up"]},{"osd":3,"uuid":"5d49c0b4-c47f-466c-bb15-52ef5e96fd02","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":17,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.117:6818","nonce":2181619050},{"type":"v1","addr":"10.20.193.117:6819","nonce":2181619050}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.117:6820","nonce":2181619050},{"type":"v1","addr":"10.20.193.117:6821","nonce":2181619050}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.117:6824","nonce":2181619050},{"type":"v1","addr":"10.20.193.117:6825","nonce":2181619050}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.117:6822","nonce":2181619050},{"type":"v1","addr":"10.20.193.117:6823","nonce":2181619050}]},"public_addr":"10.20.193.117:6819/2181619050","cluster_addr":"10.20.193.117:6821/2181619050","heartbeat_back_addr":"10.20.193.117:6825/2181619050","heartbeat_front_addr":"10.20.193.117:6823/2181619050","state":["exists","up"]},{"osd":4,"uuid":"8327850a-7ec7-4203-ae90-0a54b25d3a35","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":17,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.117:6843","nonce":2433940125},{"type":"v1","addr":"10.20.193.117:6845","nonce":2433940125}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.117:6847","nonce":2433940125},{"type":"v1","addr":"10.20.193.117:6849","nonce":2433940125}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.117:6855","nonce":2433940125},{"type":"v1","addr":"10.20.193.117:6857","nonce":2433940125}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.117:6851","nonce":2433940125},{"type":"v1","addr":"10.20.193.117:6853","nonce":2433940125}]},"public_addr":"10.20.193.117:6845/2433940125","cluster_addr":"10.20.193.117:6849/2433940125","heartbeat_back_addr":"10.20.193.117:6857/2433940125","heartbeat_front_addr":"10.20.193.117:6853/2433940125","state":["exists","up"]},{"osd":5,"uuid":"04975d9f-b1e5-465b-ae33-7a342cf7f976","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":17,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.117:6866","nonce":3722132383},{"type":"v1","addr":"10.20.193.117:6867","nonce":3722132383}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.117:6868","nonce":3722132383},{"type":"v1","addr":"10.20.193.117:6869","nonce":3722132383}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.117:6872","nonce":3722132383},{"type":"v1","addr":"10.20.193.117:6873","nonce":3722132383}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.117:6870","nonce":3722132383},{"type":"v1","addr":"10.20.193.117:6871","nonce":3722132383}]},"public_addr":"10.20.193.117:6867/3722132383","cluster_addr":"10.20.193.117:6869/3722132383","heartbeat_back_addr":"10.20.193.117:6873/3722132383","heartbeat_front_addr":"10.20.193.117:6871/3722132383","state":["exists","up"]},{"osd":6,"uuid":"6d6dfec7-2127-44bb-ad90-d3924e3d0902","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":17,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.117:6802","nonce":3926541475},{"type":"v1","addr":"10.20.193.117:6803","nonce":3926541475}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.117:6804","nonce":3926541475},{"type":"v1","addr":"10.20.193.117:6805","nonce":3926541475}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.117:6808","nonce":3926541475},{"type":"v1","addr":"10.20.193.117:6809","nonce":3926541475}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.117:6806","nonce":3926541475},{"type":"v1","addr":"10.20.193.117:6807","nonce":3926541475}]},"public_addr":"10.20.193.117:6803/3926541475","cluster_addr":"10.20.193.117:6805/3926541475","heartbeat_back_addr":"10.20.193.117:6809/3926541475","heartbeat_front_addr":"10.20.193.117:6807/3926541475","state":["exists","up"]},{"osd":7,"uuid":"617a797b-c363-4bde-a19e-ce36bedaba09","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.117:6842","nonce":1628495216},{"type":"v1","addr":"10.20.193.117:6844","nonce":1628495216}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.117:6846","nonce":1628495216},{"type":"v1","addr":"10.20.193.117:6848","nonce":1628495216}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.117:6854","nonce":1628495216},{"type":"v1","addr":"10.20.193.117:6856","nonce":1628495216}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.117:6850","nonce":1628495216},{"type":"v1","addr":"10.20.193.117:6852","nonce":1628495216}]},"public_addr":"10.20.193.117:6844/1628495216","cluster_addr":"10.20.193.117:6848/1628495216","heartbeat_back_addr":"10.20.193.117:6856/1628495216","heartbeat_front_addr":"10.20.193.117:6852/1628495216","state":["exists","up"]},{"osd":8,"uuid":"8a6d937f-608e-4cf8-9d91-fc6104b9ea99","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":17,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.117:6874","nonce":3404023726},{"type":"v1","addr":"10.20.193.117:6875","nonce":3404023726}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.117:6876","nonce":3404023726},{"type":"v1","addr":"10.20.193.117:6877","nonce":3404023726}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.117:6880","nonce":3404023726},{"type":"v1","addr":"10.20.193.117:6881","nonce":3404023726}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.117:6878","nonce":3404023726},{"type":"v1","addr":"10.20.193.117:6879","nonce":3404023726}]},"public_addr":"10.20.193.117:6875/3404023726","cluster_addr":"10.20.193.117:6877/3404023726","heartbeat_back_addr":"10.20.193.117:6881/3404023726","heartbeat_front_addr":"10.20.193.117:6879/3404023726","state":["exists","up"]},{"osd":9,"uuid":"5dba0073-60fa-4ce0-baab-a34df860f7b1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.117:6834","nonce":387956821},{"type":"v1","addr":"10.20.193.117:6835","nonce":387956821}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.117:6836","nonce":387956821},{"type":"v1","addr":"10.20.193.117:6837","nonce":387956821}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.117:6840","nonce":387956821},{"type":"v1","addr":"10.20.193.117:6841","nonce":387956821}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.117:6838","nonce":387956821},{"type":"v1","addr":"10.20.193.117:6839","nonce":387956821}]},"public_addr":"10.20.193.117:6835/387956821","cluster_addr":"10.20.193.117:6837/387956821","heartbeat_back_addr":"10.20.193.117:6841/387956821","heartbeat_front_addr":"10.20.193.117:6839/387956821","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":1,"snaps":[{"begin":2,"length":1}]}],"new_removed_snaps":[{"pool":1,"snaps":[{"begin":2,"length":1}]}],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-25T12:01:12.601 DEBUG:teuthology.orchestra.run.trial117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2026-02-25T12:01:12.601 DEBUG:teuthology.orchestra.run.trial117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2026-02-25T12:01:12.601 DEBUG:teuthology.orchestra.run.trial117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2026-02-25T12:01:12.601 DEBUG:teuthology.orchestra.run.trial117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 flush_pg_stats 2026-02-25T12:01:12.601 DEBUG:teuthology.orchestra.run.trial117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.4 flush_pg_stats 2026-02-25T12:01:12.602 DEBUG:teuthology.orchestra.run.trial117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.5 flush_pg_stats 2026-02-25T12:01:12.602 DEBUG:teuthology.orchestra.run.trial117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.6 flush_pg_stats 2026-02-25T12:01:12.602 DEBUG:teuthology.orchestra.run.trial117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.7 flush_pg_stats 2026-02-25T12:01:12.602 DEBUG:teuthology.orchestra.run.trial117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.8 flush_pg_stats 2026-02-25T12:01:12.602 DEBUG:teuthology.orchestra.run.trial117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.9 flush_pg_stats 2026-02-25T12:01:12.697 INFO:teuthology.orchestra.run.trial117.stdout:68719476739 2026-02-25T12:01:12.697 DEBUG:teuthology.orchestra.run.trial117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2026-02-25T12:01:12.697 INFO:teuthology.orchestra.run.trial117.stdout:64424509443 2026-02-25T12:01:12.698 DEBUG:teuthology.orchestra.run.trial117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2026-02-25T12:01:12.698 INFO:teuthology.orchestra.run.trial117.stdout:64424509444 2026-02-25T12:01:12.698 DEBUG:teuthology.orchestra.run.trial117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2026-02-25T12:01:12.699 INFO:teuthology.orchestra.run.trial117.stdout:64424509443 2026-02-25T12:01:12.699 DEBUG:teuthology.orchestra.run.trial117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.6 2026-02-25T12:01:12.699 INFO:teuthology.orchestra.run.trial117.stdout:64424509444 2026-02-25T12:01:12.700 DEBUG:teuthology.orchestra.run.trial117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.5 2026-02-25T12:01:12.700 INFO:teuthology.orchestra.run.trial117.stdout:68719476740 2026-02-25T12:01:12.700 DEBUG:teuthology.orchestra.run.trial117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.8 2026-02-25T12:01:12.702 INFO:teuthology.orchestra.run.trial117.stdout:68719476739 2026-02-25T12:01:12.703 DEBUG:teuthology.orchestra.run.trial117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.9 2026-02-25T12:01:12.704 INFO:teuthology.orchestra.run.trial117.stdout:64424509443 2026-02-25T12:01:12.704 INFO:teuthology.orchestra.run.trial117.stdout:68719476739 2026-02-25T12:01:12.704 DEBUG:teuthology.orchestra.run.trial117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.4 2026-02-25T12:01:12.705 DEBUG:teuthology.orchestra.run.trial117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.7 2026-02-25T12:01:12.705 INFO:teuthology.orchestra.run.trial117.stdout:64424509443 2026-02-25T12:01:12.705 DEBUG:teuthology.orchestra.run.trial117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2026-02-25T12:01:12.883 INFO:teuthology.orchestra.run.trial117.stdout:64424509443 2026-02-25T12:01:12.883 INFO:teuthology.orchestra.run.trial117.stdout:68719476739 2026-02-25T12:01:12.883 INFO:tasks.ceph.ceph_manager.ceph:need seq 64424509444 got 64424509443 for osd.5 2026-02-25T12:01:12.883 INFO:teuthology.orchestra.run.trial117.stdout:68719476738 2026-02-25T12:01:12.883 INFO:teuthology.orchestra.run.trial117.stdout:64424509442 2026-02-25T12:01:12.889 INFO:tasks.ceph.ceph_manager.ceph:need seq 68719476740 got 68719476739 for osd.8 2026-02-25T12:01:12.895 INFO:tasks.ceph.ceph_manager.ceph:need seq 68719476739 got 68719476738 for osd.2 2026-02-25T12:01:12.896 INFO:tasks.ceph.ceph_manager.ceph:need seq 64424509443 got 64424509442 for osd.1 2026-02-25T12:01:12.898 INFO:teuthology.orchestra.run.trial117.stdout:64424509444 2026-02-25T12:01:12.906 INFO:teuthology.orchestra.run.trial117.stdout:64424509443 2026-02-25T12:01:12.914 INFO:tasks.ceph.ceph_manager.ceph:need seq 64424509444 got 64424509444 for osd.3 2026-02-25T12:01:12.914 DEBUG:teuthology.parallel:result is None 2026-02-25T12:01:12.915 INFO:teuthology.orchestra.run.trial117.stdout:64424509443 2026-02-25T12:01:12.919 INFO:tasks.ceph.ceph_manager.ceph:need seq 64424509443 got 64424509443 for osd.6 2026-02-25T12:01:12.920 DEBUG:teuthology.parallel:result is None 2026-02-25T12:01:12.921 INFO:teuthology.orchestra.run.trial117.stdout:64424509443 2026-02-25T12:01:12.926 INFO:teuthology.orchestra.run.trial117.stdout:68719476739 2026-02-25T12:01:12.927 INFO:tasks.ceph.ceph_manager.ceph:need seq 64424509443 got 64424509443 for osd.4 2026-02-25T12:01:12.927 DEBUG:teuthology.parallel:result is None 2026-02-25T12:01:12.933 INFO:tasks.ceph.ceph_manager.ceph:need seq 64424509443 got 64424509443 for osd.0 2026-02-25T12:01:12.933 DEBUG:teuthology.parallel:result is None 2026-02-25T12:01:12.939 INFO:tasks.ceph.ceph_manager.ceph:need seq 68719476739 got 68719476739 for osd.9 2026-02-25T12:01:12.939 DEBUG:teuthology.parallel:result is None 2026-02-25T12:01:12.954 INFO:teuthology.orchestra.run.trial117.stdout:68719476739 2026-02-25T12:01:12.966 INFO:tasks.ceph.ceph_manager.ceph:need seq 68719476739 got 68719476739 for osd.7 2026-02-25T12:01:12.966 DEBUG:teuthology.parallel:result is None 2026-02-25T12:01:13.884 DEBUG:teuthology.orchestra.run.trial117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.5 2026-02-25T12:01:13.890 DEBUG:teuthology.orchestra.run.trial117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.8 2026-02-25T12:01:13.896 DEBUG:teuthology.orchestra.run.trial117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2026-02-25T12:01:13.896 DEBUG:teuthology.orchestra.run.trial117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2026-02-25T12:01:14.036 INFO:teuthology.orchestra.run.trial117.stdout:64424509444 2026-02-25T12:01:14.049 INFO:tasks.ceph.ceph_manager.ceph:need seq 64424509444 got 64424509444 for osd.5 2026-02-25T12:01:14.050 DEBUG:teuthology.parallel:result is None 2026-02-25T12:01:14.083 INFO:teuthology.orchestra.run.trial117.stdout:64424509443 2026-02-25T12:01:14.086 INFO:teuthology.orchestra.run.trial117.stdout:68719476740 2026-02-25T12:01:14.087 INFO:teuthology.orchestra.run.trial117.stdout:68719476739 2026-02-25T12:01:14.096 INFO:tasks.ceph.ceph_manager.ceph:need seq 64424509443 got 64424509443 for osd.1 2026-02-25T12:01:14.096 DEBUG:teuthology.parallel:result is None 2026-02-25T12:01:14.098 INFO:tasks.ceph.ceph_manager.ceph:need seq 68719476740 got 68719476740 for osd.8 2026-02-25T12:01:14.098 DEBUG:teuthology.parallel:result is None 2026-02-25T12:01:14.100 INFO:tasks.ceph.ceph_manager.ceph:need seq 68719476739 got 68719476739 for osd.2 2026-02-25T12:01:14.100 DEBUG:teuthology.parallel:result is None 2026-02-25T12:01:14.100 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2026-02-25T12:01:14.101 DEBUG:teuthology.orchestra.run.trial117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-25T12:01:14.292 INFO:teuthology.orchestra.run.trial117.stdout: 2026-02-25T12:01:14.293 INFO:teuthology.orchestra.run.trial117.stderr:dumped all 2026-02-25T12:01:14.304 INFO:teuthology.orchestra.run.trial117.stdout:{"pg_ready":true,"pg_map":{"version":24,"stamp":"2026-02-25T12:01:12.725837+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":7,"num_osds":10,"num_per_pool_osds":10,"num_per_pool_omap_osds":9,"kb":1048576000,"kb_used":271308,"kb_used_data":1708,"kb_used_omap":60,"kb_used_meta":269379,"kb_avail":1048304692,"statfs":{"total":1073741824000,"available":1073464004608,"internally_reserved":0,"allocated":1748992,"data_stored":553381,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":62355,"internal_metadata":275844205},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"0.000000"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":10,"reported_epoch":18,"state":"active+clean","last_fresh":"2026-02-25T12:01:09.975619+0000","last_change":"2026-02-25T12:01:09.975390+0000","last_active":"2026-02-25T12:01:09.975619+0000","last_peered":"2026-02-25T12:01:09.975619+0000","last_clean":"2026-02-25T12:01:09.975619+0000","last_became_active":"2026-02-25T12:01:09.975218+0000","last_became_peered":"2026-02-25T12:01:09.975218+0000","last_unstale":"2026-02-25T12:01:09.975619+0000","last_undegraded":"2026-02-25T12:01:09.975619+0000","last_fullsized":"2026-02-25T12:01:09.975619+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-25T12:01:08.885417+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-25T12:01:08.885417+0000","last_clean_scrub_stamp":"2026-02-25T12:01:08.885417+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-26T16:07:16.492530+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":10,"reported_epoch":18,"state":"active+clean","last_fresh":"2026-02-25T12:01:09.898921+0000","last_change":"2026-02-25T12:01:09.898736+0000","last_active":"2026-02-25T12:01:09.898921+0000","last_peered":"2026-02-25T12:01:09.898921+0000","last_clean":"2026-02-25T12:01:09.898921+0000","last_became_active":"2026-02-25T12:01:09.898592+0000","last_became_peered":"2026-02-25T12:01:09.898592+0000","last_unstale":"2026-02-25T12:01:09.898921+0000","last_undegraded":"2026-02-25T12:01:09.898921+0000","last_fullsized":"2026-02-25T12:01:09.898921+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-25T12:01:08.885417+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-25T12:01:08.885417+0000","last_clean_scrub_stamp":"2026-02-25T12:01:08.885417+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-26T14:15:49.630406+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,9],"acting":[6,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":10,"reported_epoch":18,"state":"active+clean","last_fresh":"2026-02-25T12:01:09.891781+0000","last_change":"2026-02-25T12:01:09.891515+0000","last_active":"2026-02-25T12:01:09.891781+0000","last_peered":"2026-02-25T12:01:09.891781+0000","last_clean":"2026-02-25T12:01:09.891781+0000","last_became_active":"2026-02-25T12:01:09.891350+0000","last_became_peered":"2026-02-25T12:01:09.891350+0000","last_unstale":"2026-02-25T12:01:09.891781+0000","last_undegraded":"2026-02-25T12:01:09.891781+0000","last_fullsized":"2026-02-25T12:01:09.891781+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-25T12:01:08.885417+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-25T12:01:08.885417+0000","last_clean_scrub_stamp":"2026-02-25T12:01:08.885417+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-26T21:01:21.519482+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8],"acting":[4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":13,"reported_epoch":19,"state":"active+clean","last_fresh":"2026-02-25T12:01:10.892585+0000","last_change":"2026-02-25T12:01:09.891594+0000","last_active":"2026-02-25T12:01:10.892585+0000","last_peered":"2026-02-25T12:01:10.892585+0000","last_clean":"2026-02-25T12:01:10.892585+0000","last_became_active":"2026-02-25T12:01:09.891407+0000","last_became_peered":"2026-02-25T12:01:09.891407+0000","last_unstale":"2026-02-25T12:01:10.892585+0000","last_undegraded":"2026-02-25T12:01:10.892585+0000","last_fullsized":"2026-02-25T12:01:10.892585+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-25T12:01:08.885417+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-25T12:01:08.885417+0000","last_clean_scrub_stamp":"2026-02-25T12:01:08.885417+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-26T22:33:52.805588+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"18'1","reported_seq":11,"reported_epoch":18,"state":"active+clean","last_fresh":"2026-02-25T12:01:09.966135+0000","last_change":"2026-02-25T12:01:09.965831+0000","last_active":"2026-02-25T12:01:09.966135+0000","last_peered":"2026-02-25T12:01:09.966135+0000","last_clean":"2026-02-25T12:01:09.966135+0000","last_became_active":"2026-02-25T12:01:09.965223+0000","last_became_peered":"2026-02-25T12:01:09.965223+0000","last_unstale":"2026-02-25T12:01:09.966135+0000","last_undegraded":"2026-02-25T12:01:09.966135+0000","last_fullsized":"2026-02-25T12:01:09.966135+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-25T12:01:08.885417+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-25T12:01:08.885417+0000","last_clean_scrub_stamp":"2026-02-25T12:01:08.885417+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-26T19:25:57.291465+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.2","version":"20'2","reported_seq":22,"reported_epoch":20,"state":"active+clean","last_fresh":"2026-02-25T12:01:11.896272+0000","last_change":"2026-02-25T12:01:11.895942+0000","last_active":"2026-02-25T12:01:11.896272+0000","last_peered":"2026-02-25T12:01:11.896272+0000","last_clean":"2026-02-25T12:01:11.896272+0000","last_became_active":"2026-02-25T12:01:09.975145+0000","last_became_peered":"2026-02-25T12:01:09.975145+0000","last_unstale":"2026-02-25T12:01:11.896272+0000","last_undegraded":"2026-02-25T12:01:11.896272+0000","last_fullsized":"2026-02-25T12:01:11.896272+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-25T12:01:08.885417+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-25T12:01:08.885417+0000","last_clean_scrub_stamp":"2026-02-25T12:01:08.885417+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-26T12:10:26.800456+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062137499999999997,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"1.1","version":"0'0","reported_seq":18,"reported_epoch":20,"state":"active+clean","last_fresh":"2026-02-25T12:01:11.894640+0000","last_change":"2026-02-25T12:01:11.894640+0000","last_active":"2026-02-25T12:01:11.894640+0000","last_peered":"2026-02-25T12:01:11.894640+0000","last_clean":"2026-02-25T12:01:11.894640+0000","last_became_active":"2026-02-25T12:01:09.891149+0000","last_became_peered":"2026-02-25T12:01:09.891149+0000","last_unstale":"2026-02-25T12:01:11.894640+0000","last_undegraded":"2026-02-25T12:01:11.894640+0000","last_fullsized":"2026-02-25T12:01:11.894640+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-25T12:01:08.885417+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-25T12:01:08.885417+0000","last_clean_scrub_stamp":"2026-02-25T12:01:08.885417+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-26T16:07:16.492530+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00015265299999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8],"acting":[5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"1.0","version":"0'0","reported_seq":18,"reported_epoch":20,"state":"active+clean","last_fresh":"2026-02-25T12:01:11.964129+0000","last_change":"2026-02-25T12:01:11.964129+0000","last_active":"2026-02-25T12:01:11.964129+0000","last_peered":"2026-02-25T12:01:11.964129+0000","last_clean":"2026-02-25T12:01:11.964129+0000","last_became_active":"2026-02-25T12:01:09.891951+0000","last_became_peered":"2026-02-25T12:01:09.891951+0000","last_unstale":"2026-02-25T12:01:11.964129+0000","last_undegraded":"2026-02-25T12:01:11.964129+0000","last_fullsized":"2026-02-25T12:01:11.964129+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-25T12:01:08.885417+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-25T12:01:08.885417+0000","last_clean_scrub_stamp":"2026-02-25T12:01:08.885417+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-26T22:33:52.805588+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00021116200000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0],"acting":[8,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[{"start":"2","length":"1"}]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":9}],"osd_stats":[{"osd":9,"up_from":16,"seq":68719476739,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":27124,"kb_used_data":164,"kb_used_omap":5,"kb_used_meta":26938,"kb_avail":104830476,"statfs":{"total":107374182400,"available":107346407424,"internally_reserved":0,"allocated":167936,"data_stored":52260,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":5524,"internal_metadata":27585132},"hb_peers":[0,1,2,3,4,5,6,7,8],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":16,"seq":68719476740,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":27124,"kb_used_data":164,"kb_used_omap":5,"kb_used_meta":26938,"kb_avail":104830476,"statfs":{"total":107374182400,"available":107346407424,"internally_reserved":0,"allocated":167936,"data_stored":52260,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":5519,"internal_metadata":27585137},"hb_peers":[0,1,2,3,4,5,6,7,9],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":16,"seq":68719476739,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":27152,"kb_used_data":192,"kb_used_omap":6,"kb_used_meta":26937,"kb_avail":104830448,"statfs":{"total":107374182400,"available":107346378752,"internally_reserved":0,"allocated":196608,"data_stored":64593,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6819,"internal_metadata":27583837},"hb_peers":[0,1,2,3,4,5,6,8,9],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":16,"seq":68719476739,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":104857600,"kb_used":27124,"kb_used_data":164,"kb_used_omap":5,"kb_used_meta":26938,"kb_avail":104830476,"statfs":{"total":107374182400,"available":107346407424,"internally_reserved":0,"allocated":167936,"data_stored":52260,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":5519,"internal_metadata":27585137},"hb_peers":[0,1,3,4,5,6,7,8,9],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":15,"seq":64424509443,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":27124,"kb_used_data":164,"kb_used_omap":6,"kb_used_meta":26937,"kb_avail":104830476,"statfs":{"total":107374182400,"available":107346407424,"internally_reserved":0,"allocated":167936,"data_stored":52260,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6166,"internal_metadata":27584490},"hb_peers":[0,1,2,3,4,5,7,8,9],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":15,"seq":64424509444,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":27124,"kb_used_data":164,"kb_used_omap":6,"kb_used_meta":26937,"kb_avail":104830476,"statfs":{"total":107374182400,"available":107346407424,"internally_reserved":0,"allocated":167936,"data_stored":52260,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6168,"internal_metadata":27584488},"hb_peers":[0,1,2,3,4,6,7,8,9],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":15,"seq":64424509443,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":27124,"kb_used_data":164,"kb_used_omap":6,"kb_used_meta":26937,"kb_avail":104830476,"statfs":{"total":107374182400,"available":107346407424,"internally_reserved":0,"allocated":167936,"data_stored":52260,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6175,"internal_metadata":27584481},"hb_peers":[0,1,2,3,5,6,7,8,9],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":15,"seq":64424509444,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":27136,"kb_used_data":176,"kb_used_omap":6,"kb_used_meta":26937,"kb_avail":104830464,"statfs":{"total":107374182400,"available":107346395136,"internally_reserved":0,"allocated":180224,"data_stored":58375,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6822,"internal_metadata":27583834},"hb_peers":[0,1,2,4,5,6,7,8,9],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":15,"seq":64424509443,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":27124,"kb_used_data":164,"kb_used_omap":6,"kb_used_meta":26937,"kb_avail":104830476,"statfs":{"total":107374182400,"available":107346407424,"internally_reserved":0,"allocated":167936,"data_stored":52260,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6168,"internal_metadata":27584488},"hb_peers":[0,2,3,4,5,6,7,8,9],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":15,"seq":64424509443,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":27152,"kb_used_data":192,"kb_used_omap":7,"kb_used_meta":26936,"kb_avail":104830448,"statfs":{"total":107374182400,"available":107346378752,"internally_reserved":0,"allocated":196608,"data_stored":64593,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7475,"internal_metadata":27583181},"hb_peers":[1,2,3,4,5,6,7,8,9],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-25T12:01:14.306 DEBUG:teuthology.orchestra.run.trial117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-25T12:01:14.472 INFO:teuthology.orchestra.run.trial117.stdout: 2026-02-25T12:01:14.472 INFO:teuthology.orchestra.run.trial117.stderr:dumped all 2026-02-25T12:01:14.482 INFO:teuthology.orchestra.run.trial117.stdout:{"pg_ready":true,"pg_map":{"version":24,"stamp":"2026-02-25T12:01:12.725837+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":7,"num_osds":10,"num_per_pool_osds":10,"num_per_pool_omap_osds":9,"kb":1048576000,"kb_used":271308,"kb_used_data":1708,"kb_used_omap":60,"kb_used_meta":269379,"kb_avail":1048304692,"statfs":{"total":1073741824000,"available":1073464004608,"internally_reserved":0,"allocated":1748992,"data_stored":553381,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":62355,"internal_metadata":275844205},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"0.000000"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":10,"reported_epoch":18,"state":"active+clean","last_fresh":"2026-02-25T12:01:09.975619+0000","last_change":"2026-02-25T12:01:09.975390+0000","last_active":"2026-02-25T12:01:09.975619+0000","last_peered":"2026-02-25T12:01:09.975619+0000","last_clean":"2026-02-25T12:01:09.975619+0000","last_became_active":"2026-02-25T12:01:09.975218+0000","last_became_peered":"2026-02-25T12:01:09.975218+0000","last_unstale":"2026-02-25T12:01:09.975619+0000","last_undegraded":"2026-02-25T12:01:09.975619+0000","last_fullsized":"2026-02-25T12:01:09.975619+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-25T12:01:08.885417+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-25T12:01:08.885417+0000","last_clean_scrub_stamp":"2026-02-25T12:01:08.885417+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-26T16:07:16.492530+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":10,"reported_epoch":18,"state":"active+clean","last_fresh":"2026-02-25T12:01:09.898921+0000","last_change":"2026-02-25T12:01:09.898736+0000","last_active":"2026-02-25T12:01:09.898921+0000","last_peered":"2026-02-25T12:01:09.898921+0000","last_clean":"2026-02-25T12:01:09.898921+0000","last_became_active":"2026-02-25T12:01:09.898592+0000","last_became_peered":"2026-02-25T12:01:09.898592+0000","last_unstale":"2026-02-25T12:01:09.898921+0000","last_undegraded":"2026-02-25T12:01:09.898921+0000","last_fullsized":"2026-02-25T12:01:09.898921+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-25T12:01:08.885417+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-25T12:01:08.885417+0000","last_clean_scrub_stamp":"2026-02-25T12:01:08.885417+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-26T14:15:49.630406+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,9],"acting":[6,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":10,"reported_epoch":18,"state":"active+clean","last_fresh":"2026-02-25T12:01:09.891781+0000","last_change":"2026-02-25T12:01:09.891515+0000","last_active":"2026-02-25T12:01:09.891781+0000","last_peered":"2026-02-25T12:01:09.891781+0000","last_clean":"2026-02-25T12:01:09.891781+0000","last_became_active":"2026-02-25T12:01:09.891350+0000","last_became_peered":"2026-02-25T12:01:09.891350+0000","last_unstale":"2026-02-25T12:01:09.891781+0000","last_undegraded":"2026-02-25T12:01:09.891781+0000","last_fullsized":"2026-02-25T12:01:09.891781+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-25T12:01:08.885417+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-25T12:01:08.885417+0000","last_clean_scrub_stamp":"2026-02-25T12:01:08.885417+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-26T21:01:21.519482+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8],"acting":[4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":13,"reported_epoch":19,"state":"active+clean","last_fresh":"2026-02-25T12:01:10.892585+0000","last_change":"2026-02-25T12:01:09.891594+0000","last_active":"2026-02-25T12:01:10.892585+0000","last_peered":"2026-02-25T12:01:10.892585+0000","last_clean":"2026-02-25T12:01:10.892585+0000","last_became_active":"2026-02-25T12:01:09.891407+0000","last_became_peered":"2026-02-25T12:01:09.891407+0000","last_unstale":"2026-02-25T12:01:10.892585+0000","last_undegraded":"2026-02-25T12:01:10.892585+0000","last_fullsized":"2026-02-25T12:01:10.892585+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-25T12:01:08.885417+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-25T12:01:08.885417+0000","last_clean_scrub_stamp":"2026-02-25T12:01:08.885417+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-26T22:33:52.805588+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"18'1","reported_seq":11,"reported_epoch":18,"state":"active+clean","last_fresh":"2026-02-25T12:01:09.966135+0000","last_change":"2026-02-25T12:01:09.965831+0000","last_active":"2026-02-25T12:01:09.966135+0000","last_peered":"2026-02-25T12:01:09.966135+0000","last_clean":"2026-02-25T12:01:09.966135+0000","last_became_active":"2026-02-25T12:01:09.965223+0000","last_became_peered":"2026-02-25T12:01:09.965223+0000","last_unstale":"2026-02-25T12:01:09.966135+0000","last_undegraded":"2026-02-25T12:01:09.966135+0000","last_fullsized":"2026-02-25T12:01:09.966135+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-25T12:01:08.885417+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-25T12:01:08.885417+0000","last_clean_scrub_stamp":"2026-02-25T12:01:08.885417+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-26T19:25:57.291465+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.2","version":"20'2","reported_seq":22,"reported_epoch":20,"state":"active+clean","last_fresh":"2026-02-25T12:01:11.896272+0000","last_change":"2026-02-25T12:01:11.895942+0000","last_active":"2026-02-25T12:01:11.896272+0000","last_peered":"2026-02-25T12:01:11.896272+0000","last_clean":"2026-02-25T12:01:11.896272+0000","last_became_active":"2026-02-25T12:01:09.975145+0000","last_became_peered":"2026-02-25T12:01:09.975145+0000","last_unstale":"2026-02-25T12:01:11.896272+0000","last_undegraded":"2026-02-25T12:01:11.896272+0000","last_fullsized":"2026-02-25T12:01:11.896272+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-25T12:01:08.885417+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-25T12:01:08.885417+0000","last_clean_scrub_stamp":"2026-02-25T12:01:08.885417+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-26T12:10:26.800456+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062137499999999997,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"1.1","version":"0'0","reported_seq":18,"reported_epoch":20,"state":"active+clean","last_fresh":"2026-02-25T12:01:11.894640+0000","last_change":"2026-02-25T12:01:11.894640+0000","last_active":"2026-02-25T12:01:11.894640+0000","last_peered":"2026-02-25T12:01:11.894640+0000","last_clean":"2026-02-25T12:01:11.894640+0000","last_became_active":"2026-02-25T12:01:09.891149+0000","last_became_peered":"2026-02-25T12:01:09.891149+0000","last_unstale":"2026-02-25T12:01:11.894640+0000","last_undegraded":"2026-02-25T12:01:11.894640+0000","last_fullsized":"2026-02-25T12:01:11.894640+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-25T12:01:08.885417+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-25T12:01:08.885417+0000","last_clean_scrub_stamp":"2026-02-25T12:01:08.885417+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-26T16:07:16.492530+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00015265299999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8],"acting":[5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"1.0","version":"0'0","reported_seq":18,"reported_epoch":20,"state":"active+clean","last_fresh":"2026-02-25T12:01:11.964129+0000","last_change":"2026-02-25T12:01:11.964129+0000","last_active":"2026-02-25T12:01:11.964129+0000","last_peered":"2026-02-25T12:01:11.964129+0000","last_clean":"2026-02-25T12:01:11.964129+0000","last_became_active":"2026-02-25T12:01:09.891951+0000","last_became_peered":"2026-02-25T12:01:09.891951+0000","last_unstale":"2026-02-25T12:01:11.964129+0000","last_undegraded":"2026-02-25T12:01:11.964129+0000","last_fullsized":"2026-02-25T12:01:11.964129+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-25T12:01:08.885417+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-25T12:01:08.885417+0000","last_clean_scrub_stamp":"2026-02-25T12:01:08.885417+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-26T22:33:52.805588+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00021116200000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0],"acting":[8,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[{"start":"2","length":"1"}]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":9}],"osd_stats":[{"osd":9,"up_from":16,"seq":68719476739,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":27124,"kb_used_data":164,"kb_used_omap":5,"kb_used_meta":26938,"kb_avail":104830476,"statfs":{"total":107374182400,"available":107346407424,"internally_reserved":0,"allocated":167936,"data_stored":52260,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":5524,"internal_metadata":27585132},"hb_peers":[0,1,2,3,4,5,6,7,8],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":16,"seq":68719476740,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":27124,"kb_used_data":164,"kb_used_omap":5,"kb_used_meta":26938,"kb_avail":104830476,"statfs":{"total":107374182400,"available":107346407424,"internally_reserved":0,"allocated":167936,"data_stored":52260,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":5519,"internal_metadata":27585137},"hb_peers":[0,1,2,3,4,5,6,7,9],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":16,"seq":68719476739,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":27152,"kb_used_data":192,"kb_used_omap":6,"kb_used_meta":26937,"kb_avail":104830448,"statfs":{"total":107374182400,"available":107346378752,"internally_reserved":0,"allocated":196608,"data_stored":64593,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6819,"internal_metadata":27583837},"hb_peers":[0,1,2,3,4,5,6,8,9],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":16,"seq":68719476739,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":104857600,"kb_used":27124,"kb_used_data":164,"kb_used_omap":5,"kb_used_meta":26938,"kb_avail":104830476,"statfs":{"total":107374182400,"available":107346407424,"internally_reserved":0,"allocated":167936,"data_stored":52260,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":5519,"internal_metadata":27585137},"hb_peers":[0,1,3,4,5,6,7,8,9],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":15,"seq":64424509443,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":27124,"kb_used_data":164,"kb_used_omap":6,"kb_used_meta":26937,"kb_avail":104830476,"statfs":{"total":107374182400,"available":107346407424,"internally_reserved":0,"allocated":167936,"data_stored":52260,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6166,"internal_metadata":27584490},"hb_peers":[0,1,2,3,4,5,7,8,9],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":15,"seq":64424509444,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":27124,"kb_used_data":164,"kb_used_omap":6,"kb_used_meta":26937,"kb_avail":104830476,"statfs":{"total":107374182400,"available":107346407424,"internally_reserved":0,"allocated":167936,"data_stored":52260,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6168,"internal_metadata":27584488},"hb_peers":[0,1,2,3,4,6,7,8,9],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":15,"seq":64424509443,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":27124,"kb_used_data":164,"kb_used_omap":6,"kb_used_meta":26937,"kb_avail":104830476,"statfs":{"total":107374182400,"available":107346407424,"internally_reserved":0,"allocated":167936,"data_stored":52260,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6175,"internal_metadata":27584481},"hb_peers":[0,1,2,3,5,6,7,8,9],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":15,"seq":64424509444,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":27136,"kb_used_data":176,"kb_used_omap":6,"kb_used_meta":26937,"kb_avail":104830464,"statfs":{"total":107374182400,"available":107346395136,"internally_reserved":0,"allocated":180224,"data_stored":58375,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6822,"internal_metadata":27583834},"hb_peers":[0,1,2,4,5,6,7,8,9],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":15,"seq":64424509443,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":27124,"kb_used_data":164,"kb_used_omap":6,"kb_used_meta":26937,"kb_avail":104830476,"statfs":{"total":107374182400,"available":107346407424,"internally_reserved":0,"allocated":167936,"data_stored":52260,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6168,"internal_metadata":27584488},"hb_peers":[0,2,3,4,5,6,7,8,9],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":15,"seq":64424509443,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":27152,"kb_used_data":192,"kb_used_omap":7,"kb_used_meta":26936,"kb_avail":104830448,"statfs":{"total":107374182400,"available":107346378752,"internally_reserved":0,"allocated":196608,"data_stored":64593,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7475,"internal_metadata":27583181},"hb_peers":[1,2,3,4,5,6,7,8,9],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-25T12:01:14.484 INFO:tasks.ceph.ceph_manager.ceph:clean! 2026-02-25T12:01:14.484 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2026-02-25T12:01:14.484 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy 2026-02-25T12:01:14.484 DEBUG:teuthology.orchestra.run.trial117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2026-02-25T12:01:14.652 INFO:teuthology.orchestra.run.trial117.stdout: 2026-02-25T12:01:14.653 INFO:teuthology.orchestra.run.trial117.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2026-02-25T12:01:14.665 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy done 2026-02-25T12:01:14.666 INFO:teuthology.run_tasks:Running task workunit... 2026-02-25T12:01:14.691 INFO:tasks.workunit:Pulling workunits from ref 506b402a3d3a240d0bfe073b4769eaffdf84497b 2026-02-25T12:01:14.692 INFO:tasks.workunit:Making a separate scratch dir for every client... 2026-02-25T12:01:14.692 INFO:tasks.workunit:timeout=3h 2026-02-25T12:01:14.692 INFO:tasks.workunit:cleanup=True 2026-02-25T12:01:14.692 DEBUG:teuthology.orchestra.run.trial117:> stat -- /home/ubuntu/cephtest/mnt.0 2026-02-25T12:01:14.697 DEBUG:teuthology.orchestra.run:got remote process result: 1 2026-02-25T12:01:14.697 INFO:teuthology.orchestra.run.trial117.stderr:stat: cannot statx '/home/ubuntu/cephtest/mnt.0': No such file or directory 2026-02-25T12:01:14.698 DEBUG:teuthology.orchestra.run.trial117:> mkdir -- /home/ubuntu/cephtest/mnt.0 2026-02-25T12:01:14.745 INFO:tasks.workunit:Created dir /home/ubuntu/cephtest/mnt.0 2026-02-25T12:01:14.745 DEBUG:teuthology.orchestra.run.trial117:> cd -- /home/ubuntu/cephtest/mnt.0 && mkdir -- client.0 2026-02-25T12:01:14.791 DEBUG:teuthology.orchestra.run.trial117:> rm -rf /home/ubuntu/cephtest/clone.client.0 && git clone https://git.ceph.com/ceph-ci.git /home/ubuntu/cephtest/clone.client.0 && cd /home/ubuntu/cephtest/clone.client.0 && git checkout 506b402a3d3a240d0bfe073b4769eaffdf84497b 2026-02-25T12:01:14.843 INFO:tasks.workunit.client.0.trial117.stderr:Cloning into '/home/ubuntu/cephtest/clone.client.0'... 2026-02-25T12:01:28.763 INFO:tasks.ceph.mgr.x.trial117.stderr:INFO:mgr_util:scanning for idle connections.. 2026-02-25T12:01:28.763 INFO:tasks.ceph.mgr.x.trial117.stderr:INFO:mgr_util:cleaning up connections: [] 2026-02-25T12:01:28.766 INFO:tasks.ceph.mgr.x.trial117.stderr:INFO:mgr_util:scanning for idle connections.. 2026-02-25T12:01:28.766 INFO:tasks.ceph.mgr.x.trial117.stderr:INFO:mgr_util:cleaning up connections: [] 2026-02-25T12:01:28.767 INFO:tasks.ceph.mgr.x.trial117.stderr:INFO:mgr_util:scanning for idle connections.. 2026-02-25T12:01:28.767 INFO:tasks.ceph.mgr.x.trial117.stderr:INFO:mgr_util:cleaning up connections: [] 2026-02-25T12:01:44.257 INFO:tasks.workunit.client.0.trial117.stderr:Note: switching to '506b402a3d3a240d0bfe073b4769eaffdf84497b'. 2026-02-25T12:01:44.257 INFO:tasks.workunit.client.0.trial117.stderr: 2026-02-25T12:01:44.257 INFO:tasks.workunit.client.0.trial117.stderr:You are in 'detached HEAD' state. You can look around, make experimental 2026-02-25T12:01:44.257 INFO:tasks.workunit.client.0.trial117.stderr:changes and commit them, and you can discard any commits you make in this 2026-02-25T12:01:44.257 INFO:tasks.workunit.client.0.trial117.stderr:state without impacting any branches by switching back to a branch. 2026-02-25T12:01:44.257 INFO:tasks.workunit.client.0.trial117.stderr: 2026-02-25T12:01:44.257 INFO:tasks.workunit.client.0.trial117.stderr:If you want to create a new branch to retain commits you create, you may 2026-02-25T12:01:44.257 INFO:tasks.workunit.client.0.trial117.stderr:do so (now or later) by using -c with the switch command. Example: 2026-02-25T12:01:44.257 INFO:tasks.workunit.client.0.trial117.stderr: 2026-02-25T12:01:44.257 INFO:tasks.workunit.client.0.trial117.stderr: git switch -c 2026-02-25T12:01:44.257 INFO:tasks.workunit.client.0.trial117.stderr: 2026-02-25T12:01:44.257 INFO:tasks.workunit.client.0.trial117.stderr:Or undo this operation with: 2026-02-25T12:01:44.257 INFO:tasks.workunit.client.0.trial117.stderr: 2026-02-25T12:01:44.257 INFO:tasks.workunit.client.0.trial117.stderr: git switch - 2026-02-25T12:01:44.257 INFO:tasks.workunit.client.0.trial117.stderr: 2026-02-25T12:01:44.257 INFO:tasks.workunit.client.0.trial117.stderr:Turn off this advice by setting config variable advice.detachedHead to false 2026-02-25T12:01:44.257 INFO:tasks.workunit.client.0.trial117.stderr: 2026-02-25T12:01:44.257 INFO:tasks.workunit.client.0.trial117.stderr:HEAD is now at 506b402a3d3 Integration branch metadata 2026-02-25T12:01:44.267 DEBUG:teuthology.orchestra.run.trial117:> cd -- /home/ubuntu/cephtest/clone.client.0/qa/workunits && if test -e Makefile ; then make ; fi && find -executable -type f -printf '%P\0' >/home/ubuntu/cephtest/workunits.list.client.0 2026-02-25T12:01:44.313 INFO:tasks.workunit.client.0.trial117.stdout:for d in direct_io fs ; do ( cd $d ; make all ) ; done 2026-02-25T12:01:44.314 INFO:tasks.workunit.client.0.trial117.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2026-02-25T12:01:44.314 INFO:tasks.workunit.client.0.trial117.stdout:cc -Wall -Wextra -D_GNU_SOURCE direct_io_test.c -o direct_io_test 2026-02-25T12:01:44.413 INFO:tasks.workunit.client.0.trial117.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_sync_io.c -o test_sync_io 2026-02-25T12:01:44.445 INFO:tasks.workunit.client.0.trial117.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_short_dio_read.c -o test_short_dio_read 2026-02-25T12:01:44.471 INFO:tasks.workunit.client.0.trial117.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2026-02-25T12:01:44.473 INFO:tasks.workunit.client.0.trial117.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2026-02-25T12:01:44.473 INFO:tasks.workunit.client.0.trial117.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_o_trunc.c -o test_o_trunc 2026-02-25T12:01:44.498 INFO:tasks.workunit.client.0.trial117.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2026-02-25T12:01:44.502 DEBUG:teuthology.orchestra.run.trial117:> set -ex 2026-02-25T12:01:44.502 DEBUG:teuthology.orchestra.run.trial117:> dd if=/home/ubuntu/cephtest/workunits.list.client.0 of=/dev/stdout 2026-02-25T12:01:44.548 INFO:tasks.workunit:Running workunits matching rados/test_health_warnings.sh on client.0... 2026-02-25T12:01:44.550 INFO:tasks.workunit:Running workunit rados/test_health_warnings.sh... 2026-02-25T12:01:44.550 DEBUG:teuthology.orchestra.run.trial117:workunit test rados/test_health_warnings.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=506b402a3d3a240d0bfe073b4769eaffdf84497b TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/rados/test_health_warnings.sh 2026-02-25T12:01:44.600 INFO:tasks.workunit.client.0.trial117.stderr:+ crushtool -o crushmap --build --num_osds 10 host straw 2 rack straw 2 row straw 2 root straw 0 2026-02-25T12:01:44.609 INFO:tasks.workunit.client.0.trial117.stderr:+ ceph osd setcrushmap -i crushmap 2026-02-25T12:01:45.835 INFO:tasks.workunit.client.0.trial117.stderr:7 2026-02-25T12:01:45.849 INFO:tasks.workunit.client.0.trial117.stderr:+ ceph osd tree 2026-02-25T12:01:46.050 INFO:tasks.workunit.client.0.trial117.stdout:ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF 2026-02-25T12:01:46.050 INFO:tasks.workunit.client.0.trial117.stdout:-11 10.00000 root root 2026-02-25T12:01:46.050 INFO:tasks.workunit.client.0.trial117.stdout: -9 8.00000 row row0 2026-02-25T12:01:46.050 INFO:tasks.workunit.client.0.trial117.stdout: -6 4.00000 rack rack0 2026-02-25T12:01:46.050 INFO:tasks.workunit.client.0.trial117.stdout: -1 2.00000 host host0 2026-02-25T12:01:46.050 INFO:tasks.workunit.client.0.trial117.stdout: 0 1.00000 osd.0 up 1.00000 1.00000 2026-02-25T12:01:46.050 INFO:tasks.workunit.client.0.trial117.stdout: 1 1.00000 osd.1 up 1.00000 1.00000 2026-02-25T12:01:46.050 INFO:tasks.workunit.client.0.trial117.stdout: -2 2.00000 host host1 2026-02-25T12:01:46.050 INFO:tasks.workunit.client.0.trial117.stdout: 2 1.00000 osd.2 up 1.00000 1.00000 2026-02-25T12:01:46.051 INFO:tasks.workunit.client.0.trial117.stdout: 3 1.00000 osd.3 up 1.00000 1.00000 2026-02-25T12:01:46.051 INFO:tasks.workunit.client.0.trial117.stdout: -7 4.00000 rack rack1 2026-02-25T12:01:46.051 INFO:tasks.workunit.client.0.trial117.stdout: -3 2.00000 host host2 2026-02-25T12:01:46.051 INFO:tasks.workunit.client.0.trial117.stdout: 4 1.00000 osd.4 up 1.00000 1.00000 2026-02-25T12:01:46.051 INFO:tasks.workunit.client.0.trial117.stdout: 5 1.00000 osd.5 up 1.00000 1.00000 2026-02-25T12:01:46.051 INFO:tasks.workunit.client.0.trial117.stdout: -4 2.00000 host host3 2026-02-25T12:01:46.051 INFO:tasks.workunit.client.0.trial117.stdout: 6 1.00000 osd.6 up 1.00000 1.00000 2026-02-25T12:01:46.051 INFO:tasks.workunit.client.0.trial117.stdout: 7 1.00000 osd.7 up 1.00000 1.00000 2026-02-25T12:01:46.051 INFO:tasks.workunit.client.0.trial117.stdout:-10 2.00000 row row1 2026-02-25T12:01:46.051 INFO:tasks.workunit.client.0.trial117.stdout: -8 2.00000 rack rack2 2026-02-25T12:01:46.051 INFO:tasks.workunit.client.0.trial117.stdout: -5 2.00000 host host4 2026-02-25T12:01:46.051 INFO:tasks.workunit.client.0.trial117.stdout: 8 1.00000 osd.8 up 1.00000 1.00000 2026-02-25T12:01:46.052 INFO:tasks.workunit.client.0.trial117.stdout: 9 1.00000 osd.9 up 1.00000 1.00000 2026-02-25T12:01:46.059 INFO:tasks.workunit.client.0.trial117.stderr:+ ceph tell 'osd.*' injectargs --osd_max_markdown_count 1024 --osd_max_markdown_period 1 2026-02-25T12:01:46.150 INFO:tasks.workunit.client.0.trial117.stdout:osd.0: {} 2026-02-25T12:01:46.167 INFO:tasks.workunit.client.0.trial117.stdout:osd.1: {} 2026-02-25T12:01:46.184 INFO:tasks.workunit.client.0.trial117.stdout:osd.2: {} 2026-02-25T12:01:46.201 INFO:tasks.workunit.client.0.trial117.stdout:osd.3: {} 2026-02-25T12:01:46.218 INFO:tasks.workunit.client.0.trial117.stdout:osd.4: {} 2026-02-25T12:01:46.235 INFO:tasks.workunit.client.0.trial117.stdout:osd.5: {} 2026-02-25T12:01:46.252 INFO:tasks.workunit.client.0.trial117.stdout:osd.6: {} 2026-02-25T12:01:46.269 INFO:tasks.workunit.client.0.trial117.stdout:osd.7: {} 2026-02-25T12:01:46.287 INFO:tasks.workunit.client.0.trial117.stdout:osd.8: {} 2026-02-25T12:01:46.304 INFO:tasks.workunit.client.0.trial117.stdout:osd.9: {} 2026-02-25T12:01:46.312 INFO:tasks.workunit.client.0.trial117.stderr:+ ceph osd set noout 2026-02-25T12:01:47.838 INFO:tasks.workunit.client.0.trial117.stderr:noout is set 2026-02-25T12:01:47.853 INFO:tasks.workunit.client.0.trial117.stderr:+ test_mark_two_osds_same_host_down 2026-02-25T12:01:47.853 INFO:tasks.workunit.client.0.trial117.stderr:+ ceph osd set noup 2026-02-25T12:01:49.842 INFO:tasks.workunit.client.0.trial117.stderr:noup is set 2026-02-25T12:01:49.855 INFO:tasks.workunit.client.0.trial117.stderr:+ ceph osd down osd.0 osd.1 2026-02-25T12:01:50.009 INFO:tasks.ceph.mon.a.trial117.stderr:2026-02-25T12:01:50.006+0000 7f9a477c9640 -1 mon.a@0(leader).osd e27 definitely_dead 0 2026-02-25T12:01:50.897 INFO:tasks.ceph.mon.a.trial117.stderr:2026-02-25T12:01:50.894+0000 7f9a477c9640 -1 mon.a@0(leader).osd e28 definitely_dead 0 2026-02-25T12:01:50.898 INFO:tasks.workunit.client.0.trial117.stderr:osd.0 is already down. osd.1 is already down. 2026-02-25T12:01:50.909 INFO:tasks.workunit.client.0.trial117.stderr:+ ceph health detail 2026-02-25T12:01:51.167 INFO:tasks.workunit.client.0.trial117.stdout:HEALTH_WARN noup,noout flag(s) set; 2 osds down; 1 host (2 osds) down 2026-02-25T12:01:51.167 INFO:tasks.workunit.client.0.trial117.stdout:[WRN] OSDMAP_FLAGS: noup,noout flag(s) set 2026-02-25T12:01:51.167 INFO:tasks.workunit.client.0.trial117.stdout:[WRN] OSD_DOWN: 2 osds down 2026-02-25T12:01:51.168 INFO:tasks.workunit.client.0.trial117.stdout: osd.0 (root=root,row=row0,rack=rack0,host=host0) is down 2026-02-25T12:01:51.168 INFO:tasks.workunit.client.0.trial117.stdout: osd.1 (root=root,row=row0,rack=rack0,host=host0) is down 2026-02-25T12:01:51.168 INFO:tasks.workunit.client.0.trial117.stdout:[WRN] OSD_HOST_DOWN: 1 host (2 osds) down 2026-02-25T12:01:51.168 INFO:tasks.workunit.client.0.trial117.stdout: host host0 (root=root,row=row0,rack=rack0) (2 osds) is down 2026-02-25T12:01:51.178 INFO:tasks.workunit.client.0.trial117.stderr:+ ceph health 2026-02-25T12:01:51.178 INFO:tasks.workunit.client.0.trial117.stderr:+ grep '1 host' 2026-02-25T12:01:51.431 INFO:tasks.workunit.client.0.trial117.stdout:HEALTH_WARN noup,noout flag(s) set; 2 osds down; 1 host (2 osds) down 2026-02-25T12:01:51.432 INFO:tasks.workunit.client.0.trial117.stderr:+ ceph health 2026-02-25T12:01:51.432 INFO:tasks.workunit.client.0.trial117.stderr:+ grep '2 osds' 2026-02-25T12:01:51.678 INFO:tasks.workunit.client.0.trial117.stdout:HEALTH_WARN noup,noout flag(s) set; 2 osds down; 1 host (2 osds) down 2026-02-25T12:01:51.679 INFO:tasks.workunit.client.0.trial117.stderr:+ ceph health detail 2026-02-25T12:01:51.679 INFO:tasks.workunit.client.0.trial117.stderr:+ grep osd.0 2026-02-25T12:01:51.945 INFO:tasks.workunit.client.0.trial117.stdout: osd.0 (root=root,row=row0,rack=rack0,host=host0) is down 2026-02-25T12:01:51.945 INFO:tasks.workunit.client.0.trial117.stderr:+ ceph health detail 2026-02-25T12:01:51.945 INFO:tasks.workunit.client.0.trial117.stderr:+ grep osd.1 2026-02-25T12:01:52.211 INFO:tasks.workunit.client.0.trial117.stdout: osd.1 (root=root,row=row0,rack=rack0,host=host0) is down 2026-02-25T12:01:52.211 INFO:tasks.workunit.client.0.trial117.stderr:+ ceph osd unset noup 2026-02-25T12:01:52.941 INFO:tasks.ceph.osd.1.trial117.stderr:2026-02-25T12:01:52.938+0000 7fbcc414f640 -1 osd.1 30 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2026-02-25T12:01:53.056 INFO:tasks.ceph.osd.0.trial117.stderr:2026-02-25T12:01:53.050+0000 7f8a11442640 -1 osd.0 30 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2026-02-25T12:01:53.853 INFO:tasks.workunit.client.0.trial117.stderr:noup is unset 2026-02-25T12:01:53.865 INFO:tasks.workunit.client.0.trial117.stderr:+ wait_for_healthy 2026-02-25T12:01:53.865 INFO:tasks.workunit.client.0.trial117.stderr:+ ceph health 2026-02-25T12:01:53.865 INFO:tasks.workunit.client.0.trial117.stderr:+ grep down 2026-02-25T12:01:54.108 INFO:tasks.workunit.client.0.trial117.stderr:+ test_mark_two_osds_same_rack_down 2026-02-25T12:01:54.108 INFO:tasks.workunit.client.0.trial117.stderr:+ ceph osd set noup 2026-02-25T12:01:55.855 INFO:tasks.workunit.client.0.trial117.stderr:noup is set 2026-02-25T12:01:55.869 INFO:tasks.workunit.client.0.trial117.stderr:+ ceph osd down osd.8 osd.9 2026-02-25T12:01:56.020 INFO:tasks.ceph.mon.a.trial117.stderr:2026-02-25T12:01:56.014+0000 7f9a477c9640 -1 mon.a@0(leader).osd e33 definitely_dead 0 2026-02-25T12:01:56.910 INFO:tasks.ceph.mon.a.trial117.stderr:2026-02-25T12:01:56.906+0000 7f9a477c9640 -1 mon.a@0(leader).osd e34 definitely_dead 0 2026-02-25T12:01:56.910 INFO:tasks.workunit.client.0.trial117.stderr:osd.8 is already down. osd.9 is already down. 2026-02-25T12:01:56.921 INFO:tasks.workunit.client.0.trial117.stderr:+ ceph health detail 2026-02-25T12:01:57.183 INFO:tasks.workunit.client.0.trial117.stdout:HEALTH_WARN noup,noout flag(s) set; 2 osds down; 1 host (2 osds) down; 1 rack (2 osds) down; 1 row (2 osds) down; Degraded data redundancy: 2/4 objects degraded (50.000%), 2 pgs degraded 2026-02-25T12:01:57.183 INFO:tasks.workunit.client.0.trial117.stdout:[WRN] OSDMAP_FLAGS: noup,noout flag(s) set 2026-02-25T12:01:57.183 INFO:tasks.workunit.client.0.trial117.stdout:[WRN] OSD_DOWN: 2 osds down 2026-02-25T12:01:57.183 INFO:tasks.workunit.client.0.trial117.stdout: osd.8 (root=root,row=row1,rack=rack2,host=host4) is down 2026-02-25T12:01:57.183 INFO:tasks.workunit.client.0.trial117.stdout: osd.9 (root=root,row=row1,rack=rack2,host=host4) is down 2026-02-25T12:01:57.183 INFO:tasks.workunit.client.0.trial117.stdout:[WRN] OSD_HOST_DOWN: 1 host (2 osds) down 2026-02-25T12:01:57.183 INFO:tasks.workunit.client.0.trial117.stdout: host host4 (root=root,row=row1,rack=rack2) (2 osds) is down 2026-02-25T12:01:57.183 INFO:tasks.workunit.client.0.trial117.stdout:[WRN] OSD_RACK_DOWN: 1 rack (2 osds) down 2026-02-25T12:01:57.184 INFO:tasks.workunit.client.0.trial117.stdout: rack rack2 (root=root,row=row1) (2 osds) is down 2026-02-25T12:01:57.184 INFO:tasks.workunit.client.0.trial117.stdout:[WRN] OSD_ROW_DOWN: 1 row (2 osds) down 2026-02-25T12:01:57.184 INFO:tasks.workunit.client.0.trial117.stdout: row row1 (root=root) (2 osds) is down 2026-02-25T12:01:57.184 INFO:tasks.workunit.client.0.trial117.stdout:[WRN] PG_DEGRADED: Degraded data redundancy: 2/4 objects degraded (50.000%), 2 pgs degraded 2026-02-25T12:01:57.184 INFO:tasks.workunit.client.0.trial117.stdout: pg 1.2 is active+undersized+degraded+wait, acting [4] 2026-02-25T12:01:57.184 INFO:tasks.workunit.client.0.trial117.stdout: pg 1.3 is active+undersized+degraded+wait, acting [3] 2026-02-25T12:01:57.194 INFO:tasks.workunit.client.0.trial117.stderr:+ ceph health 2026-02-25T12:01:57.195 INFO:tasks.workunit.client.0.trial117.stderr:+ grep '1 host' 2026-02-25T12:01:57.431 INFO:tasks.workunit.client.0.trial117.stdout:HEALTH_WARN noup,noout flag(s) set; 2 osds down; 1 host (2 osds) down; 1 rack (2 osds) down; 1 row (2 osds) down; Degraded data redundancy: 2/4 objects degraded (50.000%), 2 pgs degraded 2026-02-25T12:01:57.432 INFO:tasks.workunit.client.0.trial117.stderr:+ ceph health 2026-02-25T12:01:57.432 INFO:tasks.workunit.client.0.trial117.stderr:+ grep '1 rack' 2026-02-25T12:01:57.663 INFO:tasks.workunit.client.0.trial117.stdout:HEALTH_WARN noup,noout flag(s) set; 2 osds down; 1 host (2 osds) down; 1 rack (2 osds) down; 1 row (2 osds) down; Degraded data redundancy: 2/4 objects degraded (50.000%), 2 pgs degraded 2026-02-25T12:01:57.664 INFO:tasks.workunit.client.0.trial117.stderr:+ ceph health 2026-02-25T12:01:57.664 INFO:tasks.workunit.client.0.trial117.stderr:+ grep '1 row' 2026-02-25T12:01:57.899 INFO:tasks.workunit.client.0.trial117.stdout:HEALTH_WARN noup,noout flag(s) set; 2 osds down; 1 host (2 osds) down; 1 rack (2 osds) down; 1 row (2 osds) down; Degraded data redundancy: 2/4 objects degraded (50.000%), 2 pgs degraded 2026-02-25T12:01:57.900 INFO:tasks.workunit.client.0.trial117.stderr:+ ceph health 2026-02-25T12:01:57.900 INFO:tasks.workunit.client.0.trial117.stderr:+ grep '2 osds' 2026-02-25T12:01:57.962 INFO:tasks.ceph.osd.9.trial117.stderr:2026-02-25T12:01:57.958+0000 7f64fdb7a640 -1 osd.9 35 osdmap NOUP flag is set, waiting for it to clear 2026-02-25T12:01:58.146 INFO:tasks.workunit.client.0.trial117.stdout:HEALTH_WARN noup,noout flag(s) set; 2 osds down; 1 host (2 osds) down; 1 rack (2 osds) down; 1 row (2 osds) down; Degraded data redundancy: 2/4 objects degraded (50.000%), 2 pgs degraded 2026-02-25T12:01:58.146 INFO:tasks.workunit.client.0.trial117.stderr:+ ceph health detail 2026-02-25T12:01:58.147 INFO:tasks.workunit.client.0.trial117.stderr:+ grep osd.8 2026-02-25T12:01:58.423 INFO:tasks.workunit.client.0.trial117.stdout: osd.8 (root=root,row=row1,rack=rack2,host=host4) is down 2026-02-25T12:01:58.423 INFO:tasks.workunit.client.0.trial117.stderr:+ ceph health detail 2026-02-25T12:01:58.424 INFO:tasks.workunit.client.0.trial117.stderr:+ grep osd.9 2026-02-25T12:01:58.679 INFO:tasks.ceph.osd.8.trial117.stderr:2026-02-25T12:01:58.674+0000 7fddc33b1640 -1 osd.8 35 osdmap NOUP flag is set, waiting for it to clear 2026-02-25T12:01:58.691 INFO:tasks.workunit.client.0.trial117.stdout: osd.9 (root=root,row=row1,rack=rack2,host=host4) is down 2026-02-25T12:01:58.691 INFO:tasks.workunit.client.0.trial117.stderr:+ ceph osd unset noup 2026-02-25T12:01:58.763 INFO:tasks.ceph.mgr.x.trial117.stderr:INFO:mgr_util:scanning for idle connections.. 2026-02-25T12:01:58.764 INFO:tasks.ceph.mgr.x.trial117.stderr:INFO:mgr_util:cleaning up connections: [] 2026-02-25T12:01:58.766 INFO:tasks.ceph.mgr.x.trial117.stderr:INFO:mgr_util:scanning for idle connections.. 2026-02-25T12:01:58.766 INFO:tasks.ceph.mgr.x.trial117.stderr:INFO:mgr_util:cleaning up connections: [] 2026-02-25T12:01:58.767 INFO:tasks.ceph.mgr.x.trial117.stderr:INFO:mgr_util:scanning for idle connections.. 2026-02-25T12:01:58.767 INFO:tasks.ceph.mgr.x.trial117.stderr:INFO:mgr_util:cleaning up connections: [] 2026-02-25T12:01:58.864 INFO:tasks.ceph.osd.8.trial117.stderr:2026-02-25T12:01:58.858+0000 7fddbe287640 -1 osd.8 36 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2026-02-25T12:01:58.864 INFO:tasks.ceph.osd.9.trial117.stderr:2026-02-25T12:01:58.858+0000 7f64f8a50640 -1 osd.9 36 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2026-02-25T12:01:59.863 INFO:tasks.workunit.client.0.trial117.stderr:noup is unset 2026-02-25T12:01:59.875 INFO:tasks.workunit.client.0.trial117.stderr:+ wait_for_healthy 2026-02-25T12:01:59.875 INFO:tasks.workunit.client.0.trial117.stderr:+ ceph health 2026-02-25T12:01:59.875 INFO:tasks.workunit.client.0.trial117.stderr:+ grep down 2026-02-25T12:02:00.113 INFO:tasks.workunit.client.0.trial117.stderr:+ test_mark_all_but_last_osds_down 2026-02-25T12:02:00.113 INFO:tasks.workunit.client.0.trial117.stderr:+ ceph osd set noup 2026-02-25T12:02:01.868 INFO:tasks.workunit.client.0.trial117.stderr:noup is set 2026-02-25T12:02:01.881 INFO:tasks.workunit.client.0.trial117.stderr:++ ceph osd ls 2026-02-25T12:02:01.881 INFO:tasks.workunit.client.0.trial117.stderr:++ sed '$d' 2026-02-25T12:02:02.089 INFO:tasks.workunit.client.0.trial117.stderr:+ ceph osd down 0 1 2 3 4 5 6 7 8 2026-02-25T12:02:02.241 INFO:tasks.ceph.mon.a.trial117.stderr:2026-02-25T12:02:02.238+0000 7f9a477c9640 -1 mon.a@0(leader).osd e39 definitely_dead 0 2026-02-25T12:02:02.922 INFO:tasks.ceph.mon.a.trial117.stderr:2026-02-25T12:02:02.918+0000 7f9a477c9640 -1 mon.a@0(leader).osd e40 definitely_dead 0 2026-02-25T12:02:02.922 INFO:tasks.workunit.client.0.trial117.stderr:osd.0 is already down. osd.1 is already down. osd.2 is already down. osd.3 is already down. osd.4 is already down. osd.5 is already down. osd.6 is already down. osd.7 is already down. osd.8 is already down. 2026-02-25T12:02:02.932 INFO:tasks.workunit.client.0.trial117.stderr:+ ceph health detail 2026-02-25T12:02:03.199 INFO:tasks.workunit.client.0.trial117.stdout:HEALTH_WARN noup,noout flag(s) set; 9 osds down; 4 hosts (8 osds) down; 2 racks (8 osds) down; 1 row (8 osds) down 2026-02-25T12:02:03.199 INFO:tasks.workunit.client.0.trial117.stdout:[WRN] OSDMAP_FLAGS: noup,noout flag(s) set 2026-02-25T12:02:03.199 INFO:tasks.workunit.client.0.trial117.stdout:[WRN] OSD_DOWN: 9 osds down 2026-02-25T12:02:03.199 INFO:tasks.workunit.client.0.trial117.stdout: osd.0 (root=root,row=row0,rack=rack0,host=host0) is down 2026-02-25T12:02:03.199 INFO:tasks.workunit.client.0.trial117.stdout: osd.1 (root=root,row=row0,rack=rack0,host=host0) is down 2026-02-25T12:02:03.199 INFO:tasks.workunit.client.0.trial117.stdout: osd.2 (root=root,row=row0,rack=rack0,host=host1) is down 2026-02-25T12:02:03.199 INFO:tasks.workunit.client.0.trial117.stdout: osd.3 (root=root,row=row0,rack=rack0,host=host1) is down 2026-02-25T12:02:03.199 INFO:tasks.workunit.client.0.trial117.stdout: osd.4 (root=root,row=row0,rack=rack1,host=host2) is down 2026-02-25T12:02:03.199 INFO:tasks.workunit.client.0.trial117.stdout: osd.5 (root=root,row=row0,rack=rack1,host=host2) is down 2026-02-25T12:02:03.199 INFO:tasks.workunit.client.0.trial117.stdout: osd.6 (root=root,row=row0,rack=rack1,host=host3) is down 2026-02-25T12:02:03.199 INFO:tasks.workunit.client.0.trial117.stdout: osd.7 (root=root,row=row0,rack=rack1,host=host3) is down 2026-02-25T12:02:03.199 INFO:tasks.workunit.client.0.trial117.stdout: osd.8 (root=root,row=row1,rack=rack2,host=host4) is down 2026-02-25T12:02:03.199 INFO:tasks.workunit.client.0.trial117.stdout:[WRN] OSD_HOST_DOWN: 4 hosts (8 osds) down 2026-02-25T12:02:03.199 INFO:tasks.workunit.client.0.trial117.stdout: host host0 (root=root,row=row0,rack=rack0) (2 osds) is down 2026-02-25T12:02:03.199 INFO:tasks.workunit.client.0.trial117.stdout: host host1 (root=root,row=row0,rack=rack0) (2 osds) is down 2026-02-25T12:02:03.199 INFO:tasks.workunit.client.0.trial117.stdout: host host2 (root=root,row=row0,rack=rack1) (2 osds) is down 2026-02-25T12:02:03.199 INFO:tasks.workunit.client.0.trial117.stdout: host host3 (root=root,row=row0,rack=rack1) (2 osds) is down 2026-02-25T12:02:03.199 INFO:tasks.workunit.client.0.trial117.stdout:[WRN] OSD_RACK_DOWN: 2 racks (8 osds) down 2026-02-25T12:02:03.199 INFO:tasks.workunit.client.0.trial117.stdout: rack rack0 (root=root,row=row0) (4 osds) is down 2026-02-25T12:02:03.199 INFO:tasks.workunit.client.0.trial117.stdout: rack rack1 (root=root,row=row0) (4 osds) is down 2026-02-25T12:02:03.199 INFO:tasks.workunit.client.0.trial117.stdout:[WRN] OSD_ROW_DOWN: 1 row (8 osds) down 2026-02-25T12:02:03.199 INFO:tasks.workunit.client.0.trial117.stdout: row row0 (root=root) (8 osds) is down 2026-02-25T12:02:03.208 INFO:tasks.workunit.client.0.trial117.stderr:+ ceph health 2026-02-25T12:02:03.208 INFO:tasks.workunit.client.0.trial117.stderr:+ grep '1 row' 2026-02-25T12:02:03.466 INFO:tasks.workunit.client.0.trial117.stdout:HEALTH_WARN noup,noout flag(s) set; 9 osds down; 4 hosts (8 osds) down; 2 racks (8 osds) down; 1 row (8 osds) down 2026-02-25T12:02:03.466 INFO:tasks.workunit.client.0.trial117.stderr:+ ceph health 2026-02-25T12:02:03.466 INFO:tasks.workunit.client.0.trial117.stderr:+ grep '2 racks' 2026-02-25T12:02:03.582 INFO:tasks.ceph.osd.8.trial117.stderr:2026-02-25T12:02:03.578+0000 7fddc2bb0640 -1 osd.8 40 osdmap NOUP flag is set, waiting for it to clear 2026-02-25T12:02:03.702 INFO:tasks.workunit.client.0.trial117.stdout:HEALTH_WARN noup,noout flag(s) set; 9 osds down; 4 hosts (8 osds) down; 2 racks (8 osds) down; 1 row (8 osds) down 2026-02-25T12:02:03.702 INFO:tasks.workunit.client.0.trial117.stderr:+ ceph health 2026-02-25T12:02:03.703 INFO:tasks.workunit.client.0.trial117.stderr:+ grep '4 hosts' 2026-02-25T12:02:03.713 INFO:tasks.ceph.osd.4.trial117.stderr:2026-02-25T12:02:03.710+0000 7efcac471640 -1 osd.4 40 osdmap NOUP flag is set, waiting for it to clear 2026-02-25T12:02:03.842 INFO:tasks.ceph.osd.7.trial117.stderr:2026-02-25T12:02:03.838+0000 7f0396606640 -1 osd.7 40 osdmap NOUP flag is set, waiting for it to clear 2026-02-25T12:02:03.874 INFO:tasks.ceph.osd.7.trial117.stderr:2026-02-25T12:02:03.870+0000 7f0389192640 -1 osd.7 41 osdmap NOUP flag is set, waiting for it to clear 2026-02-25T12:02:03.874 INFO:tasks.ceph.osd.4.trial117.stderr:2026-02-25T12:02:03.870+0000 7efc9aff5640 -1 osd.4 41 osdmap NOUP flag is set, waiting for it to clear 2026-02-25T12:02:03.874 INFO:tasks.ceph.osd.8.trial117.stderr:2026-02-25T12:02:03.870+0000 7fddb5f3d640 -1 osd.8 41 osdmap NOUP flag is set, waiting for it to clear 2026-02-25T12:02:03.942 INFO:tasks.workunit.client.0.trial117.stdout:HEALTH_WARN noup,noout flag(s) set; 9 osds down; 4 hosts (8 osds) down; 2 racks (8 osds) down; 1 row (8 osds) down 2026-02-25T12:02:03.942 INFO:tasks.workunit.client.0.trial117.stderr:+ ceph health 2026-02-25T12:02:03.942 INFO:tasks.workunit.client.0.trial117.stderr:+ grep '9 osds' 2026-02-25T12:02:04.058 INFO:tasks.ceph.osd.1.trial117.stderr:2026-02-25T12:02:04.054+0000 7fbccca80640 -1 osd.1 41 osdmap NOUP flag is set, waiting for it to clear 2026-02-25T12:02:04.064 INFO:tasks.ceph.osd.5.trial117.stderr:2026-02-25T12:02:04.058+0000 7f0fe326f640 -1 osd.5 41 osdmap NOUP flag is set, waiting for it to clear 2026-02-25T12:02:04.130 INFO:tasks.ceph.osd.3.trial117.stderr:2026-02-25T12:02:04.126+0000 7fb68ad7d640 -1 osd.3 41 osdmap NOUP flag is set, waiting for it to clear 2026-02-25T12:02:04.186 INFO:tasks.workunit.client.0.trial117.stdout:HEALTH_WARN noup,noout flag(s) set; 9 osds down; 4 hosts (8 osds) down; 2 racks (8 osds) down; 1 row (8 osds) down 2026-02-25T12:02:04.186 INFO:tasks.workunit.client.0.trial117.stderr:+ ceph osd unset noup 2026-02-25T12:02:04.878 INFO:tasks.ceph.osd.4.trial117.stderr:2026-02-25T12:02:04.873+0000 7efca333f640 -1 osd.4 42 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2026-02-25T12:02:04.879 INFO:tasks.ceph.osd.5.trial117.stderr:2026-02-25T12:02:04.873+0000 7f0fda13d640 -1 osd.5 42 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2026-02-25T12:02:04.879 INFO:tasks.ceph.osd.7.trial117.stderr:2026-02-25T12:02:04.873+0000 7f03914dc640 -1 osd.7 42 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2026-02-25T12:02:04.879 INFO:tasks.ceph.osd.8.trial117.stderr:2026-02-25T12:02:04.873+0000 7fddbe287640 -1 osd.8 42 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2026-02-25T12:02:04.879 INFO:tasks.ceph.osd.1.trial117.stderr:2026-02-25T12:02:04.873+0000 7fbcc414f640 -1 osd.1 42 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2026-02-25T12:02:04.879 INFO:tasks.ceph.osd.3.trial117.stderr:2026-02-25T12:02:04.873+0000 7fb685c53640 -1 osd.3 42 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2026-02-25T12:02:05.296 INFO:tasks.ceph.osd.6.trial117.stderr:2026-02-25T12:02:05.289+0000 7fe191c40640 -1 osd.6 42 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2026-02-25T12:02:05.829 INFO:tasks.ceph.osd.2.trial117.stderr:2026-02-25T12:02:05.825+0000 7f0b4e037640 -1 osd.2 42 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2026-02-25T12:02:05.877 INFO:tasks.workunit.client.0.trial117.stderr:noup is unset 2026-02-25T12:02:05.891 INFO:tasks.workunit.client.0.trial117.stderr:+ wait_for_healthy 2026-02-25T12:02:05.891 INFO:tasks.workunit.client.0.trial117.stderr:+ ceph health 2026-02-25T12:02:05.891 INFO:tasks.workunit.client.0.trial117.stderr:+ grep down 2026-02-25T12:02:06.128 INFO:tasks.workunit.client.0.trial117.stdout:HEALTH_WARN noout flag(s) set; 1 osds down 2026-02-25T12:02:06.128 INFO:tasks.workunit.client.0.trial117.stderr:+ sleep 1 2026-02-25T12:02:07.059 INFO:tasks.ceph.osd.0.trial117.stderr:2026-02-25T12:02:07.053+0000 7f8a11442640 -1 osd.0 44 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2026-02-25T12:02:07.129 INFO:tasks.workunit.client.0.trial117.stderr:+ ceph health 2026-02-25T12:02:07.129 INFO:tasks.workunit.client.0.trial117.stderr:+ grep down 2026-02-25T12:02:07.377 INFO:tasks.workunit.client.0.trial117.stdout:HEALTH_WARN noout flag(s) set; 1 osds down 2026-02-25T12:02:07.378 INFO:tasks.workunit.client.0.trial117.stderr:+ sleep 1 2026-02-25T12:02:08.379 INFO:tasks.workunit.client.0.trial117.stderr:+ ceph health 2026-02-25T12:02:08.379 INFO:tasks.workunit.client.0.trial117.stderr:+ grep down 2026-02-25T12:02:08.619 INFO:tasks.workunit.client.0.trial117.stderr:+ test_mark_two_osds_same_host_down_with_classes 2026-02-25T12:02:08.620 INFO:tasks.workunit.client.0.trial117.stderr:+ ceph osd set noup 2026-02-25T12:02:09.888 INFO:tasks.workunit.client.0.trial117.stderr:noup is set 2026-02-25T12:02:09.902 INFO:tasks.workunit.client.0.trial117.stderr:+ ceph osd crush set-device-class ssd osd.0 osd.2 osd.4 osd.6 osd.8 2026-02-25T12:02:11.892 INFO:tasks.workunit.client.0.trial117.stderr:osd.0 already set to class ssd. set-device-class item id 0 name 'osd.0' device_class 'ssd': no change. osd.2 already set to class ssd. set-device-class item id 2 name 'osd.2' device_class 'ssd': no change. osd.4 already set to class ssd. set-device-class item id 4 name 'osd.4' device_class 'ssd': no change. osd.6 already set to class ssd. set-device-class item id 6 name 'osd.6' device_class 'ssd': no change. osd.8 already set to class ssd. set-device-class item id 8 name 'osd.8' device_class 'ssd': no change. set osd(s) to class 'ssd' 2026-02-25T12:02:11.904 INFO:tasks.workunit.client.0.trial117.stderr:+ ceph osd crush set-device-class hdd osd.1 osd.3 osd.5 osd.7 osd.9 2026-02-25T12:02:13.894 INFO:tasks.workunit.client.0.trial117.stderr:osd.1 already set to class hdd. set-device-class item id 1 name 'osd.1' device_class 'hdd': no change. osd.3 already set to class hdd. set-device-class item id 3 name 'osd.3' device_class 'hdd': no change. osd.5 already set to class hdd. set-device-class item id 5 name 'osd.5' device_class 'hdd': no change. osd.7 already set to class hdd. set-device-class item id 7 name 'osd.7' device_class 'hdd': no change. osd.9 already set to class hdd. set-device-class item id 9 name 'osd.9' device_class 'hdd': no change. set osd(s) to class 'hdd' 2026-02-25T12:02:13.906 INFO:tasks.workunit.client.0.trial117.stderr:+ ceph osd down osd.0 osd.1 2026-02-25T12:02:14.051 INFO:tasks.ceph.mon.a.trial117.stderr:2026-02-25T12:02:14.045+0000 7f9a477c9640 -1 mon.a@0(leader).osd e51 definitely_dead 0 2026-02-25T12:02:14.950 INFO:tasks.ceph.mon.a.trial117.stderr:2026-02-25T12:02:14.945+0000 7f9a477c9640 -1 mon.a@0(leader).osd e52 definitely_dead 0 2026-02-25T12:02:14.950 INFO:tasks.workunit.client.0.trial117.stderr:osd.0 is already down. osd.1 is already down. 2026-02-25T12:02:14.962 INFO:tasks.workunit.client.0.trial117.stderr:+ ceph health detail 2026-02-25T12:02:15.234 INFO:tasks.workunit.client.0.trial117.stdout:HEALTH_WARN noup,noout flag(s) set; 2 osds down; 1 host (2 osds) down 2026-02-25T12:02:15.234 INFO:tasks.workunit.client.0.trial117.stdout:[WRN] OSDMAP_FLAGS: noup,noout flag(s) set 2026-02-25T12:02:15.234 INFO:tasks.workunit.client.0.trial117.stdout:[WRN] OSD_DOWN: 2 osds down 2026-02-25T12:02:15.235 INFO:tasks.workunit.client.0.trial117.stdout: osd.0 (root=root,row=row0,rack=rack0,host=host0) is down 2026-02-25T12:02:15.235 INFO:tasks.workunit.client.0.trial117.stdout: osd.1 (root=root,row=row0,rack=rack0,host=host0) is down 2026-02-25T12:02:15.235 INFO:tasks.workunit.client.0.trial117.stdout:[WRN] OSD_HOST_DOWN: 1 host (2 osds) down 2026-02-25T12:02:15.235 INFO:tasks.workunit.client.0.trial117.stdout: host host0 (root=root,row=row0,rack=rack0) (2 osds) is down 2026-02-25T12:02:15.244 INFO:tasks.workunit.client.0.trial117.stderr:+ ceph health 2026-02-25T12:02:15.244 INFO:tasks.workunit.client.0.trial117.stderr:+ grep '1 host' 2026-02-25T12:02:15.506 INFO:tasks.workunit.client.0.trial117.stdout:HEALTH_WARN noup,noout flag(s) set; 2 osds down; 1 host (2 osds) down 2026-02-25T12:02:15.507 INFO:tasks.workunit.client.0.trial117.stderr:+ ceph health 2026-02-25T12:02:15.507 INFO:tasks.workunit.client.0.trial117.stderr:+ grep '2 osds' 2026-02-25T12:02:15.746 INFO:tasks.workunit.client.0.trial117.stdout:HEALTH_WARN noup,noout flag(s) set; 2 osds down; 1 host (2 osds) down 2026-02-25T12:02:15.747 INFO:tasks.workunit.client.0.trial117.stderr:+ ceph health detail 2026-02-25T12:02:15.747 INFO:tasks.workunit.client.0.trial117.stderr:+ grep osd.0 2026-02-25T12:02:16.015 INFO:tasks.workunit.client.0.trial117.stdout: osd.0 (root=root,row=row0,rack=rack0,host=host0) is down 2026-02-25T12:02:16.016 INFO:tasks.workunit.client.0.trial117.stderr:+ ceph health detail 2026-02-25T12:02:16.017 INFO:tasks.workunit.client.0.trial117.stderr:+ grep osd.1 2026-02-25T12:02:16.064 INFO:tasks.ceph.osd.0.trial117.stderr:2026-02-25T12:02:16.057+0000 7f8a1a574640 -1 osd.0 53 osdmap NOUP flag is set, waiting for it to clear 2026-02-25T12:02:16.274 INFO:tasks.workunit.client.0.trial117.stdout: osd.1 (root=root,row=row0,rack=rack0,host=host0) is down 2026-02-25T12:02:16.274 INFO:tasks.workunit.client.0.trial117.stderr:+ ceph osd unset noup 2026-02-25T12:02:16.904 INFO:tasks.ceph.osd.0.trial117.stderr:2026-02-25T12:02:16.897+0000 7f8a11442640 -1 osd.0 54 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2026-02-25T12:02:17.086 INFO:tasks.ceph.osd.1.trial117.stderr:2026-02-25T12:02:17.081+0000 7fbcc414f640 -1 osd.1 54 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2026-02-25T12:02:17.903 INFO:tasks.workunit.client.0.trial117.stderr:noup is unset 2026-02-25T12:02:17.916 INFO:tasks.workunit.client.0.trial117.stderr:+ wait_for_healthy 2026-02-25T12:02:17.916 INFO:tasks.workunit.client.0.trial117.stderr:+ ceph health 2026-02-25T12:02:17.916 INFO:tasks.workunit.client.0.trial117.stderr:+ grep down 2026-02-25T12:02:18.155 INFO:tasks.workunit.client.0.trial117.stderr:+ exit 0 2026-02-25T12:02:18.156 INFO:teuthology.orchestra.run:Running command with timeout 3600 2026-02-25T12:02:18.156 DEBUG:teuthology.orchestra.run.trial117:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2026-02-25T12:02:18.205 INFO:tasks.workunit:Stopping ['rados/test_health_warnings.sh'] on client.0... 2026-02-25T12:02:18.205 DEBUG:teuthology.orchestra.run.trial117:> sudo rm -rf -- /home/ubuntu/cephtest/workunits.list.client.0 /home/ubuntu/cephtest/clone.client.0 2026-02-25T12:02:18.565 DEBUG:teuthology.parallel:result is None 2026-02-25T12:02:18.565 DEBUG:teuthology.orchestra.run.trial117:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0 2026-02-25T12:02:18.572 INFO:tasks.workunit:Deleted dir /home/ubuntu/cephtest/mnt.0/client.0 2026-02-25T12:02:18.572 DEBUG:teuthology.orchestra.run.trial117:> rmdir -- /home/ubuntu/cephtest/mnt.0 2026-02-25T12:02:18.620 INFO:tasks.workunit:Deleted artificial mount point /home/ubuntu/cephtest/mnt.0/client.0 2026-02-25T12:02:18.621 DEBUG:teuthology.run_tasks:Unwinding manager ceph 2026-02-25T12:02:18.629 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2026-02-25T12:02:18.629 DEBUG:teuthology.orchestra.run.trial117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-25T12:02:18.817 INFO:teuthology.orchestra.run.trial117.stdout: 2026-02-25T12:02:18.818 INFO:teuthology.orchestra.run.trial117.stderr:dumped all 2026-02-25T12:02:18.827 INFO:teuthology.orchestra.run.trial117.stdout:{"pg_ready":true,"pg_map":{"version":92,"stamp":"2026-02-25T12:02:18.734848+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":14,"acting":14,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":10,"num_per_pool_osds":10,"num_per_pool_omap_osds":8,"kb":1048576000,"kb_used":275256,"kb_used_data":5656,"kb_used_omap":236,"kb_used_meta":269203,"kb_avail":1048300744,"statfs":{"total":1073741824000,"available":1073459961856,"internally_reserved":0,"allocated":5791744,"data_stored":2694898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":242260,"internal_metadata":275664300},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.000589"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":69,"reported_epoch":54,"state":"active+clean","last_fresh":"2026-02-25T12:02:16.933086+0000","last_change":"2026-02-25T12:02:06.883264+0000","last_active":"2026-02-25T12:02:16.933086+0000","last_peered":"2026-02-25T12:02:16.933086+0000","last_clean":"2026-02-25T12:02:16.933086+0000","last_became_active":"2026-02-25T12:02:06.883022+0000","last_became_peered":"2026-02-25T12:02:06.883022+0000","last_unstale":"2026-02-25T12:02:16.933086+0000","last_undegraded":"2026-02-25T12:02:16.933086+0000","last_fullsized":"2026-02-25T12:02:16.933086+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-25T12:01:08.885417+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-25T12:01:08.885417+0000","last_clean_scrub_stamp":"2026-02-25T12:01:08.885417+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-26T22:09:05.780239+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00026608399999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":87,"reported_epoch":54,"state":"active+clean","last_fresh":"2026-02-25T12:02:16.933144+0000","last_change":"2026-02-25T12:02:06.883262+0000","last_active":"2026-02-25T12:02:16.933144+0000","last_peered":"2026-02-25T12:02:16.933144+0000","last_clean":"2026-02-25T12:02:16.933144+0000","last_became_active":"2026-02-25T12:02:06.883003+0000","last_became_peered":"2026-02-25T12:02:06.883003+0000","last_unstale":"2026-02-25T12:02:16.933144+0000","last_undegraded":"2026-02-25T12:02:16.933144+0000","last_fullsized":"2026-02-25T12:02:16.933144+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-25T12:01:08.885417+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-25T12:01:08.885417+0000","last_clean_scrub_stamp":"2026-02-25T12:01:08.885417+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-26T15:40:20.663132+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00030945499999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":86,"reported_epoch":50,"state":"active+clean","last_fresh":"2026-02-25T12:02:12.933109+0000","last_change":"2026-02-25T12:02:09.321129+0000","last_active":"2026-02-25T12:02:12.933109+0000","last_peered":"2026-02-25T12:02:12.933109+0000","last_clean":"2026-02-25T12:02:12.933109+0000","last_became_active":"2026-02-25T12:02:09.320974+0000","last_became_peered":"2026-02-25T12:02:09.320974+0000","last_unstale":"2026-02-25T12:02:12.933109+0000","last_undegraded":"2026-02-25T12:02:12.933109+0000","last_fullsized":"2026-02-25T12:02:12.933109+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-25T12:01:08.885417+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-25T12:01:08.885417+0000","last_clean_scrub_stamp":"2026-02-25T12:01:08.885417+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-26T20:04:59.198096+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00017141899999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0],"acting":[9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":54,"reported_epoch":54,"state":"active+undersized+wait","last_fresh":"2026-02-25T12:02:16.933130+0000","last_change":"2026-02-25T12:02:15.902089+0000","last_active":"2026-02-25T12:02:16.933130+0000","last_peered":"2026-02-25T12:02:16.933130+0000","last_clean":"2026-02-25T12:02:02.711424+0000","last_became_active":"2026-02-25T12:02:15.902089+0000","last_became_peered":"2026-02-25T12:02:15.902089+0000","last_unstale":"2026-02-25T12:02:16.933130+0000","last_undegraded":"2026-02-25T12:02:16.933130+0000","last_fullsized":"2026-02-25T12:02:15.901707+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-25T12:01:08.885417+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-25T12:01:08.885417+0000","last_clean_scrub_stamp":"2026-02-25T12:01:08.885417+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-26T23:11:50.865351+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6],"acting":[6],"avail_no_missing":["6"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.3","version":"18'1","reported_seq":83,"reported_epoch":54,"state":"active+undersized+degraded+wait","last_fresh":"2026-02-25T12:02:16.933084+0000","last_change":"2026-02-25T12:02:15.902037+0000","last_active":"2026-02-25T12:02:16.933084+0000","last_peered":"2026-02-25T12:02:16.933084+0000","last_clean":"2026-02-25T12:02:13.894638+0000","last_became_active":"2026-02-25T12:02:15.902037+0000","last_became_peered":"2026-02-25T12:02:15.902037+0000","last_unstale":"2026-02-25T12:02:16.933084+0000","last_undegraded":"2026-02-25T12:02:15.901816+0000","last_fullsized":"2026-02-25T12:02:15.901757+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-25T12:01:08.885417+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-25T12:01:08.885417+0000","last_clean_scrub_stamp":"2026-02-25T12:01:08.885417+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-26T17:32:07.261620+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[{"shards":"3","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"20'2","reported_seq":77,"reported_epoch":50,"state":"active+clean","last_fresh":"2026-02-25T12:02:13.433185+0000","last_change":"2026-02-25T12:02:06.883155+0000","last_active":"2026-02-25T12:02:13.433185+0000","last_peered":"2026-02-25T12:02:13.433185+0000","last_clean":"2026-02-25T12:02:13.433185+0000","last_became_active":"2026-02-25T12:02:06.882955+0000","last_became_peered":"2026-02-25T12:02:06.882955+0000","last_unstale":"2026-02-25T12:02:13.433185+0000","last_undegraded":"2026-02-25T12:02:13.433185+0000","last_fullsized":"2026-02-25T12:02:13.433185+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-25T12:01:08.885417+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-25T12:01:08.885417+0000","last_clean_scrub_stamp":"2026-02-25T12:01:08.885417+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-26T21:17:56.111673+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062137499999999997,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":63,"reported_epoch":50,"state":"active+clean","last_fresh":"2026-02-25T12:02:12.894142+0000","last_change":"2026-02-25T12:02:06.883333+0000","last_active":"2026-02-25T12:02:12.894142+0000","last_peered":"2026-02-25T12:02:12.894142+0000","last_clean":"2026-02-25T12:02:12.894142+0000","last_became_active":"2026-02-25T12:02:06.883131+0000","last_became_peered":"2026-02-25T12:02:06.883131+0000","last_unstale":"2026-02-25T12:02:12.894142+0000","last_undegraded":"2026-02-25T12:02:12.894142+0000","last_fullsized":"2026-02-25T12:02:12.894142+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-25T12:01:08.885417+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-25T12:01:08.885417+0000","last_clean_scrub_stamp":"2026-02-25T12:01:08.885417+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-26T17:10:55.006863+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00015265299999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":74,"reported_epoch":50,"state":"active+clean","last_fresh":"2026-02-25T12:02:13.433241+0000","last_change":"2026-02-25T12:02:07.090070+0000","last_active":"2026-02-25T12:02:13.433241+0000","last_peered":"2026-02-25T12:02:13.433241+0000","last_clean":"2026-02-25T12:02:13.433241+0000","last_became_active":"2026-02-25T12:02:07.089906+0000","last_became_peered":"2026-02-25T12:02:07.089906+0000","last_unstale":"2026-02-25T12:02:13.433241+0000","last_undegraded":"2026-02-25T12:02:13.433241+0000","last_fullsized":"2026-02-25T12:02:13.433241+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-25T12:01:08.885417+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-25T12:01:08.885417+0000","last_clean_scrub_stamp":"2026-02-25T12:01:08.885417+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-26T23:02:05.903788+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9],"acting":[5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":14,"acting":14,"num_store_stats":9}],"osd_stats":[{"osd":9,"up_from":37,"seq":158913789968,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":27508,"kb_used_data":548,"kb_used_omap":24,"kb_used_meta":26919,"kb_avail":104830092,"statfs":{"total":107374182400,"available":107346014208,"internally_reserved":0,"allocated":561152,"data_stored":258172,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":25143,"internal_metadata":27565513},"hb_peers":[0,1,2,3,4,5,6,7,8],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":43,"seq":184683593745,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":104857600,"kb_used":27508,"kb_used_data":548,"kb_used_omap":22,"kb_used_meta":26921,"kb_avail":104830092,"statfs":{"total":107374182400,"available":107346014208,"internally_reserved":0,"allocated":561152,"data_stored":258172,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":23185,"internal_metadata":27567471},"hb_peers":[0,1,2,3,4,5,6,7,9],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":43,"seq":184683593744,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":27508,"kb_used_data":548,"kb_used_omap":24,"kb_used_meta":26919,"kb_avail":104830092,"statfs":{"total":107374182400,"available":107346014208,"internally_reserved":0,"allocated":561152,"data_stored":258172,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":25132,"internal_metadata":27565524},"hb_peers":[0,1,2,3,4,5,6,8,9],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":43,"seq":184683593744,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":104857600,"kb_used":27496,"kb_used_data":536,"kb_used_omap":22,"kb_used_meta":26921,"kb_avail":104830104,"statfs":{"total":107374182400,"available":107346026496,"internally_reserved":0,"allocated":548864,"data_stored":247770,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":23185,"internal_metadata":27567471},"hb_peers":[0,1,3,4,5,6,7,8,9],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":43,"seq":184683593745,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":27556,"kb_used_data":596,"kb_used_omap":24,"kb_used_meta":26919,"kb_avail":104830044,"statfs":{"total":107374182400,"available":107345965056,"internally_reserved":0,"allocated":610304,"data_stored":291658,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":25136,"internal_metadata":27565520},"hb_peers":[2,3,4,5,7,8,9],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":43,"seq":184683593745,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":27508,"kb_used_data":548,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":104830092,"statfs":{"total":107374182400,"available":107346014208,"internally_reserved":0,"allocated":561152,"data_stored":258172,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17326,"internal_metadata":27573330},"hb_peers":[0,1,2,3,4,6,7,8,9],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":43,"seq":184683593744,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":27500,"kb_used_data":540,"kb_used_omap":24,"kb_used_meta":26919,"kb_avail":104830100,"statfs":{"total":107374182400,"available":107346022400,"internally_reserved":0,"allocated":552960,"data_stored":247789,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":25138,"internal_metadata":27565518},"hb_peers":[0,1,2,3,5,6,7,8,9],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":43,"seq":184683593746,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":27556,"kb_used_data":596,"kb_used_omap":26,"kb_used_meta":26917,"kb_avail":104830044,"statfs":{"total":107374182400,"available":107345965056,"internally_reserved":0,"allocated":610304,"data_stored":291658,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":27088,"internal_metadata":27563568},"hb_peers":[2,4,5,6,7,8,9],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":55,"seq":236223201297,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":27560,"kb_used_data":600,"kb_used_omap":23,"kb_used_meta":26920,"kb_avail":104830040,"statfs":{"total":107374182400,"available":107345960960,"internally_reserved":0,"allocated":614400,"data_stored":291677,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":24487,"internal_metadata":27566169},"hb_peers":[2,3,4,5,6,7,8,9],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":55,"seq":236223201297,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":27556,"kb_used_data":596,"kb_used_omap":25,"kb_used_meta":26918,"kb_avail":104830044,"statfs":{"total":107374182400,"available":107345965056,"internally_reserved":0,"allocated":610304,"data_stored":291658,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":26440,"internal_metadata":27564216},"hb_peers":[2,3,4,5,6,7,8,9],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-25T12:02:18.828 DEBUG:teuthology.orchestra.run.trial117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-25T12:02:18.981 INFO:teuthology.orchestra.run.trial117.stdout: 2026-02-25T12:02:18.982 INFO:teuthology.orchestra.run.trial117.stderr:dumped all 2026-02-25T12:02:18.993 INFO:teuthology.orchestra.run.trial117.stdout:{"pg_ready":true,"pg_map":{"version":93,"stamp":"2026-02-25T12:02:18.902998+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":14,"acting":14,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":10,"num_per_pool_osds":10,"num_per_pool_omap_osds":8,"kb":1048576000,"kb_used":275376,"kb_used_data":5776,"kb_used_omap":239,"kb_used_meta":269200,"kb_avail":1048300624,"statfs":{"total":1073741824000,"available":1073459838976,"internally_reserved":0,"allocated":5914624,"data_stored":2780746,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":244864,"internal_metadata":275661696},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.005419"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":69,"reported_epoch":54,"state":"active+clean","last_fresh":"2026-02-25T12:02:16.933086+0000","last_change":"2026-02-25T12:02:06.883264+0000","last_active":"2026-02-25T12:02:16.933086+0000","last_peered":"2026-02-25T12:02:16.933086+0000","last_clean":"2026-02-25T12:02:16.933086+0000","last_became_active":"2026-02-25T12:02:06.883022+0000","last_became_peered":"2026-02-25T12:02:06.883022+0000","last_unstale":"2026-02-25T12:02:16.933086+0000","last_undegraded":"2026-02-25T12:02:16.933086+0000","last_fullsized":"2026-02-25T12:02:16.933086+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-25T12:01:08.885417+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-25T12:01:08.885417+0000","last_clean_scrub_stamp":"2026-02-25T12:01:08.885417+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-26T22:09:05.780239+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00026608399999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":87,"reported_epoch":54,"state":"active+clean","last_fresh":"2026-02-25T12:02:16.933144+0000","last_change":"2026-02-25T12:02:06.883262+0000","last_active":"2026-02-25T12:02:16.933144+0000","last_peered":"2026-02-25T12:02:16.933144+0000","last_clean":"2026-02-25T12:02:16.933144+0000","last_became_active":"2026-02-25T12:02:06.883003+0000","last_became_peered":"2026-02-25T12:02:06.883003+0000","last_unstale":"2026-02-25T12:02:16.933144+0000","last_undegraded":"2026-02-25T12:02:16.933144+0000","last_fullsized":"2026-02-25T12:02:16.933144+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-25T12:01:08.885417+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-25T12:01:08.885417+0000","last_clean_scrub_stamp":"2026-02-25T12:01:08.885417+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-26T15:40:20.663132+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00030945499999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":86,"reported_epoch":50,"state":"active+clean","last_fresh":"2026-02-25T12:02:12.933109+0000","last_change":"2026-02-25T12:02:09.321129+0000","last_active":"2026-02-25T12:02:12.933109+0000","last_peered":"2026-02-25T12:02:12.933109+0000","last_clean":"2026-02-25T12:02:12.933109+0000","last_became_active":"2026-02-25T12:02:09.320974+0000","last_became_peered":"2026-02-25T12:02:09.320974+0000","last_unstale":"2026-02-25T12:02:12.933109+0000","last_undegraded":"2026-02-25T12:02:12.933109+0000","last_fullsized":"2026-02-25T12:02:12.933109+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-25T12:01:08.885417+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-25T12:01:08.885417+0000","last_clean_scrub_stamp":"2026-02-25T12:01:08.885417+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-26T20:04:59.198096+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00017141899999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0],"acting":[9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":54,"reported_epoch":54,"state":"active+undersized+wait","last_fresh":"2026-02-25T12:02:16.933130+0000","last_change":"2026-02-25T12:02:15.902089+0000","last_active":"2026-02-25T12:02:16.933130+0000","last_peered":"2026-02-25T12:02:16.933130+0000","last_clean":"2026-02-25T12:02:02.711424+0000","last_became_active":"2026-02-25T12:02:15.902089+0000","last_became_peered":"2026-02-25T12:02:15.902089+0000","last_unstale":"2026-02-25T12:02:16.933130+0000","last_undegraded":"2026-02-25T12:02:16.933130+0000","last_fullsized":"2026-02-25T12:02:15.901707+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-25T12:01:08.885417+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-25T12:01:08.885417+0000","last_clean_scrub_stamp":"2026-02-25T12:01:08.885417+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-26T23:11:50.865351+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6],"acting":[6],"avail_no_missing":["6"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.3","version":"18'1","reported_seq":83,"reported_epoch":54,"state":"active+undersized+degraded+wait","last_fresh":"2026-02-25T12:02:16.933084+0000","last_change":"2026-02-25T12:02:15.902037+0000","last_active":"2026-02-25T12:02:16.933084+0000","last_peered":"2026-02-25T12:02:16.933084+0000","last_clean":"2026-02-25T12:02:13.894638+0000","last_became_active":"2026-02-25T12:02:15.902037+0000","last_became_peered":"2026-02-25T12:02:15.902037+0000","last_unstale":"2026-02-25T12:02:16.933084+0000","last_undegraded":"2026-02-25T12:02:15.901816+0000","last_fullsized":"2026-02-25T12:02:15.901757+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-25T12:01:08.885417+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-25T12:01:08.885417+0000","last_clean_scrub_stamp":"2026-02-25T12:01:08.885417+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-26T17:32:07.261620+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[{"shards":"3","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"20'2","reported_seq":88,"reported_epoch":54,"state":"peering","last_fresh":"2026-02-25T12:02:16.933285+0000","last_change":"2026-02-25T12:02:18.346734+0000","last_active":"2026-02-25T12:02:18.346734+0000","last_peered":"2026-02-25T12:02:16.933285+0000","last_clean":"2026-02-25T12:02:15.360485+0000","last_became_active":"2026-02-25T12:02:15.902237+0000","last_became_peered":"2026-02-25T12:02:15.902237+0000","last_unstale":"2026-02-25T12:02:16.933285+0000","last_undegraded":"2026-02-25T12:02:15.901810+0000","last_fullsized":"2026-02-25T12:02:15.901745+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-25T12:01:08.885417+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-25T12:01:08.885417+0000","last_clean_scrub_stamp":"2026-02-25T12:01:08.885417+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062137499999999997,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":63,"reported_epoch":50,"state":"active+clean","last_fresh":"2026-02-25T12:02:12.894142+0000","last_change":"2026-02-25T12:02:06.883333+0000","last_active":"2026-02-25T12:02:12.894142+0000","last_peered":"2026-02-25T12:02:12.894142+0000","last_clean":"2026-02-25T12:02:12.894142+0000","last_became_active":"2026-02-25T12:02:06.883131+0000","last_became_peered":"2026-02-25T12:02:06.883131+0000","last_unstale":"2026-02-25T12:02:12.894142+0000","last_undegraded":"2026-02-25T12:02:12.894142+0000","last_fullsized":"2026-02-25T12:02:12.894142+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-25T12:01:08.885417+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-25T12:01:08.885417+0000","last_clean_scrub_stamp":"2026-02-25T12:01:08.885417+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-26T17:10:55.006863+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00015265299999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":84,"reported_epoch":55,"state":"active+clean","last_fresh":"2026-02-25T12:02:18.563160+0000","last_change":"2026-02-25T12:02:07.090070+0000","last_active":"2026-02-25T12:02:18.563160+0000","last_peered":"2026-02-25T12:02:18.563160+0000","last_clean":"2026-02-25T12:02:18.563160+0000","last_became_active":"2026-02-25T12:02:07.089906+0000","last_became_peered":"2026-02-25T12:02:07.089906+0000","last_unstale":"2026-02-25T12:02:18.563160+0000","last_undegraded":"2026-02-25T12:02:18.563160+0000","last_fullsized":"2026-02-25T12:02:18.563160+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-25T12:01:08.885417+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-25T12:01:08.885417+0000","last_clean_scrub_stamp":"2026-02-25T12:01:08.885417+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-26T23:02:05.903788+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9],"acting":[5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":14,"acting":14,"num_store_stats":9}],"osd_stats":[{"osd":9,"up_from":37,"seq":158913789968,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":27508,"kb_used_data":548,"kb_used_omap":24,"kb_used_meta":26919,"kb_avail":104830092,"statfs":{"total":107374182400,"available":107346014208,"internally_reserved":0,"allocated":561152,"data_stored":258172,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":25143,"internal_metadata":27565513},"hb_peers":[0,1,2,3,4,5,6,7,8],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":43,"seq":184683593745,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":104857600,"kb_used":27508,"kb_used_data":548,"kb_used_omap":22,"kb_used_meta":26921,"kb_avail":104830092,"statfs":{"total":107374182400,"available":107346014208,"internally_reserved":0,"allocated":561152,"data_stored":258172,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":23185,"internal_metadata":27567471},"hb_peers":[0,1,2,3,4,5,6,7,9],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":43,"seq":184683593744,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":27508,"kb_used_data":548,"kb_used_omap":24,"kb_used_meta":26919,"kb_avail":104830092,"statfs":{"total":107374182400,"available":107346014208,"internally_reserved":0,"allocated":561152,"data_stored":258172,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":25132,"internal_metadata":27565524},"hb_peers":[0,1,2,3,4,5,6,8,9],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":43,"seq":184683593744,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":104857600,"kb_used":27496,"kb_used_data":536,"kb_used_omap":22,"kb_used_meta":26921,"kb_avail":104830104,"statfs":{"total":107374182400,"available":107346026496,"internally_reserved":0,"allocated":548864,"data_stored":247770,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":23185,"internal_metadata":27567471},"hb_peers":[0,1,3,4,5,6,7,8,9],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":43,"seq":184683593745,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":27556,"kb_used_data":596,"kb_used_omap":24,"kb_used_meta":26919,"kb_avail":104830044,"statfs":{"total":107374182400,"available":107345965056,"internally_reserved":0,"allocated":610304,"data_stored":291658,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":25136,"internal_metadata":27565520},"hb_peers":[2,3,4,5,7,8,9],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":43,"seq":184683593746,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":27568,"kb_used_data":608,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":104830032,"statfs":{"total":107374182400,"available":107345952768,"internally_reserved":0,"allocated":622592,"data_stored":300132,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17326,"internal_metadata":27573330},"hb_peers":[2,3,4,6,7,8,9],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":43,"seq":184683593745,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":27560,"kb_used_data":600,"kb_used_omap":27,"kb_used_meta":26916,"kb_avail":104830040,"statfs":{"total":107374182400,"available":107345960960,"internally_reserved":0,"allocated":614400,"data_stored":291677,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":27742,"internal_metadata":27562914},"hb_peers":[2,3,5,6,7,8,9],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":43,"seq":184683593746,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":27556,"kb_used_data":596,"kb_used_omap":26,"kb_used_meta":26917,"kb_avail":104830044,"statfs":{"total":107374182400,"available":107345965056,"internally_reserved":0,"allocated":610304,"data_stored":291658,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":27088,"internal_metadata":27563568},"hb_peers":[2,4,5,6,7,8,9],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":55,"seq":236223201297,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":27560,"kb_used_data":600,"kb_used_omap":23,"kb_used_meta":26920,"kb_avail":104830040,"statfs":{"total":107374182400,"available":107345960960,"internally_reserved":0,"allocated":614400,"data_stored":291677,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":24487,"internal_metadata":27566169},"hb_peers":[2,3,4,5,6,7,8,9],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":55,"seq":236223201297,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":27556,"kb_used_data":596,"kb_used_omap":25,"kb_used_meta":26918,"kb_avail":104830044,"statfs":{"total":107374182400,"available":107345965056,"internally_reserved":0,"allocated":610304,"data_stored":291658,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":26440,"internal_metadata":27564216},"hb_peers":[2,3,4,5,6,7,8,9],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-25T12:02:18.993 DEBUG:teuthology.orchestra.run.trial117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-25T12:02:19.147 INFO:teuthology.orchestra.run.trial117.stdout: 2026-02-25T12:02:19.147 INFO:teuthology.orchestra.run.trial117.stderr:dumped all 2026-02-25T12:02:19.158 INFO:teuthology.orchestra.run.trial117.stdout:{"pg_ready":true,"pg_map":{"version":93,"stamp":"2026-02-25T12:02:18.902998+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":14,"acting":14,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":10,"num_per_pool_osds":10,"num_per_pool_omap_osds":8,"kb":1048576000,"kb_used":275376,"kb_used_data":5776,"kb_used_omap":239,"kb_used_meta":269200,"kb_avail":1048300624,"statfs":{"total":1073741824000,"available":1073459838976,"internally_reserved":0,"allocated":5914624,"data_stored":2780746,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":244864,"internal_metadata":275661696},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.005419"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":69,"reported_epoch":54,"state":"active+clean","last_fresh":"2026-02-25T12:02:16.933086+0000","last_change":"2026-02-25T12:02:06.883264+0000","last_active":"2026-02-25T12:02:16.933086+0000","last_peered":"2026-02-25T12:02:16.933086+0000","last_clean":"2026-02-25T12:02:16.933086+0000","last_became_active":"2026-02-25T12:02:06.883022+0000","last_became_peered":"2026-02-25T12:02:06.883022+0000","last_unstale":"2026-02-25T12:02:16.933086+0000","last_undegraded":"2026-02-25T12:02:16.933086+0000","last_fullsized":"2026-02-25T12:02:16.933086+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-25T12:01:08.885417+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-25T12:01:08.885417+0000","last_clean_scrub_stamp":"2026-02-25T12:01:08.885417+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-26T22:09:05.780239+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00026608399999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":87,"reported_epoch":54,"state":"active+clean","last_fresh":"2026-02-25T12:02:16.933144+0000","last_change":"2026-02-25T12:02:06.883262+0000","last_active":"2026-02-25T12:02:16.933144+0000","last_peered":"2026-02-25T12:02:16.933144+0000","last_clean":"2026-02-25T12:02:16.933144+0000","last_became_active":"2026-02-25T12:02:06.883003+0000","last_became_peered":"2026-02-25T12:02:06.883003+0000","last_unstale":"2026-02-25T12:02:16.933144+0000","last_undegraded":"2026-02-25T12:02:16.933144+0000","last_fullsized":"2026-02-25T12:02:16.933144+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-25T12:01:08.885417+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-25T12:01:08.885417+0000","last_clean_scrub_stamp":"2026-02-25T12:01:08.885417+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-26T15:40:20.663132+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00030945499999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":86,"reported_epoch":50,"state":"active+clean","last_fresh":"2026-02-25T12:02:12.933109+0000","last_change":"2026-02-25T12:02:09.321129+0000","last_active":"2026-02-25T12:02:12.933109+0000","last_peered":"2026-02-25T12:02:12.933109+0000","last_clean":"2026-02-25T12:02:12.933109+0000","last_became_active":"2026-02-25T12:02:09.320974+0000","last_became_peered":"2026-02-25T12:02:09.320974+0000","last_unstale":"2026-02-25T12:02:12.933109+0000","last_undegraded":"2026-02-25T12:02:12.933109+0000","last_fullsized":"2026-02-25T12:02:12.933109+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-25T12:01:08.885417+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-25T12:01:08.885417+0000","last_clean_scrub_stamp":"2026-02-25T12:01:08.885417+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-26T20:04:59.198096+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00017141899999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0],"acting":[9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":54,"reported_epoch":54,"state":"active+undersized+wait","last_fresh":"2026-02-25T12:02:16.933130+0000","last_change":"2026-02-25T12:02:15.902089+0000","last_active":"2026-02-25T12:02:16.933130+0000","last_peered":"2026-02-25T12:02:16.933130+0000","last_clean":"2026-02-25T12:02:02.711424+0000","last_became_active":"2026-02-25T12:02:15.902089+0000","last_became_peered":"2026-02-25T12:02:15.902089+0000","last_unstale":"2026-02-25T12:02:16.933130+0000","last_undegraded":"2026-02-25T12:02:16.933130+0000","last_fullsized":"2026-02-25T12:02:15.901707+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-25T12:01:08.885417+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-25T12:01:08.885417+0000","last_clean_scrub_stamp":"2026-02-25T12:01:08.885417+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-26T23:11:50.865351+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6],"acting":[6],"avail_no_missing":["6"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.3","version":"18'1","reported_seq":83,"reported_epoch":54,"state":"active+undersized+degraded+wait","last_fresh":"2026-02-25T12:02:16.933084+0000","last_change":"2026-02-25T12:02:15.902037+0000","last_active":"2026-02-25T12:02:16.933084+0000","last_peered":"2026-02-25T12:02:16.933084+0000","last_clean":"2026-02-25T12:02:13.894638+0000","last_became_active":"2026-02-25T12:02:15.902037+0000","last_became_peered":"2026-02-25T12:02:15.902037+0000","last_unstale":"2026-02-25T12:02:16.933084+0000","last_undegraded":"2026-02-25T12:02:15.901816+0000","last_fullsized":"2026-02-25T12:02:15.901757+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-25T12:01:08.885417+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-25T12:01:08.885417+0000","last_clean_scrub_stamp":"2026-02-25T12:01:08.885417+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-26T17:32:07.261620+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[{"shards":"3","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"20'2","reported_seq":88,"reported_epoch":54,"state":"peering","last_fresh":"2026-02-25T12:02:16.933285+0000","last_change":"2026-02-25T12:02:18.346734+0000","last_active":"2026-02-25T12:02:18.346734+0000","last_peered":"2026-02-25T12:02:16.933285+0000","last_clean":"2026-02-25T12:02:15.360485+0000","last_became_active":"2026-02-25T12:02:15.902237+0000","last_became_peered":"2026-02-25T12:02:15.902237+0000","last_unstale":"2026-02-25T12:02:16.933285+0000","last_undegraded":"2026-02-25T12:02:15.901810+0000","last_fullsized":"2026-02-25T12:02:15.901745+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-25T12:01:08.885417+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-25T12:01:08.885417+0000","last_clean_scrub_stamp":"2026-02-25T12:01:08.885417+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062137499999999997,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":63,"reported_epoch":50,"state":"active+clean","last_fresh":"2026-02-25T12:02:12.894142+0000","last_change":"2026-02-25T12:02:06.883333+0000","last_active":"2026-02-25T12:02:12.894142+0000","last_peered":"2026-02-25T12:02:12.894142+0000","last_clean":"2026-02-25T12:02:12.894142+0000","last_became_active":"2026-02-25T12:02:06.883131+0000","last_became_peered":"2026-02-25T12:02:06.883131+0000","last_unstale":"2026-02-25T12:02:12.894142+0000","last_undegraded":"2026-02-25T12:02:12.894142+0000","last_fullsized":"2026-02-25T12:02:12.894142+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-25T12:01:08.885417+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-25T12:01:08.885417+0000","last_clean_scrub_stamp":"2026-02-25T12:01:08.885417+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-26T17:10:55.006863+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00015265299999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":84,"reported_epoch":55,"state":"active+clean","last_fresh":"2026-02-25T12:02:18.563160+0000","last_change":"2026-02-25T12:02:07.090070+0000","last_active":"2026-02-25T12:02:18.563160+0000","last_peered":"2026-02-25T12:02:18.563160+0000","last_clean":"2026-02-25T12:02:18.563160+0000","last_became_active":"2026-02-25T12:02:07.089906+0000","last_became_peered":"2026-02-25T12:02:07.089906+0000","last_unstale":"2026-02-25T12:02:18.563160+0000","last_undegraded":"2026-02-25T12:02:18.563160+0000","last_fullsized":"2026-02-25T12:02:18.563160+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-25T12:01:08.885417+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-25T12:01:08.885417+0000","last_clean_scrub_stamp":"2026-02-25T12:01:08.885417+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-26T23:02:05.903788+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9],"acting":[5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":14,"acting":14,"num_store_stats":9}],"osd_stats":[{"osd":9,"up_from":37,"seq":158913789968,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":27508,"kb_used_data":548,"kb_used_omap":24,"kb_used_meta":26919,"kb_avail":104830092,"statfs":{"total":107374182400,"available":107346014208,"internally_reserved":0,"allocated":561152,"data_stored":258172,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":25143,"internal_metadata":27565513},"hb_peers":[0,1,2,3,4,5,6,7,8],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":43,"seq":184683593745,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":104857600,"kb_used":27508,"kb_used_data":548,"kb_used_omap":22,"kb_used_meta":26921,"kb_avail":104830092,"statfs":{"total":107374182400,"available":107346014208,"internally_reserved":0,"allocated":561152,"data_stored":258172,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":23185,"internal_metadata":27567471},"hb_peers":[0,1,2,3,4,5,6,7,9],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":43,"seq":184683593744,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":27508,"kb_used_data":548,"kb_used_omap":24,"kb_used_meta":26919,"kb_avail":104830092,"statfs":{"total":107374182400,"available":107346014208,"internally_reserved":0,"allocated":561152,"data_stored":258172,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":25132,"internal_metadata":27565524},"hb_peers":[0,1,2,3,4,5,6,8,9],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":43,"seq":184683593744,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":104857600,"kb_used":27496,"kb_used_data":536,"kb_used_omap":22,"kb_used_meta":26921,"kb_avail":104830104,"statfs":{"total":107374182400,"available":107346026496,"internally_reserved":0,"allocated":548864,"data_stored":247770,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":23185,"internal_metadata":27567471},"hb_peers":[0,1,3,4,5,6,7,8,9],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":43,"seq":184683593745,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":27556,"kb_used_data":596,"kb_used_omap":24,"kb_used_meta":26919,"kb_avail":104830044,"statfs":{"total":107374182400,"available":107345965056,"internally_reserved":0,"allocated":610304,"data_stored":291658,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":25136,"internal_metadata":27565520},"hb_peers":[2,3,4,5,7,8,9],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":43,"seq":184683593746,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":27568,"kb_used_data":608,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":104830032,"statfs":{"total":107374182400,"available":107345952768,"internally_reserved":0,"allocated":622592,"data_stored":300132,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17326,"internal_metadata":27573330},"hb_peers":[2,3,4,6,7,8,9],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":43,"seq":184683593745,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":27560,"kb_used_data":600,"kb_used_omap":27,"kb_used_meta":26916,"kb_avail":104830040,"statfs":{"total":107374182400,"available":107345960960,"internally_reserved":0,"allocated":614400,"data_stored":291677,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":27742,"internal_metadata":27562914},"hb_peers":[2,3,5,6,7,8,9],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":43,"seq":184683593746,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":27556,"kb_used_data":596,"kb_used_omap":26,"kb_used_meta":26917,"kb_avail":104830044,"statfs":{"total":107374182400,"available":107345965056,"internally_reserved":0,"allocated":610304,"data_stored":291658,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":27088,"internal_metadata":27563568},"hb_peers":[2,4,5,6,7,8,9],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":55,"seq":236223201297,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":27560,"kb_used_data":600,"kb_used_omap":23,"kb_used_meta":26920,"kb_avail":104830040,"statfs":{"total":107374182400,"available":107345960960,"internally_reserved":0,"allocated":614400,"data_stored":291677,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":24487,"internal_metadata":27566169},"hb_peers":[2,3,4,5,6,7,8,9],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":55,"seq":236223201297,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":27556,"kb_used_data":596,"kb_used_omap":25,"kb_used_meta":26918,"kb_avail":104830044,"statfs":{"total":107374182400,"available":107345965056,"internally_reserved":0,"allocated":610304,"data_stored":291658,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":26440,"internal_metadata":27564216},"hb_peers":[2,3,4,5,6,7,8,9],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-25T12:02:19.159 INFO:tasks.ceph.ceph_manager.ceph:PG 1.4 is not active+clean 2026-02-25T12:02:19.159 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '1.4', 'version': "0'0", 'reported_seq': 54, 'reported_epoch': 54, 'state': 'active+undersized+wait', 'last_fresh': '2026-02-25T12:02:16.933130+0000', 'last_change': '2026-02-25T12:02:15.902089+0000', 'last_active': '2026-02-25T12:02:16.933130+0000', 'last_peered': '2026-02-25T12:02:16.933130+0000', 'last_clean': '2026-02-25T12:02:02.711424+0000', 'last_became_active': '2026-02-25T12:02:15.902089+0000', 'last_became_peered': '2026-02-25T12:02:15.902089+0000', 'last_unstale': '2026-02-25T12:02:16.933130+0000', 'last_undegraded': '2026-02-25T12:02:16.933130+0000', 'last_fullsized': '2026-02-25T12:02:15.901707+0000', 'mapping_epoch': 52, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 17, 'last_epoch_clean': 44, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2026-02-25T12:01:08.885417+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2026-02-25T12:01:08.885417+0000', 'last_clean_scrub_stamp': '2026-02-25T12:01:08.885417+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2026-02-26T23:11:50.865351+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6], 'acting': [6], 'avail_no_missing': ['6'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2026-02-25T12:02:19.159 INFO:tasks.ceph.ceph_manager.ceph:PG 1.3 is not active+clean 2026-02-25T12:02:19.159 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '1.3', 'version': "18'1", 'reported_seq': 83, 'reported_epoch': 54, 'state': 'active+undersized+degraded+wait', 'last_fresh': '2026-02-25T12:02:16.933084+0000', 'last_change': '2026-02-25T12:02:15.902037+0000', 'last_active': '2026-02-25T12:02:16.933084+0000', 'last_peered': '2026-02-25T12:02:16.933084+0000', 'last_clean': '2026-02-25T12:02:13.894638+0000', 'last_became_active': '2026-02-25T12:02:15.902037+0000', 'last_became_peered': '2026-02-25T12:02:15.902037+0000', 'last_unstale': '2026-02-25T12:02:16.933084+0000', 'last_undegraded': '2026-02-25T12:02:15.901816+0000', 'last_fullsized': '2026-02-25T12:02:15.901757+0000', 'mapping_epoch': 52, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 17, 'last_epoch_clean': 44, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2026-02-25T12:01:08.885417+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2026-02-25T12:01:08.885417+0000', 'last_clean_scrub_stamp': '2026-02-25T12:01:08.885417+0000', 'objects_scrubbed': 0, 'log_size': 1, 'log_dups_size': 0, 'ondisk_log_size': 1, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2026-02-26T17:32:07.261620+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3], 'acting': [3], 'avail_no_missing': ['3'], 'object_location_counts': [{'shards': '3', 'objects': 1}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-02-25T12:02:19.159 INFO:tasks.ceph.ceph_manager.ceph:PG 1.2 is not active+clean 2026-02-25T12:02:19.159 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '1.2', 'version': "20'2", 'reported_seq': 88, 'reported_epoch': 54, 'state': 'peering', 'last_fresh': '2026-02-25T12:02:16.933285+0000', 'last_change': '2026-02-25T12:02:18.346734+0000', 'last_active': '2026-02-25T12:02:18.346734+0000', 'last_peered': '2026-02-25T12:02:16.933285+0000', 'last_clean': '2026-02-25T12:02:15.360485+0000', 'last_became_active': '2026-02-25T12:02:15.902237+0000', 'last_became_peered': '2026-02-25T12:02:15.902237+0000', 'last_unstale': '2026-02-25T12:02:16.933285+0000', 'last_undegraded': '2026-02-25T12:02:15.901810+0000', 'last_fullsized': '2026-02-25T12:02:15.901745+0000', 'mapping_epoch': 55, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 17, 'last_epoch_clean': 44, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2026-02-25T12:01:08.885417+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2026-02-25T12:01:08.885417+0000', 'last_clean_scrub_stamp': '2026-02-25T12:01:08.885417+0000', 'objects_scrubbed': 0, 'log_size': 2, 'log_dups_size': 0, 'ondisk_log_size': 2, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.000621375, 'stat_sum': {'num_bytes': 19, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2, 'num_write_kb': 2, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 19, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 1], 'acting': [4, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [1], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2026-02-25T12:02:19.159 DEBUG:teuthology.orchestra.run.trial117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2026-02-25T12:02:19.346 INFO:teuthology.orchestra.run.trial117.stdout: 2026-02-25T12:02:19.346 INFO:teuthology.orchestra.run.trial117.stdout:{"fsid":"43e63318-add1-4399-a1fc-80188594353b","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"noout flag(s) set","count":5},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 1/4 objects degraded (25.000%), 1 pg degraded","count":1},"muted":false}},"mutes":[]},"election_epoch":3,"quorum":[0],"quorum_names":["a"],"quorum_age":81,"monmap":{"epoch":1,"min_mon_release_name":"tentacle","num_mons":1},"osdmap":{"epoch":56,"num_osds":10,"num_up_osds":10,"osd_up_since":1772020937,"num_in_osds":10,"osd_in_since":1772020859,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":6},{"state_name":"active+undersized+wait","count":1},{"state_name":"active+undersized+degraded+wait","count":1}],"num_pgs":8,"num_pools":1,"num_objects":2,"data_bytes":19,"bytes_used":281862144,"bytes_avail":1073459961856,"bytes_total":1073741824000,"degraded_objects":1,"degraded_total":4,"degraded_ratio":0.25},"fsmap":{"epoch":1,"btime":"2026-02-25T12:00:57:865346+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs"],"services":{}},"servicemap":{"epoch":3,"modified":"2026-02-25T12:02:04.733030+0000","services":{}},"progress_events":{}} 2026-02-25T12:02:19.357 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2026-02-25T12:02:19.357 DEBUG:teuthology.orchestra.run.trial117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-25T12:02:19.524 INFO:teuthology.orchestra.run.trial117.stdout: 2026-02-25T12:02:19.524 INFO:teuthology.orchestra.run.trial117.stderr:dumped all 2026-02-25T12:02:19.536 INFO:teuthology.orchestra.run.trial117.stdout:{"pg_ready":true,"pg_map":{"version":93,"stamp":"2026-02-25T12:02:18.902998+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":14,"acting":14,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":10,"num_per_pool_osds":10,"num_per_pool_omap_osds":8,"kb":1048576000,"kb_used":275376,"kb_used_data":5776,"kb_used_omap":239,"kb_used_meta":269200,"kb_avail":1048300624,"statfs":{"total":1073741824000,"available":1073459838976,"internally_reserved":0,"allocated":5914624,"data_stored":2780746,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":244864,"internal_metadata":275661696},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.005419"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":69,"reported_epoch":54,"state":"active+clean","last_fresh":"2026-02-25T12:02:16.933086+0000","last_change":"2026-02-25T12:02:06.883264+0000","last_active":"2026-02-25T12:02:16.933086+0000","last_peered":"2026-02-25T12:02:16.933086+0000","last_clean":"2026-02-25T12:02:16.933086+0000","last_became_active":"2026-02-25T12:02:06.883022+0000","last_became_peered":"2026-02-25T12:02:06.883022+0000","last_unstale":"2026-02-25T12:02:16.933086+0000","last_undegraded":"2026-02-25T12:02:16.933086+0000","last_fullsized":"2026-02-25T12:02:16.933086+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-25T12:01:08.885417+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-25T12:01:08.885417+0000","last_clean_scrub_stamp":"2026-02-25T12:01:08.885417+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-26T22:09:05.780239+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00026608399999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":87,"reported_epoch":54,"state":"active+clean","last_fresh":"2026-02-25T12:02:16.933144+0000","last_change":"2026-02-25T12:02:06.883262+0000","last_active":"2026-02-25T12:02:16.933144+0000","last_peered":"2026-02-25T12:02:16.933144+0000","last_clean":"2026-02-25T12:02:16.933144+0000","last_became_active":"2026-02-25T12:02:06.883003+0000","last_became_peered":"2026-02-25T12:02:06.883003+0000","last_unstale":"2026-02-25T12:02:16.933144+0000","last_undegraded":"2026-02-25T12:02:16.933144+0000","last_fullsized":"2026-02-25T12:02:16.933144+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-25T12:01:08.885417+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-25T12:01:08.885417+0000","last_clean_scrub_stamp":"2026-02-25T12:01:08.885417+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-26T15:40:20.663132+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00030945499999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":86,"reported_epoch":50,"state":"active+clean","last_fresh":"2026-02-25T12:02:12.933109+0000","last_change":"2026-02-25T12:02:09.321129+0000","last_active":"2026-02-25T12:02:12.933109+0000","last_peered":"2026-02-25T12:02:12.933109+0000","last_clean":"2026-02-25T12:02:12.933109+0000","last_became_active":"2026-02-25T12:02:09.320974+0000","last_became_peered":"2026-02-25T12:02:09.320974+0000","last_unstale":"2026-02-25T12:02:12.933109+0000","last_undegraded":"2026-02-25T12:02:12.933109+0000","last_fullsized":"2026-02-25T12:02:12.933109+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-25T12:01:08.885417+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-25T12:01:08.885417+0000","last_clean_scrub_stamp":"2026-02-25T12:01:08.885417+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-26T20:04:59.198096+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00017141899999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0],"acting":[9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":54,"reported_epoch":54,"state":"active+undersized+wait","last_fresh":"2026-02-25T12:02:16.933130+0000","last_change":"2026-02-25T12:02:15.902089+0000","last_active":"2026-02-25T12:02:16.933130+0000","last_peered":"2026-02-25T12:02:16.933130+0000","last_clean":"2026-02-25T12:02:02.711424+0000","last_became_active":"2026-02-25T12:02:15.902089+0000","last_became_peered":"2026-02-25T12:02:15.902089+0000","last_unstale":"2026-02-25T12:02:16.933130+0000","last_undegraded":"2026-02-25T12:02:16.933130+0000","last_fullsized":"2026-02-25T12:02:15.901707+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-25T12:01:08.885417+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-25T12:01:08.885417+0000","last_clean_scrub_stamp":"2026-02-25T12:01:08.885417+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-26T23:11:50.865351+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6],"acting":[6],"avail_no_missing":["6"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.3","version":"18'1","reported_seq":83,"reported_epoch":54,"state":"active+undersized+degraded+wait","last_fresh":"2026-02-25T12:02:16.933084+0000","last_change":"2026-02-25T12:02:15.902037+0000","last_active":"2026-02-25T12:02:16.933084+0000","last_peered":"2026-02-25T12:02:16.933084+0000","last_clean":"2026-02-25T12:02:13.894638+0000","last_became_active":"2026-02-25T12:02:15.902037+0000","last_became_peered":"2026-02-25T12:02:15.902037+0000","last_unstale":"2026-02-25T12:02:16.933084+0000","last_undegraded":"2026-02-25T12:02:15.901816+0000","last_fullsized":"2026-02-25T12:02:15.901757+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-25T12:01:08.885417+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-25T12:01:08.885417+0000","last_clean_scrub_stamp":"2026-02-25T12:01:08.885417+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-26T17:32:07.261620+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[{"shards":"3","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"20'2","reported_seq":88,"reported_epoch":54,"state":"peering","last_fresh":"2026-02-25T12:02:16.933285+0000","last_change":"2026-02-25T12:02:18.346734+0000","last_active":"2026-02-25T12:02:18.346734+0000","last_peered":"2026-02-25T12:02:16.933285+0000","last_clean":"2026-02-25T12:02:15.360485+0000","last_became_active":"2026-02-25T12:02:15.902237+0000","last_became_peered":"2026-02-25T12:02:15.902237+0000","last_unstale":"2026-02-25T12:02:16.933285+0000","last_undegraded":"2026-02-25T12:02:15.901810+0000","last_fullsized":"2026-02-25T12:02:15.901745+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-25T12:01:08.885417+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-25T12:01:08.885417+0000","last_clean_scrub_stamp":"2026-02-25T12:01:08.885417+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062137499999999997,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":63,"reported_epoch":50,"state":"active+clean","last_fresh":"2026-02-25T12:02:12.894142+0000","last_change":"2026-02-25T12:02:06.883333+0000","last_active":"2026-02-25T12:02:12.894142+0000","last_peered":"2026-02-25T12:02:12.894142+0000","last_clean":"2026-02-25T12:02:12.894142+0000","last_became_active":"2026-02-25T12:02:06.883131+0000","last_became_peered":"2026-02-25T12:02:06.883131+0000","last_unstale":"2026-02-25T12:02:12.894142+0000","last_undegraded":"2026-02-25T12:02:12.894142+0000","last_fullsized":"2026-02-25T12:02:12.894142+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-25T12:01:08.885417+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-25T12:01:08.885417+0000","last_clean_scrub_stamp":"2026-02-25T12:01:08.885417+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-26T17:10:55.006863+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00015265299999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":84,"reported_epoch":55,"state":"active+clean","last_fresh":"2026-02-25T12:02:18.563160+0000","last_change":"2026-02-25T12:02:07.090070+0000","last_active":"2026-02-25T12:02:18.563160+0000","last_peered":"2026-02-25T12:02:18.563160+0000","last_clean":"2026-02-25T12:02:18.563160+0000","last_became_active":"2026-02-25T12:02:07.089906+0000","last_became_peered":"2026-02-25T12:02:07.089906+0000","last_unstale":"2026-02-25T12:02:18.563160+0000","last_undegraded":"2026-02-25T12:02:18.563160+0000","last_fullsized":"2026-02-25T12:02:18.563160+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-25T12:01:08.885417+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-25T12:01:08.885417+0000","last_clean_scrub_stamp":"2026-02-25T12:01:08.885417+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-26T23:02:05.903788+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9],"acting":[5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":14,"acting":14,"num_store_stats":9}],"osd_stats":[{"osd":9,"up_from":37,"seq":158913789968,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":27508,"kb_used_data":548,"kb_used_omap":24,"kb_used_meta":26919,"kb_avail":104830092,"statfs":{"total":107374182400,"available":107346014208,"internally_reserved":0,"allocated":561152,"data_stored":258172,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":25143,"internal_metadata":27565513},"hb_peers":[0,1,2,3,4,5,6,7,8],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":43,"seq":184683593745,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":104857600,"kb_used":27508,"kb_used_data":548,"kb_used_omap":22,"kb_used_meta":26921,"kb_avail":104830092,"statfs":{"total":107374182400,"available":107346014208,"internally_reserved":0,"allocated":561152,"data_stored":258172,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":23185,"internal_metadata":27567471},"hb_peers":[0,1,2,3,4,5,6,7,9],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":43,"seq":184683593744,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":27508,"kb_used_data":548,"kb_used_omap":24,"kb_used_meta":26919,"kb_avail":104830092,"statfs":{"total":107374182400,"available":107346014208,"internally_reserved":0,"allocated":561152,"data_stored":258172,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":25132,"internal_metadata":27565524},"hb_peers":[0,1,2,3,4,5,6,8,9],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":43,"seq":184683593744,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":104857600,"kb_used":27496,"kb_used_data":536,"kb_used_omap":22,"kb_used_meta":26921,"kb_avail":104830104,"statfs":{"total":107374182400,"available":107346026496,"internally_reserved":0,"allocated":548864,"data_stored":247770,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":23185,"internal_metadata":27567471},"hb_peers":[0,1,3,4,5,6,7,8,9],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":43,"seq":184683593745,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":27556,"kb_used_data":596,"kb_used_omap":24,"kb_used_meta":26919,"kb_avail":104830044,"statfs":{"total":107374182400,"available":107345965056,"internally_reserved":0,"allocated":610304,"data_stored":291658,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":25136,"internal_metadata":27565520},"hb_peers":[2,3,4,5,7,8,9],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":43,"seq":184683593746,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":27568,"kb_used_data":608,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":104830032,"statfs":{"total":107374182400,"available":107345952768,"internally_reserved":0,"allocated":622592,"data_stored":300132,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17326,"internal_metadata":27573330},"hb_peers":[2,3,4,6,7,8,9],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":43,"seq":184683593745,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":27560,"kb_used_data":600,"kb_used_omap":27,"kb_used_meta":26916,"kb_avail":104830040,"statfs":{"total":107374182400,"available":107345960960,"internally_reserved":0,"allocated":614400,"data_stored":291677,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":27742,"internal_metadata":27562914},"hb_peers":[2,3,5,6,7,8,9],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":43,"seq":184683593746,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":27556,"kb_used_data":596,"kb_used_omap":26,"kb_used_meta":26917,"kb_avail":104830044,"statfs":{"total":107374182400,"available":107345965056,"internally_reserved":0,"allocated":610304,"data_stored":291658,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":27088,"internal_metadata":27563568},"hb_peers":[2,4,5,6,7,8,9],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":55,"seq":236223201297,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":27560,"kb_used_data":600,"kb_used_omap":23,"kb_used_meta":26920,"kb_avail":104830040,"statfs":{"total":107374182400,"available":107345960960,"internally_reserved":0,"allocated":614400,"data_stored":291677,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":24487,"internal_metadata":27566169},"hb_peers":[2,3,4,5,6,7,8,9],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":55,"seq":236223201297,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":27556,"kb_used_data":596,"kb_used_omap":25,"kb_used_meta":26918,"kb_avail":104830044,"statfs":{"total":107374182400,"available":107345965056,"internally_reserved":0,"allocated":610304,"data_stored":291658,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":26440,"internal_metadata":27564216},"hb_peers":[2,3,4,5,6,7,8,9],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-25T12:02:22.538 DEBUG:teuthology.orchestra.run.trial117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-25T12:02:22.693 INFO:teuthology.orchestra.run.trial117.stdout: 2026-02-25T12:02:22.694 INFO:teuthology.orchestra.run.trial117.stderr:dumped all 2026-02-25T12:02:22.705 INFO:teuthology.orchestra.run.trial117.stdout:{"pg_ready":true,"pg_map":{"version":94,"stamp":"2026-02-25T12:02:20.735098+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":14,"acting":14,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":10,"num_per_pool_osds":10,"num_per_pool_omap_osds":8,"kb":1048576000,"kb_used":275616,"kb_used_data":6016,"kb_used_omap":248,"kb_used_meta":269191,"kb_avail":1048300384,"statfs":{"total":1073741824000,"available":1073459593216,"internally_reserved":0,"allocated":6160384,"data_stored":2950514,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":254629,"internal_metadata":275651931},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.834965"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":69,"reported_epoch":54,"state":"active+clean","last_fresh":"2026-02-25T12:02:16.933086+0000","last_change":"2026-02-25T12:02:06.883264+0000","last_active":"2026-02-25T12:02:16.933086+0000","last_peered":"2026-02-25T12:02:16.933086+0000","last_clean":"2026-02-25T12:02:16.933086+0000","last_became_active":"2026-02-25T12:02:06.883022+0000","last_became_peered":"2026-02-25T12:02:06.883022+0000","last_unstale":"2026-02-25T12:02:16.933086+0000","last_undegraded":"2026-02-25T12:02:16.933086+0000","last_fullsized":"2026-02-25T12:02:16.933086+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-25T12:01:08.885417+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-25T12:01:08.885417+0000","last_clean_scrub_stamp":"2026-02-25T12:01:08.885417+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-26T22:09:05.780239+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00026608399999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":87,"reported_epoch":54,"state":"active+clean","last_fresh":"2026-02-25T12:02:16.933144+0000","last_change":"2026-02-25T12:02:06.883262+0000","last_active":"2026-02-25T12:02:16.933144+0000","last_peered":"2026-02-25T12:02:16.933144+0000","last_clean":"2026-02-25T12:02:16.933144+0000","last_became_active":"2026-02-25T12:02:06.883003+0000","last_became_peered":"2026-02-25T12:02:06.883003+0000","last_unstale":"2026-02-25T12:02:16.933144+0000","last_undegraded":"2026-02-25T12:02:16.933144+0000","last_fullsized":"2026-02-25T12:02:16.933144+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-25T12:01:08.885417+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-25T12:01:08.885417+0000","last_clean_scrub_stamp":"2026-02-25T12:01:08.885417+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-26T15:40:20.663132+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00030945499999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":100,"reported_epoch":56,"state":"activating","last_fresh":"2026-02-25T12:02:18.904793+0000","last_change":"2026-02-25T12:02:18.904907+0000","last_active":"2026-02-25T12:02:17.903390+0000","last_peered":"2026-02-25T12:02:17.217325+0000","last_clean":"2026-02-25T12:02:15.427768+0000","last_became_active":"2026-02-25T12:02:15.902144+0000","last_became_peered":"2026-02-25T12:02:15.902144+0000","last_unstale":"2026-02-25T12:02:18.904793+0000","last_undegraded":"2026-02-25T12:02:18.904793+0000","last_fullsized":"2026-02-25T12:02:18.904793+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-25T12:01:08.885417+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-25T12:01:08.885417+0000","last_clean_scrub_stamp":"2026-02-25T12:01:08.885417+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00017141899999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0],"acting":[9,0],"avail_no_missing":["9","0"],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":54,"reported_epoch":54,"state":"active+undersized+wait","last_fresh":"2026-02-25T12:02:16.933130+0000","last_change":"2026-02-25T12:02:15.902089+0000","last_active":"2026-02-25T12:02:16.933130+0000","last_peered":"2026-02-25T12:02:16.933130+0000","last_clean":"2026-02-25T12:02:02.711424+0000","last_became_active":"2026-02-25T12:02:15.902089+0000","last_became_peered":"2026-02-25T12:02:15.902089+0000","last_unstale":"2026-02-25T12:02:16.933130+0000","last_undegraded":"2026-02-25T12:02:16.933130+0000","last_fullsized":"2026-02-25T12:02:15.901707+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-25T12:01:08.885417+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-25T12:01:08.885417+0000","last_clean_scrub_stamp":"2026-02-25T12:01:08.885417+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-26T23:11:50.865351+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6],"acting":[6],"avail_no_missing":["6"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.3","version":"18'1","reported_seq":83,"reported_epoch":54,"state":"active+undersized+degraded+wait","last_fresh":"2026-02-25T12:02:16.933084+0000","last_change":"2026-02-25T12:02:15.902037+0000","last_active":"2026-02-25T12:02:16.933084+0000","last_peered":"2026-02-25T12:02:16.933084+0000","last_clean":"2026-02-25T12:02:13.894638+0000","last_became_active":"2026-02-25T12:02:15.902037+0000","last_became_peered":"2026-02-25T12:02:15.902037+0000","last_unstale":"2026-02-25T12:02:16.933084+0000","last_undegraded":"2026-02-25T12:02:15.901816+0000","last_fullsized":"2026-02-25T12:02:15.901757+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-25T12:01:08.885417+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-25T12:01:08.885417+0000","last_clean_scrub_stamp":"2026-02-25T12:01:08.885417+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-26T17:32:07.261620+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[{"shards":"3","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"20'2","reported_seq":88,"reported_epoch":54,"state":"peering","last_fresh":"2026-02-25T12:02:16.933285+0000","last_change":"2026-02-25T12:02:18.346734+0000","last_active":"2026-02-25T12:02:18.346734+0000","last_peered":"2026-02-25T12:02:16.933285+0000","last_clean":"2026-02-25T12:02:15.360485+0000","last_became_active":"2026-02-25T12:02:15.902237+0000","last_became_peered":"2026-02-25T12:02:15.902237+0000","last_unstale":"2026-02-25T12:02:16.933285+0000","last_undegraded":"2026-02-25T12:02:15.901810+0000","last_fullsized":"2026-02-25T12:02:15.901745+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-25T12:01:08.885417+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-25T12:01:08.885417+0000","last_clean_scrub_stamp":"2026-02-25T12:01:08.885417+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062137499999999997,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":72,"reported_epoch":55,"state":"active+clean","last_fresh":"2026-02-25T12:02:17.937145+0000","last_change":"2026-02-25T12:02:06.883333+0000","last_active":"2026-02-25T12:02:17.937145+0000","last_peered":"2026-02-25T12:02:17.937145+0000","last_clean":"2026-02-25T12:02:17.937145+0000","last_became_active":"2026-02-25T12:02:06.883131+0000","last_became_peered":"2026-02-25T12:02:06.883131+0000","last_unstale":"2026-02-25T12:02:17.937145+0000","last_undegraded":"2026-02-25T12:02:17.937145+0000","last_fullsized":"2026-02-25T12:02:17.937145+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-25T12:01:08.885417+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-25T12:01:08.885417+0000","last_clean_scrub_stamp":"2026-02-25T12:01:08.885417+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-26T17:10:55.006863+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00015265299999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":84,"reported_epoch":55,"state":"active+clean","last_fresh":"2026-02-25T12:02:18.563160+0000","last_change":"2026-02-25T12:02:07.090070+0000","last_active":"2026-02-25T12:02:18.563160+0000","last_peered":"2026-02-25T12:02:18.563160+0000","last_clean":"2026-02-25T12:02:18.563160+0000","last_became_active":"2026-02-25T12:02:07.089906+0000","last_became_peered":"2026-02-25T12:02:07.089906+0000","last_unstale":"2026-02-25T12:02:18.563160+0000","last_undegraded":"2026-02-25T12:02:18.563160+0000","last_fullsized":"2026-02-25T12:02:18.563160+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-25T12:01:08.885417+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-25T12:01:08.885417+0000","last_clean_scrub_stamp":"2026-02-25T12:01:08.885417+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-26T23:02:05.903788+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9],"acting":[5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":14,"acting":14,"num_store_stats":9}],"osd_stats":[{"osd":9,"up_from":37,"seq":158913789969,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":27568,"kb_used_data":608,"kb_used_omap":27,"kb_used_meta":26916,"kb_avail":104830032,"statfs":{"total":107374182400,"available":107345952768,"internally_reserved":0,"allocated":622592,"data_stored":300132,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":28398,"internal_metadata":27562258},"hb_peers":[0,1,2,3,4,5,6,7,8],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":43,"seq":184683593746,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":104857600,"kb_used":27556,"kb_used_data":596,"kb_used_omap":22,"kb_used_meta":26921,"kb_avail":104830044,"statfs":{"total":107374182400,"available":107345965056,"internally_reserved":0,"allocated":610304,"data_stored":291658,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":23185,"internal_metadata":27567471},"hb_peers":[2,3,4,5,6,7,9],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":43,"seq":184683593745,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":27568,"kb_used_data":608,"kb_used_omap":27,"kb_used_meta":26916,"kb_avail":104830032,"statfs":{"total":107374182400,"available":107345952768,"internally_reserved":0,"allocated":622592,"data_stored":300132,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":27736,"internal_metadata":27562920},"hb_peers":[2,3,4,5,6,8,9],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":43,"seq":184683593745,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":104857600,"kb_used":27568,"kb_used_data":608,"kb_used_omap":26,"kb_used_meta":26917,"kb_avail":104830032,"statfs":{"total":107374182400,"available":107345952768,"internally_reserved":0,"allocated":622592,"data_stored":300132,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":27091,"internal_metadata":27563565},"hb_peers":[0,1,3,4,5,6,7,8,9],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":43,"seq":184683593745,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":27556,"kb_used_data":596,"kb_used_omap":24,"kb_used_meta":26919,"kb_avail":104830044,"statfs":{"total":107374182400,"available":107345965056,"internally_reserved":0,"allocated":610304,"data_stored":291658,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":25136,"internal_metadata":27565520},"hb_peers":[2,3,4,5,7,8,9],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":43,"seq":184683593746,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":27568,"kb_used_data":608,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":104830032,"statfs":{"total":107374182400,"available":107345952768,"internally_reserved":0,"allocated":622592,"data_stored":300132,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17326,"internal_metadata":27573330},"hb_peers":[2,3,4,6,7,8,9],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":43,"seq":184683593745,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":27560,"kb_used_data":600,"kb_used_omap":27,"kb_used_meta":26916,"kb_avail":104830040,"statfs":{"total":107374182400,"available":107345960960,"internally_reserved":0,"allocated":614400,"data_stored":291677,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":27742,"internal_metadata":27562914},"hb_peers":[2,3,5,6,7,8,9],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":43,"seq":184683593746,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":27556,"kb_used_data":596,"kb_used_omap":26,"kb_used_meta":26917,"kb_avail":104830044,"statfs":{"total":107374182400,"available":107345965056,"internally_reserved":0,"allocated":610304,"data_stored":291658,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":27088,"internal_metadata":27563568},"hb_peers":[2,4,5,6,7,8,9],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":55,"seq":236223201297,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":27560,"kb_used_data":600,"kb_used_omap":23,"kb_used_meta":26920,"kb_avail":104830040,"statfs":{"total":107374182400,"available":107345960960,"internally_reserved":0,"allocated":614400,"data_stored":291677,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":24487,"internal_metadata":27566169},"hb_peers":[2,3,4,5,6,7,8,9],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":55,"seq":236223201297,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":27556,"kb_used_data":596,"kb_used_omap":25,"kb_used_meta":26918,"kb_avail":104830044,"statfs":{"total":107374182400,"available":107345965056,"internally_reserved":0,"allocated":610304,"data_stored":291658,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":26440,"internal_metadata":27564216},"hb_peers":[2,3,4,5,6,7,8,9],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-25T12:02:22.707 DEBUG:teuthology.orchestra.run.trial117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-25T12:02:22.868 INFO:teuthology.orchestra.run.trial117.stdout: 2026-02-25T12:02:22.869 INFO:teuthology.orchestra.run.trial117.stderr:dumped all 2026-02-25T12:02:22.879 INFO:teuthology.orchestra.run.trial117.stdout:{"pg_ready":true,"pg_map":{"version":95,"stamp":"2026-02-25T12:02:22.735364+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":14,"acting":14,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":10,"num_per_pool_osds":10,"num_per_pool_omap_osds":8,"kb":1048576000,"kb_used":275616,"kb_used_data":6016,"kb_used_omap":248,"kb_used_meta":269191,"kb_avail":1048300384,"statfs":{"total":1073741824000,"available":1073459593216,"internally_reserved":0,"allocated":6160384,"data_stored":2950514,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":254629,"internal_metadata":275651931},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.000766"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":69,"reported_epoch":54,"state":"active+clean","last_fresh":"2026-02-25T12:02:16.933086+0000","last_change":"2026-02-25T12:02:06.883264+0000","last_active":"2026-02-25T12:02:16.933086+0000","last_peered":"2026-02-25T12:02:16.933086+0000","last_clean":"2026-02-25T12:02:16.933086+0000","last_became_active":"2026-02-25T12:02:06.883022+0000","last_became_peered":"2026-02-25T12:02:06.883022+0000","last_unstale":"2026-02-25T12:02:16.933086+0000","last_undegraded":"2026-02-25T12:02:16.933086+0000","last_fullsized":"2026-02-25T12:02:16.933086+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-25T12:01:08.885417+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-25T12:01:08.885417+0000","last_clean_scrub_stamp":"2026-02-25T12:01:08.885417+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-26T22:09:05.780239+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00026608399999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":87,"reported_epoch":54,"state":"active+clean","last_fresh":"2026-02-25T12:02:16.933144+0000","last_change":"2026-02-25T12:02:06.883262+0000","last_active":"2026-02-25T12:02:16.933144+0000","last_peered":"2026-02-25T12:02:16.933144+0000","last_clean":"2026-02-25T12:02:16.933144+0000","last_became_active":"2026-02-25T12:02:06.883003+0000","last_became_peered":"2026-02-25T12:02:06.883003+0000","last_unstale":"2026-02-25T12:02:16.933144+0000","last_undegraded":"2026-02-25T12:02:16.933144+0000","last_fullsized":"2026-02-25T12:02:16.933144+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-25T12:01:08.885417+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-25T12:01:08.885417+0000","last_clean_scrub_stamp":"2026-02-25T12:01:08.885417+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-26T15:40:20.663132+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00030945499999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":100,"reported_epoch":56,"state":"activating","last_fresh":"2026-02-25T12:02:18.904793+0000","last_change":"2026-02-25T12:02:18.904907+0000","last_active":"2026-02-25T12:02:17.903390+0000","last_peered":"2026-02-25T12:02:17.217325+0000","last_clean":"2026-02-25T12:02:15.427768+0000","last_became_active":"2026-02-25T12:02:15.902144+0000","last_became_peered":"2026-02-25T12:02:15.902144+0000","last_unstale":"2026-02-25T12:02:18.904793+0000","last_undegraded":"2026-02-25T12:02:18.904793+0000","last_fullsized":"2026-02-25T12:02:18.904793+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-25T12:01:08.885417+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-25T12:01:08.885417+0000","last_clean_scrub_stamp":"2026-02-25T12:01:08.885417+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00017141899999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0],"acting":[9,0],"avail_no_missing":["9","0"],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":54,"reported_epoch":54,"state":"active+undersized+wait","last_fresh":"2026-02-25T12:02:16.933130+0000","last_change":"2026-02-25T12:02:15.902089+0000","last_active":"2026-02-25T12:02:16.933130+0000","last_peered":"2026-02-25T12:02:16.933130+0000","last_clean":"2026-02-25T12:02:02.711424+0000","last_became_active":"2026-02-25T12:02:15.902089+0000","last_became_peered":"2026-02-25T12:02:15.902089+0000","last_unstale":"2026-02-25T12:02:16.933130+0000","last_undegraded":"2026-02-25T12:02:16.933130+0000","last_fullsized":"2026-02-25T12:02:15.901707+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-25T12:01:08.885417+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-25T12:01:08.885417+0000","last_clean_scrub_stamp":"2026-02-25T12:01:08.885417+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-26T23:11:50.865351+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6],"acting":[6],"avail_no_missing":["6"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.3","version":"18'1","reported_seq":83,"reported_epoch":54,"state":"active+undersized+degraded+wait","last_fresh":"2026-02-25T12:02:16.933084+0000","last_change":"2026-02-25T12:02:15.902037+0000","last_active":"2026-02-25T12:02:16.933084+0000","last_peered":"2026-02-25T12:02:16.933084+0000","last_clean":"2026-02-25T12:02:13.894638+0000","last_became_active":"2026-02-25T12:02:15.902037+0000","last_became_peered":"2026-02-25T12:02:15.902037+0000","last_unstale":"2026-02-25T12:02:16.933084+0000","last_undegraded":"2026-02-25T12:02:15.901816+0000","last_fullsized":"2026-02-25T12:02:15.901757+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-25T12:01:08.885417+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-25T12:01:08.885417+0000","last_clean_scrub_stamp":"2026-02-25T12:01:08.885417+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-26T17:32:07.261620+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[{"shards":"3","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"20'2","reported_seq":88,"reported_epoch":54,"state":"peering","last_fresh":"2026-02-25T12:02:16.933285+0000","last_change":"2026-02-25T12:02:18.346734+0000","last_active":"2026-02-25T12:02:18.346734+0000","last_peered":"2026-02-25T12:02:16.933285+0000","last_clean":"2026-02-25T12:02:15.360485+0000","last_became_active":"2026-02-25T12:02:15.902237+0000","last_became_peered":"2026-02-25T12:02:15.902237+0000","last_unstale":"2026-02-25T12:02:16.933285+0000","last_undegraded":"2026-02-25T12:02:15.901810+0000","last_fullsized":"2026-02-25T12:02:15.901745+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-25T12:01:08.885417+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-25T12:01:08.885417+0000","last_clean_scrub_stamp":"2026-02-25T12:01:08.885417+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062137499999999997,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":72,"reported_epoch":55,"state":"active+clean","last_fresh":"2026-02-25T12:02:17.937145+0000","last_change":"2026-02-25T12:02:06.883333+0000","last_active":"2026-02-25T12:02:17.937145+0000","last_peered":"2026-02-25T12:02:17.937145+0000","last_clean":"2026-02-25T12:02:17.937145+0000","last_became_active":"2026-02-25T12:02:06.883131+0000","last_became_peered":"2026-02-25T12:02:06.883131+0000","last_unstale":"2026-02-25T12:02:17.937145+0000","last_undegraded":"2026-02-25T12:02:17.937145+0000","last_fullsized":"2026-02-25T12:02:17.937145+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-25T12:01:08.885417+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-25T12:01:08.885417+0000","last_clean_scrub_stamp":"2026-02-25T12:01:08.885417+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-26T17:10:55.006863+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00015265299999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":84,"reported_epoch":55,"state":"active+clean","last_fresh":"2026-02-25T12:02:18.563160+0000","last_change":"2026-02-25T12:02:07.090070+0000","last_active":"2026-02-25T12:02:18.563160+0000","last_peered":"2026-02-25T12:02:18.563160+0000","last_clean":"2026-02-25T12:02:18.563160+0000","last_became_active":"2026-02-25T12:02:07.089906+0000","last_became_peered":"2026-02-25T12:02:07.089906+0000","last_unstale":"2026-02-25T12:02:18.563160+0000","last_undegraded":"2026-02-25T12:02:18.563160+0000","last_fullsized":"2026-02-25T12:02:18.563160+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-25T12:01:08.885417+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-25T12:01:08.885417+0000","last_clean_scrub_stamp":"2026-02-25T12:01:08.885417+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-26T23:02:05.903788+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9],"acting":[5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":14,"acting":14,"num_store_stats":9}],"osd_stats":[{"osd":9,"up_from":37,"seq":158913789969,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":27568,"kb_used_data":608,"kb_used_omap":27,"kb_used_meta":26916,"kb_avail":104830032,"statfs":{"total":107374182400,"available":107345952768,"internally_reserved":0,"allocated":622592,"data_stored":300132,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":28398,"internal_metadata":27562258},"hb_peers":[0,1,2,3,4,5,6,7,8],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":43,"seq":184683593746,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":104857600,"kb_used":27556,"kb_used_data":596,"kb_used_omap":22,"kb_used_meta":26921,"kb_avail":104830044,"statfs":{"total":107374182400,"available":107345965056,"internally_reserved":0,"allocated":610304,"data_stored":291658,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":23185,"internal_metadata":27567471},"hb_peers":[2,3,4,5,6,7,9],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":43,"seq":184683593745,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":27568,"kb_used_data":608,"kb_used_omap":27,"kb_used_meta":26916,"kb_avail":104830032,"statfs":{"total":107374182400,"available":107345952768,"internally_reserved":0,"allocated":622592,"data_stored":300132,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":27736,"internal_metadata":27562920},"hb_peers":[2,3,4,5,6,8,9],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":43,"seq":184683593745,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":104857600,"kb_used":27568,"kb_used_data":608,"kb_used_omap":26,"kb_used_meta":26917,"kb_avail":104830032,"statfs":{"total":107374182400,"available":107345952768,"internally_reserved":0,"allocated":622592,"data_stored":300132,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":27091,"internal_metadata":27563565},"hb_peers":[0,1,3,4,5,6,7,8,9],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":43,"seq":184683593745,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":27556,"kb_used_data":596,"kb_used_omap":24,"kb_used_meta":26919,"kb_avail":104830044,"statfs":{"total":107374182400,"available":107345965056,"internally_reserved":0,"allocated":610304,"data_stored":291658,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":25136,"internal_metadata":27565520},"hb_peers":[2,3,4,5,7,8,9],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":43,"seq":184683593746,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":27568,"kb_used_data":608,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":104830032,"statfs":{"total":107374182400,"available":107345952768,"internally_reserved":0,"allocated":622592,"data_stored":300132,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17326,"internal_metadata":27573330},"hb_peers":[2,3,4,6,7,8,9],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":43,"seq":184683593745,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":27560,"kb_used_data":600,"kb_used_omap":27,"kb_used_meta":26916,"kb_avail":104830040,"statfs":{"total":107374182400,"available":107345960960,"internally_reserved":0,"allocated":614400,"data_stored":291677,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":27742,"internal_metadata":27562914},"hb_peers":[2,3,5,6,7,8,9],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":43,"seq":184683593746,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":27556,"kb_used_data":596,"kb_used_omap":26,"kb_used_meta":26917,"kb_avail":104830044,"statfs":{"total":107374182400,"available":107345965056,"internally_reserved":0,"allocated":610304,"data_stored":291658,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":27088,"internal_metadata":27563568},"hb_peers":[2,4,5,6,7,8,9],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":55,"seq":236223201297,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":27560,"kb_used_data":600,"kb_used_omap":23,"kb_used_meta":26920,"kb_avail":104830040,"statfs":{"total":107374182400,"available":107345960960,"internally_reserved":0,"allocated":614400,"data_stored":291677,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":24487,"internal_metadata":27566169},"hb_peers":[2,3,4,5,6,7,8,9],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":55,"seq":236223201297,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":27556,"kb_used_data":596,"kb_used_omap":25,"kb_used_meta":26918,"kb_avail":104830044,"statfs":{"total":107374182400,"available":107345965056,"internally_reserved":0,"allocated":610304,"data_stored":291658,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":26440,"internal_metadata":27564216},"hb_peers":[2,3,4,5,6,7,8,9],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-25T12:02:22.880 INFO:tasks.ceph.ceph_manager.ceph:PG 1.5 is not active+clean 2026-02-25T12:02:22.880 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '1.5', 'version': "0'0", 'reported_seq': 100, 'reported_epoch': 56, 'state': 'activating', 'last_fresh': '2026-02-25T12:02:18.904793+0000', 'last_change': '2026-02-25T12:02:18.904907+0000', 'last_active': '2026-02-25T12:02:17.903390+0000', 'last_peered': '2026-02-25T12:02:17.217325+0000', 'last_clean': '2026-02-25T12:02:15.427768+0000', 'last_became_active': '2026-02-25T12:02:15.902144+0000', 'last_became_peered': '2026-02-25T12:02:15.902144+0000', 'last_unstale': '2026-02-25T12:02:18.904793+0000', 'last_undegraded': '2026-02-25T12:02:18.904793+0000', 'last_fullsized': '2026-02-25T12:02:18.904793+0000', 'mapping_epoch': 55, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 17, 'last_epoch_clean': 46, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2026-02-25T12:01:08.885417+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2026-02-25T12:01:08.885417+0000', 'last_clean_scrub_stamp': '2026-02-25T12:01:08.885417+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.000171419, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [9, 0], 'acting': [9, 0], 'avail_no_missing': ['9', '0'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 9, 'acting_primary': 9, 'purged_snaps': []} 2026-02-25T12:02:22.880 INFO:tasks.ceph.ceph_manager.ceph:PG 1.4 is not active+clean 2026-02-25T12:02:22.880 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '1.4', 'version': "0'0", 'reported_seq': 54, 'reported_epoch': 54, 'state': 'active+undersized+wait', 'last_fresh': '2026-02-25T12:02:16.933130+0000', 'last_change': '2026-02-25T12:02:15.902089+0000', 'last_active': '2026-02-25T12:02:16.933130+0000', 'last_peered': '2026-02-25T12:02:16.933130+0000', 'last_clean': '2026-02-25T12:02:02.711424+0000', 'last_became_active': '2026-02-25T12:02:15.902089+0000', 'last_became_peered': '2026-02-25T12:02:15.902089+0000', 'last_unstale': '2026-02-25T12:02:16.933130+0000', 'last_undegraded': '2026-02-25T12:02:16.933130+0000', 'last_fullsized': '2026-02-25T12:02:15.901707+0000', 'mapping_epoch': 52, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 17, 'last_epoch_clean': 44, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2026-02-25T12:01:08.885417+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2026-02-25T12:01:08.885417+0000', 'last_clean_scrub_stamp': '2026-02-25T12:01:08.885417+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2026-02-26T23:11:50.865351+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6], 'acting': [6], 'avail_no_missing': ['6'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2026-02-25T12:02:22.880 INFO:tasks.ceph.ceph_manager.ceph:PG 1.3 is not active+clean 2026-02-25T12:02:22.880 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '1.3', 'version': "18'1", 'reported_seq': 83, 'reported_epoch': 54, 'state': 'active+undersized+degraded+wait', 'last_fresh': '2026-02-25T12:02:16.933084+0000', 'last_change': '2026-02-25T12:02:15.902037+0000', 'last_active': '2026-02-25T12:02:16.933084+0000', 'last_peered': '2026-02-25T12:02:16.933084+0000', 'last_clean': '2026-02-25T12:02:13.894638+0000', 'last_became_active': '2026-02-25T12:02:15.902037+0000', 'last_became_peered': '2026-02-25T12:02:15.902037+0000', 'last_unstale': '2026-02-25T12:02:16.933084+0000', 'last_undegraded': '2026-02-25T12:02:15.901816+0000', 'last_fullsized': '2026-02-25T12:02:15.901757+0000', 'mapping_epoch': 52, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 17, 'last_epoch_clean': 44, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2026-02-25T12:01:08.885417+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2026-02-25T12:01:08.885417+0000', 'last_clean_scrub_stamp': '2026-02-25T12:01:08.885417+0000', 'objects_scrubbed': 0, 'log_size': 1, 'log_dups_size': 0, 'ondisk_log_size': 1, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2026-02-26T17:32:07.261620+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3], 'acting': [3], 'avail_no_missing': ['3'], 'object_location_counts': [{'shards': '3', 'objects': 1}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-02-25T12:02:22.880 INFO:tasks.ceph.ceph_manager.ceph:PG 1.2 is not active+clean 2026-02-25T12:02:22.880 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '1.2', 'version': "20'2", 'reported_seq': 88, 'reported_epoch': 54, 'state': 'peering', 'last_fresh': '2026-02-25T12:02:16.933285+0000', 'last_change': '2026-02-25T12:02:18.346734+0000', 'last_active': '2026-02-25T12:02:18.346734+0000', 'last_peered': '2026-02-25T12:02:16.933285+0000', 'last_clean': '2026-02-25T12:02:15.360485+0000', 'last_became_active': '2026-02-25T12:02:15.902237+0000', 'last_became_peered': '2026-02-25T12:02:15.902237+0000', 'last_unstale': '2026-02-25T12:02:16.933285+0000', 'last_undegraded': '2026-02-25T12:02:15.901810+0000', 'last_fullsized': '2026-02-25T12:02:15.901745+0000', 'mapping_epoch': 55, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 17, 'last_epoch_clean': 44, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2026-02-25T12:01:08.885417+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2026-02-25T12:01:08.885417+0000', 'last_clean_scrub_stamp': '2026-02-25T12:01:08.885417+0000', 'objects_scrubbed': 0, 'log_size': 2, 'log_dups_size': 0, 'ondisk_log_size': 2, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.000621375, 'stat_sum': {'num_bytes': 19, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2, 'num_write_kb': 2, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 19, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 1], 'acting': [4, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [1], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2026-02-25T12:02:22.880 DEBUG:teuthology.orchestra.run.trial117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2026-02-25T12:02:23.061 INFO:teuthology.orchestra.run.trial117.stdout: 2026-02-25T12:02:23.061 INFO:teuthology.orchestra.run.trial117.stdout:{"fsid":"43e63318-add1-4399-a1fc-80188594353b","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"noout flag(s) set","count":5},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 1/4 objects degraded (25.000%), 1 pg degraded","count":1},"muted":false}},"mutes":[]},"election_epoch":3,"quorum":[0],"quorum_names":["a"],"quorum_age":85,"monmap":{"epoch":1,"min_mon_release_name":"tentacle","num_mons":1},"osdmap":{"epoch":56,"num_osds":10,"num_up_osds":10,"osd_up_since":1772020937,"num_in_osds":10,"osd_in_since":1772020859,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":4},{"state_name":"active+undersized+wait","count":1},{"state_name":"active+undersized+degraded+wait","count":1},{"state_name":"peering","count":1},{"state_name":"activating","count":1}],"num_pgs":8,"num_pools":1,"num_objects":2,"data_bytes":19,"bytes_used":282230784,"bytes_avail":1073459593216,"bytes_total":1073741824000,"inactive_pgs_ratio":0.25,"degraded_objects":1,"degraded_total":4,"degraded_ratio":0.25},"fsmap":{"epoch":1,"btime":"2026-02-25T12:00:57:865346+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs"],"services":{}},"servicemap":{"epoch":3,"modified":"2026-02-25T12:02:04.733030+0000","services":{}},"progress_events":{}} 2026-02-25T12:02:23.073 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2026-02-25T12:02:23.073 DEBUG:teuthology.orchestra.run.trial117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-25T12:02:23.224 INFO:teuthology.orchestra.run.trial117.stdout: 2026-02-25T12:02:23.224 INFO:teuthology.orchestra.run.trial117.stderr:dumped all 2026-02-25T12:02:23.235 INFO:teuthology.orchestra.run.trial117.stdout:{"pg_ready":true,"pg_map":{"version":95,"stamp":"2026-02-25T12:02:22.735364+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":14,"acting":14,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":10,"num_per_pool_osds":10,"num_per_pool_omap_osds":8,"kb":1048576000,"kb_used":275616,"kb_used_data":6016,"kb_used_omap":248,"kb_used_meta":269191,"kb_avail":1048300384,"statfs":{"total":1073741824000,"available":1073459593216,"internally_reserved":0,"allocated":6160384,"data_stored":2950514,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":254629,"internal_metadata":275651931},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.000766"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":69,"reported_epoch":54,"state":"active+clean","last_fresh":"2026-02-25T12:02:16.933086+0000","last_change":"2026-02-25T12:02:06.883264+0000","last_active":"2026-02-25T12:02:16.933086+0000","last_peered":"2026-02-25T12:02:16.933086+0000","last_clean":"2026-02-25T12:02:16.933086+0000","last_became_active":"2026-02-25T12:02:06.883022+0000","last_became_peered":"2026-02-25T12:02:06.883022+0000","last_unstale":"2026-02-25T12:02:16.933086+0000","last_undegraded":"2026-02-25T12:02:16.933086+0000","last_fullsized":"2026-02-25T12:02:16.933086+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-25T12:01:08.885417+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-25T12:01:08.885417+0000","last_clean_scrub_stamp":"2026-02-25T12:01:08.885417+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-26T22:09:05.780239+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00026608399999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":87,"reported_epoch":54,"state":"active+clean","last_fresh":"2026-02-25T12:02:16.933144+0000","last_change":"2026-02-25T12:02:06.883262+0000","last_active":"2026-02-25T12:02:16.933144+0000","last_peered":"2026-02-25T12:02:16.933144+0000","last_clean":"2026-02-25T12:02:16.933144+0000","last_became_active":"2026-02-25T12:02:06.883003+0000","last_became_peered":"2026-02-25T12:02:06.883003+0000","last_unstale":"2026-02-25T12:02:16.933144+0000","last_undegraded":"2026-02-25T12:02:16.933144+0000","last_fullsized":"2026-02-25T12:02:16.933144+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-25T12:01:08.885417+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-25T12:01:08.885417+0000","last_clean_scrub_stamp":"2026-02-25T12:01:08.885417+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-26T15:40:20.663132+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00030945499999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":100,"reported_epoch":56,"state":"activating","last_fresh":"2026-02-25T12:02:18.904793+0000","last_change":"2026-02-25T12:02:18.904907+0000","last_active":"2026-02-25T12:02:17.903390+0000","last_peered":"2026-02-25T12:02:17.217325+0000","last_clean":"2026-02-25T12:02:15.427768+0000","last_became_active":"2026-02-25T12:02:15.902144+0000","last_became_peered":"2026-02-25T12:02:15.902144+0000","last_unstale":"2026-02-25T12:02:18.904793+0000","last_undegraded":"2026-02-25T12:02:18.904793+0000","last_fullsized":"2026-02-25T12:02:18.904793+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-25T12:01:08.885417+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-25T12:01:08.885417+0000","last_clean_scrub_stamp":"2026-02-25T12:01:08.885417+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00017141899999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0],"acting":[9,0],"avail_no_missing":["9","0"],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":54,"reported_epoch":54,"state":"active+undersized+wait","last_fresh":"2026-02-25T12:02:16.933130+0000","last_change":"2026-02-25T12:02:15.902089+0000","last_active":"2026-02-25T12:02:16.933130+0000","last_peered":"2026-02-25T12:02:16.933130+0000","last_clean":"2026-02-25T12:02:02.711424+0000","last_became_active":"2026-02-25T12:02:15.902089+0000","last_became_peered":"2026-02-25T12:02:15.902089+0000","last_unstale":"2026-02-25T12:02:16.933130+0000","last_undegraded":"2026-02-25T12:02:16.933130+0000","last_fullsized":"2026-02-25T12:02:15.901707+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-25T12:01:08.885417+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-25T12:01:08.885417+0000","last_clean_scrub_stamp":"2026-02-25T12:01:08.885417+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-26T23:11:50.865351+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6],"acting":[6],"avail_no_missing":["6"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.3","version":"18'1","reported_seq":83,"reported_epoch":54,"state":"active+undersized+degraded+wait","last_fresh":"2026-02-25T12:02:16.933084+0000","last_change":"2026-02-25T12:02:15.902037+0000","last_active":"2026-02-25T12:02:16.933084+0000","last_peered":"2026-02-25T12:02:16.933084+0000","last_clean":"2026-02-25T12:02:13.894638+0000","last_became_active":"2026-02-25T12:02:15.902037+0000","last_became_peered":"2026-02-25T12:02:15.902037+0000","last_unstale":"2026-02-25T12:02:16.933084+0000","last_undegraded":"2026-02-25T12:02:15.901816+0000","last_fullsized":"2026-02-25T12:02:15.901757+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-25T12:01:08.885417+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-25T12:01:08.885417+0000","last_clean_scrub_stamp":"2026-02-25T12:01:08.885417+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-26T17:32:07.261620+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[{"shards":"3","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"20'2","reported_seq":88,"reported_epoch":54,"state":"peering","last_fresh":"2026-02-25T12:02:16.933285+0000","last_change":"2026-02-25T12:02:18.346734+0000","last_active":"2026-02-25T12:02:18.346734+0000","last_peered":"2026-02-25T12:02:16.933285+0000","last_clean":"2026-02-25T12:02:15.360485+0000","last_became_active":"2026-02-25T12:02:15.902237+0000","last_became_peered":"2026-02-25T12:02:15.902237+0000","last_unstale":"2026-02-25T12:02:16.933285+0000","last_undegraded":"2026-02-25T12:02:15.901810+0000","last_fullsized":"2026-02-25T12:02:15.901745+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-25T12:01:08.885417+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-25T12:01:08.885417+0000","last_clean_scrub_stamp":"2026-02-25T12:01:08.885417+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062137499999999997,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":72,"reported_epoch":55,"state":"active+clean","last_fresh":"2026-02-25T12:02:17.937145+0000","last_change":"2026-02-25T12:02:06.883333+0000","last_active":"2026-02-25T12:02:17.937145+0000","last_peered":"2026-02-25T12:02:17.937145+0000","last_clean":"2026-02-25T12:02:17.937145+0000","last_became_active":"2026-02-25T12:02:06.883131+0000","last_became_peered":"2026-02-25T12:02:06.883131+0000","last_unstale":"2026-02-25T12:02:17.937145+0000","last_undegraded":"2026-02-25T12:02:17.937145+0000","last_fullsized":"2026-02-25T12:02:17.937145+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-25T12:01:08.885417+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-25T12:01:08.885417+0000","last_clean_scrub_stamp":"2026-02-25T12:01:08.885417+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-26T17:10:55.006863+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00015265299999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":84,"reported_epoch":55,"state":"active+clean","last_fresh":"2026-02-25T12:02:18.563160+0000","last_change":"2026-02-25T12:02:07.090070+0000","last_active":"2026-02-25T12:02:18.563160+0000","last_peered":"2026-02-25T12:02:18.563160+0000","last_clean":"2026-02-25T12:02:18.563160+0000","last_became_active":"2026-02-25T12:02:07.089906+0000","last_became_peered":"2026-02-25T12:02:07.089906+0000","last_unstale":"2026-02-25T12:02:18.563160+0000","last_undegraded":"2026-02-25T12:02:18.563160+0000","last_fullsized":"2026-02-25T12:02:18.563160+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-25T12:01:08.885417+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-25T12:01:08.885417+0000","last_clean_scrub_stamp":"2026-02-25T12:01:08.885417+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-26T23:02:05.903788+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9],"acting":[5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":14,"acting":14,"num_store_stats":9}],"osd_stats":[{"osd":9,"up_from":37,"seq":158913789969,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":27568,"kb_used_data":608,"kb_used_omap":27,"kb_used_meta":26916,"kb_avail":104830032,"statfs":{"total":107374182400,"available":107345952768,"internally_reserved":0,"allocated":622592,"data_stored":300132,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":28398,"internal_metadata":27562258},"hb_peers":[0,1,2,3,4,5,6,7,8],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":43,"seq":184683593746,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":104857600,"kb_used":27556,"kb_used_data":596,"kb_used_omap":22,"kb_used_meta":26921,"kb_avail":104830044,"statfs":{"total":107374182400,"available":107345965056,"internally_reserved":0,"allocated":610304,"data_stored":291658,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":23185,"internal_metadata":27567471},"hb_peers":[2,3,4,5,6,7,9],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":43,"seq":184683593745,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":27568,"kb_used_data":608,"kb_used_omap":27,"kb_used_meta":26916,"kb_avail":104830032,"statfs":{"total":107374182400,"available":107345952768,"internally_reserved":0,"allocated":622592,"data_stored":300132,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":27736,"internal_metadata":27562920},"hb_peers":[2,3,4,5,6,8,9],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":43,"seq":184683593745,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":104857600,"kb_used":27568,"kb_used_data":608,"kb_used_omap":26,"kb_used_meta":26917,"kb_avail":104830032,"statfs":{"total":107374182400,"available":107345952768,"internally_reserved":0,"allocated":622592,"data_stored":300132,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":27091,"internal_metadata":27563565},"hb_peers":[0,1,3,4,5,6,7,8,9],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":43,"seq":184683593745,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":27556,"kb_used_data":596,"kb_used_omap":24,"kb_used_meta":26919,"kb_avail":104830044,"statfs":{"total":107374182400,"available":107345965056,"internally_reserved":0,"allocated":610304,"data_stored":291658,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":25136,"internal_metadata":27565520},"hb_peers":[2,3,4,5,7,8,9],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":43,"seq":184683593746,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":27568,"kb_used_data":608,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":104830032,"statfs":{"total":107374182400,"available":107345952768,"internally_reserved":0,"allocated":622592,"data_stored":300132,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17326,"internal_metadata":27573330},"hb_peers":[2,3,4,6,7,8,9],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":43,"seq":184683593745,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":27560,"kb_used_data":600,"kb_used_omap":27,"kb_used_meta":26916,"kb_avail":104830040,"statfs":{"total":107374182400,"available":107345960960,"internally_reserved":0,"allocated":614400,"data_stored":291677,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":27742,"internal_metadata":27562914},"hb_peers":[2,3,5,6,7,8,9],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":43,"seq":184683593746,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":27556,"kb_used_data":596,"kb_used_omap":26,"kb_used_meta":26917,"kb_avail":104830044,"statfs":{"total":107374182400,"available":107345965056,"internally_reserved":0,"allocated":610304,"data_stored":291658,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":27088,"internal_metadata":27563568},"hb_peers":[2,4,5,6,7,8,9],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":55,"seq":236223201297,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":27560,"kb_used_data":600,"kb_used_omap":23,"kb_used_meta":26920,"kb_avail":104830040,"statfs":{"total":107374182400,"available":107345960960,"internally_reserved":0,"allocated":614400,"data_stored":291677,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":24487,"internal_metadata":27566169},"hb_peers":[2,3,4,5,6,7,8,9],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":55,"seq":236223201297,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":27556,"kb_used_data":596,"kb_used_omap":25,"kb_used_meta":26918,"kb_avail":104830044,"statfs":{"total":107374182400,"available":107345965056,"internally_reserved":0,"allocated":610304,"data_stored":291658,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":26440,"internal_metadata":27564216},"hb_peers":[2,3,4,5,6,7,8,9],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-25T12:02:26.237 DEBUG:teuthology.orchestra.run.trial117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-25T12:02:26.390 INFO:teuthology.orchestra.run.trial117.stdout: 2026-02-25T12:02:26.390 INFO:teuthology.orchestra.run.trial117.stderr:dumped all 2026-02-25T12:02:26.401 INFO:teuthology.orchestra.run.trial117.stdout:{"pg_ready":true,"pg_map":{"version":96,"stamp":"2026-02-25T12:02:24.735598+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":10,"num_per_pool_osds":10,"num_per_pool_omap_osds":8,"kb":1048576000,"kb_used":275808,"kb_used_data":6208,"kb_used_omap":256,"kb_used_meta":269183,"kb_avail":1048300192,"statfs":{"total":1073741824000,"available":1073459396608,"internally_reserved":0,"allocated":6356992,"data_stored":3078018,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":262441,"internal_metadata":275644119},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"7.833735"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":73,"reported_epoch":56,"state":"active+clean","last_fresh":"2026-02-25T12:02:18.903755+0000","last_change":"2026-02-25T12:02:06.883264+0000","last_active":"2026-02-25T12:02:18.903755+0000","last_peered":"2026-02-25T12:02:18.903755+0000","last_clean":"2026-02-25T12:02:18.903755+0000","last_became_active":"2026-02-25T12:02:06.883022+0000","last_became_peered":"2026-02-25T12:02:06.883022+0000","last_unstale":"2026-02-25T12:02:18.903755+0000","last_undegraded":"2026-02-25T12:02:18.903755+0000","last_fullsized":"2026-02-25T12:02:18.903755+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-25T12:01:08.885417+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-25T12:01:08.885417+0000","last_clean_scrub_stamp":"2026-02-25T12:01:08.885417+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-26T22:09:05.780239+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00026608399999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":91,"reported_epoch":56,"state":"active+clean","last_fresh":"2026-02-25T12:02:19.135470+0000","last_change":"2026-02-25T12:02:06.883262+0000","last_active":"2026-02-25T12:02:19.135470+0000","last_peered":"2026-02-25T12:02:19.135470+0000","last_clean":"2026-02-25T12:02:19.135470+0000","last_became_active":"2026-02-25T12:02:06.883003+0000","last_became_peered":"2026-02-25T12:02:06.883003+0000","last_unstale":"2026-02-25T12:02:19.135470+0000","last_undegraded":"2026-02-25T12:02:19.135470+0000","last_fullsized":"2026-02-25T12:02:19.135470+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-25T12:01:08.885417+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-25T12:01:08.885417+0000","last_clean_scrub_stamp":"2026-02-25T12:01:08.885417+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-26T15:40:20.663132+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00030945499999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":107,"reported_epoch":56,"state":"active+clean","last_fresh":"2026-02-25T12:02:19.227443+0000","last_change":"2026-02-25T12:02:19.227382+0000","last_active":"2026-02-25T12:02:19.227443+0000","last_peered":"2026-02-25T12:02:19.227443+0000","last_clean":"2026-02-25T12:02:19.227443+0000","last_became_active":"2026-02-25T12:02:19.227208+0000","last_became_peered":"2026-02-25T12:02:19.227208+0000","last_unstale":"2026-02-25T12:02:19.227443+0000","last_undegraded":"2026-02-25T12:02:19.227443+0000","last_fullsized":"2026-02-25T12:02:19.227443+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-25T12:01:08.885417+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-25T12:01:08.885417+0000","last_clean_scrub_stamp":"2026-02-25T12:01:08.885417+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-26T22:28:26.067831+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00017141899999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0],"acting":[9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":78,"reported_epoch":56,"state":"active+clean","last_fresh":"2026-02-25T12:02:19.136181+0000","last_change":"2026-02-25T12:02:19.136117+0000","last_active":"2026-02-25T12:02:19.136181+0000","last_peered":"2026-02-25T12:02:19.136181+0000","last_clean":"2026-02-25T12:02:19.136181+0000","last_became_active":"2026-02-25T12:02:19.135936+0000","last_became_peered":"2026-02-25T12:02:19.135936+0000","last_unstale":"2026-02-25T12:02:19.136181+0000","last_undegraded":"2026-02-25T12:02:19.136181+0000","last_fullsized":"2026-02-25T12:02:19.136181+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-25T12:01:08.885417+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-25T12:01:08.885417+0000","last_clean_scrub_stamp":"2026-02-25T12:01:08.885417+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-26T17:27:50.663072+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.3","version":"18'1","reported_seq":93,"reported_epoch":56,"state":"active+clean","last_fresh":"2026-02-25T12:02:18.905437+0000","last_change":"2026-02-25T12:02:18.905365+0000","last_active":"2026-02-25T12:02:18.905437+0000","last_peered":"2026-02-25T12:02:18.905437+0000","last_clean":"2026-02-25T12:02:18.905437+0000","last_became_active":"2026-02-25T12:02:18.905150+0000","last_became_peered":"2026-02-25T12:02:18.905150+0000","last_unstale":"2026-02-25T12:02:18.905437+0000","last_undegraded":"2026-02-25T12:02:18.905437+0000","last_fullsized":"2026-02-25T12:02:18.905437+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-25T12:01:08.885417+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-25T12:01:08.885417+0000","last_clean_scrub_stamp":"2026-02-25T12:01:08.885417+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-26T16:20:44.992590+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"20'2","reported_seq":97,"reported_epoch":56,"state":"active+clean","last_fresh":"2026-02-25T12:02:18.905682+0000","last_change":"2026-02-25T12:02:18.905622+0000","last_active":"2026-02-25T12:02:18.905682+0000","last_peered":"2026-02-25T12:02:18.905682+0000","last_clean":"2026-02-25T12:02:18.905682+0000","last_became_active":"2026-02-25T12:02:18.905430+0000","last_became_peered":"2026-02-25T12:02:18.905430+0000","last_unstale":"2026-02-25T12:02:18.905682+0000","last_undegraded":"2026-02-25T12:02:18.905682+0000","last_fullsized":"2026-02-25T12:02:18.905682+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-25T12:01:08.885417+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-25T12:01:08.885417+0000","last_clean_scrub_stamp":"2026-02-25T12:01:08.885417+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-26T22:29:58.049830+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062137499999999997,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":74,"reported_epoch":56,"state":"active+clean","last_fresh":"2026-02-25T12:02:19.135280+0000","last_change":"2026-02-25T12:02:06.883333+0000","last_active":"2026-02-25T12:02:19.135280+0000","last_peered":"2026-02-25T12:02:19.135280+0000","last_clean":"2026-02-25T12:02:19.135280+0000","last_became_active":"2026-02-25T12:02:06.883131+0000","last_became_peered":"2026-02-25T12:02:06.883131+0000","last_unstale":"2026-02-25T12:02:19.135280+0000","last_undegraded":"2026-02-25T12:02:19.135280+0000","last_fullsized":"2026-02-25T12:02:19.135280+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-25T12:01:08.885417+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-25T12:01:08.885417+0000","last_clean_scrub_stamp":"2026-02-25T12:01:08.885417+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-26T17:10:55.006863+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00015265299999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":86,"reported_epoch":56,"state":"active+clean","last_fresh":"2026-02-25T12:02:18.935442+0000","last_change":"2026-02-25T12:02:07.090070+0000","last_active":"2026-02-25T12:02:18.935442+0000","last_peered":"2026-02-25T12:02:18.935442+0000","last_clean":"2026-02-25T12:02:18.935442+0000","last_became_active":"2026-02-25T12:02:07.089906+0000","last_became_peered":"2026-02-25T12:02:07.089906+0000","last_unstale":"2026-02-25T12:02:18.935442+0000","last_undegraded":"2026-02-25T12:02:18.935442+0000","last_fullsized":"2026-02-25T12:02:18.935442+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-25T12:01:08.885417+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-25T12:01:08.885417+0000","last_clean_scrub_stamp":"2026-02-25T12:01:08.885417+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-26T23:02:05.903788+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9],"acting":[5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":9}],"osd_stats":[{"osd":9,"up_from":37,"seq":158913789970,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":27580,"kb_used_data":620,"kb_used_omap":27,"kb_used_meta":26916,"kb_avail":104830020,"statfs":{"total":107374182400,"available":107345940480,"internally_reserved":0,"allocated":634880,"data_stored":307798,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":28398,"internal_metadata":27562258},"hb_peers":[0,1,2,3,4,5,6,7,8],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":43,"seq":184683593747,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":104857600,"kb_used":27580,"kb_used_data":620,"kb_used_omap":22,"kb_used_meta":26921,"kb_avail":104830020,"statfs":{"total":107374182400,"available":107345940480,"internally_reserved":0,"allocated":634880,"data_stored":307798,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":23185,"internal_metadata":27567471},"hb_peers":[0,1,2,3,4,5,6,7,9],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":43,"seq":184683593746,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":27580,"kb_used_data":620,"kb_used_omap":27,"kb_used_meta":26916,"kb_avail":104830020,"statfs":{"total":107374182400,"available":107345940480,"internally_reserved":0,"allocated":634880,"data_stored":307798,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":28387,"internal_metadata":27562269},"hb_peers":[0,1,2,3,4,5,6,8,9],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":43,"seq":184683593746,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":104857600,"kb_used":27580,"kb_used_data":620,"kb_used_omap":27,"kb_used_meta":26916,"kb_avail":104830020,"statfs":{"total":107374182400,"available":107345940480,"internally_reserved":0,"allocated":634880,"data_stored":307798,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":27742,"internal_metadata":27562914},"hb_peers":[0,1,3,4,5,6,7,8,9],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":43,"seq":184683593746,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":27580,"kb_used_data":620,"kb_used_omap":25,"kb_used_meta":26918,"kb_avail":104830020,"statfs":{"total":107374182400,"available":107345940480,"internally_reserved":0,"allocated":634880,"data_stored":307798,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":26438,"internal_metadata":27564218},"hb_peers":[0,1,2,3,4,5,7,8,9],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":43,"seq":184683593747,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":27580,"kb_used_data":620,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":104830020,"statfs":{"total":107374182400,"available":107345940480,"internally_reserved":0,"allocated":634880,"data_stored":307798,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17326,"internal_metadata":27573330},"hb_peers":[0,1,2,3,4,6,7,8,9],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":43,"seq":184683593746,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":27584,"kb_used_data":624,"kb_used_omap":28,"kb_used_meta":26915,"kb_avail":104830016,"statfs":{"total":107374182400,"available":107345936384,"internally_reserved":0,"allocated":638976,"data_stored":307817,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":29044,"internal_metadata":27561612},"hb_peers":[0,1,2,3,5,6,7,8,9],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":43,"seq":184683593747,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":27580,"kb_used_data":620,"kb_used_omap":27,"kb_used_meta":26916,"kb_avail":104830020,"statfs":{"total":107374182400,"available":107345940480,"internally_reserved":0,"allocated":634880,"data_stored":307798,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":28390,"internal_metadata":27562266},"hb_peers":[0,1,2,4,5,6,7,8,9],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":55,"seq":236223201298,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":27584,"kb_used_data":624,"kb_used_omap":25,"kb_used_meta":26918,"kb_avail":104830016,"statfs":{"total":107374182400,"available":107345936384,"internally_reserved":0,"allocated":638976,"data_stored":307817,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":25789,"internal_metadata":27564867},"hb_peers":[0,2,3,4,5,6,7,8,9],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":55,"seq":236223201298,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":27580,"kb_used_data":620,"kb_used_omap":27,"kb_used_meta":26916,"kb_avail":104830020,"statfs":{"total":107374182400,"available":107345940480,"internally_reserved":0,"allocated":634880,"data_stored":307798,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":27742,"internal_metadata":27562914},"hb_peers":[1,2,3,4,5,6,7,8,9],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-25T12:02:26.402 INFO:tasks.ceph.ceph_manager.ceph:clean! 2026-02-25T12:02:26.402 DEBUG:teuthology.orchestra.run.trial117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-25T12:02:26.559 INFO:teuthology.orchestra.run.trial117.stdout: 2026-02-25T12:02:26.560 INFO:teuthology.orchestra.run.trial117.stderr:dumped all 2026-02-25T12:02:26.570 INFO:teuthology.orchestra.run.trial117.stdout:{"pg_ready":true,"pg_map":{"version":96,"stamp":"2026-02-25T12:02:24.735598+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":10,"num_per_pool_osds":10,"num_per_pool_omap_osds":8,"kb":1048576000,"kb_used":275808,"kb_used_data":6208,"kb_used_omap":256,"kb_used_meta":269183,"kb_avail":1048300192,"statfs":{"total":1073741824000,"available":1073459396608,"internally_reserved":0,"allocated":6356992,"data_stored":3078018,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":262441,"internal_metadata":275644119},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"7.833735"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":73,"reported_epoch":56,"state":"active+clean","last_fresh":"2026-02-25T12:02:18.903755+0000","last_change":"2026-02-25T12:02:06.883264+0000","last_active":"2026-02-25T12:02:18.903755+0000","last_peered":"2026-02-25T12:02:18.903755+0000","last_clean":"2026-02-25T12:02:18.903755+0000","last_became_active":"2026-02-25T12:02:06.883022+0000","last_became_peered":"2026-02-25T12:02:06.883022+0000","last_unstale":"2026-02-25T12:02:18.903755+0000","last_undegraded":"2026-02-25T12:02:18.903755+0000","last_fullsized":"2026-02-25T12:02:18.903755+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-25T12:01:08.885417+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-25T12:01:08.885417+0000","last_clean_scrub_stamp":"2026-02-25T12:01:08.885417+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-26T22:09:05.780239+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00026608399999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":91,"reported_epoch":56,"state":"active+clean","last_fresh":"2026-02-25T12:02:19.135470+0000","last_change":"2026-02-25T12:02:06.883262+0000","last_active":"2026-02-25T12:02:19.135470+0000","last_peered":"2026-02-25T12:02:19.135470+0000","last_clean":"2026-02-25T12:02:19.135470+0000","last_became_active":"2026-02-25T12:02:06.883003+0000","last_became_peered":"2026-02-25T12:02:06.883003+0000","last_unstale":"2026-02-25T12:02:19.135470+0000","last_undegraded":"2026-02-25T12:02:19.135470+0000","last_fullsized":"2026-02-25T12:02:19.135470+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-25T12:01:08.885417+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-25T12:01:08.885417+0000","last_clean_scrub_stamp":"2026-02-25T12:01:08.885417+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-26T15:40:20.663132+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00030945499999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":107,"reported_epoch":56,"state":"active+clean","last_fresh":"2026-02-25T12:02:19.227443+0000","last_change":"2026-02-25T12:02:19.227382+0000","last_active":"2026-02-25T12:02:19.227443+0000","last_peered":"2026-02-25T12:02:19.227443+0000","last_clean":"2026-02-25T12:02:19.227443+0000","last_became_active":"2026-02-25T12:02:19.227208+0000","last_became_peered":"2026-02-25T12:02:19.227208+0000","last_unstale":"2026-02-25T12:02:19.227443+0000","last_undegraded":"2026-02-25T12:02:19.227443+0000","last_fullsized":"2026-02-25T12:02:19.227443+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-25T12:01:08.885417+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-25T12:01:08.885417+0000","last_clean_scrub_stamp":"2026-02-25T12:01:08.885417+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-26T22:28:26.067831+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00017141899999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0],"acting":[9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":78,"reported_epoch":56,"state":"active+clean","last_fresh":"2026-02-25T12:02:19.136181+0000","last_change":"2026-02-25T12:02:19.136117+0000","last_active":"2026-02-25T12:02:19.136181+0000","last_peered":"2026-02-25T12:02:19.136181+0000","last_clean":"2026-02-25T12:02:19.136181+0000","last_became_active":"2026-02-25T12:02:19.135936+0000","last_became_peered":"2026-02-25T12:02:19.135936+0000","last_unstale":"2026-02-25T12:02:19.136181+0000","last_undegraded":"2026-02-25T12:02:19.136181+0000","last_fullsized":"2026-02-25T12:02:19.136181+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-25T12:01:08.885417+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-25T12:01:08.885417+0000","last_clean_scrub_stamp":"2026-02-25T12:01:08.885417+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-26T17:27:50.663072+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.3","version":"18'1","reported_seq":93,"reported_epoch":56,"state":"active+clean","last_fresh":"2026-02-25T12:02:18.905437+0000","last_change":"2026-02-25T12:02:18.905365+0000","last_active":"2026-02-25T12:02:18.905437+0000","last_peered":"2026-02-25T12:02:18.905437+0000","last_clean":"2026-02-25T12:02:18.905437+0000","last_became_active":"2026-02-25T12:02:18.905150+0000","last_became_peered":"2026-02-25T12:02:18.905150+0000","last_unstale":"2026-02-25T12:02:18.905437+0000","last_undegraded":"2026-02-25T12:02:18.905437+0000","last_fullsized":"2026-02-25T12:02:18.905437+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-25T12:01:08.885417+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-25T12:01:08.885417+0000","last_clean_scrub_stamp":"2026-02-25T12:01:08.885417+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-26T16:20:44.992590+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"20'2","reported_seq":97,"reported_epoch":56,"state":"active+clean","last_fresh":"2026-02-25T12:02:18.905682+0000","last_change":"2026-02-25T12:02:18.905622+0000","last_active":"2026-02-25T12:02:18.905682+0000","last_peered":"2026-02-25T12:02:18.905682+0000","last_clean":"2026-02-25T12:02:18.905682+0000","last_became_active":"2026-02-25T12:02:18.905430+0000","last_became_peered":"2026-02-25T12:02:18.905430+0000","last_unstale":"2026-02-25T12:02:18.905682+0000","last_undegraded":"2026-02-25T12:02:18.905682+0000","last_fullsized":"2026-02-25T12:02:18.905682+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-25T12:01:08.885417+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-25T12:01:08.885417+0000","last_clean_scrub_stamp":"2026-02-25T12:01:08.885417+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-26T22:29:58.049830+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062137499999999997,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":74,"reported_epoch":56,"state":"active+clean","last_fresh":"2026-02-25T12:02:19.135280+0000","last_change":"2026-02-25T12:02:06.883333+0000","last_active":"2026-02-25T12:02:19.135280+0000","last_peered":"2026-02-25T12:02:19.135280+0000","last_clean":"2026-02-25T12:02:19.135280+0000","last_became_active":"2026-02-25T12:02:06.883131+0000","last_became_peered":"2026-02-25T12:02:06.883131+0000","last_unstale":"2026-02-25T12:02:19.135280+0000","last_undegraded":"2026-02-25T12:02:19.135280+0000","last_fullsized":"2026-02-25T12:02:19.135280+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-25T12:01:08.885417+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-25T12:01:08.885417+0000","last_clean_scrub_stamp":"2026-02-25T12:01:08.885417+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-26T17:10:55.006863+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00015265299999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":86,"reported_epoch":56,"state":"active+clean","last_fresh":"2026-02-25T12:02:18.935442+0000","last_change":"2026-02-25T12:02:07.090070+0000","last_active":"2026-02-25T12:02:18.935442+0000","last_peered":"2026-02-25T12:02:18.935442+0000","last_clean":"2026-02-25T12:02:18.935442+0000","last_became_active":"2026-02-25T12:02:07.089906+0000","last_became_peered":"2026-02-25T12:02:07.089906+0000","last_unstale":"2026-02-25T12:02:18.935442+0000","last_undegraded":"2026-02-25T12:02:18.935442+0000","last_fullsized":"2026-02-25T12:02:18.935442+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-25T12:01:08.885417+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-25T12:01:08.885417+0000","last_clean_scrub_stamp":"2026-02-25T12:01:08.885417+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-26T23:02:05.903788+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9],"acting":[5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":9}],"osd_stats":[{"osd":9,"up_from":37,"seq":158913789970,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":27580,"kb_used_data":620,"kb_used_omap":27,"kb_used_meta":26916,"kb_avail":104830020,"statfs":{"total":107374182400,"available":107345940480,"internally_reserved":0,"allocated":634880,"data_stored":307798,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":28398,"internal_metadata":27562258},"hb_peers":[0,1,2,3,4,5,6,7,8],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":43,"seq":184683593747,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":104857600,"kb_used":27580,"kb_used_data":620,"kb_used_omap":22,"kb_used_meta":26921,"kb_avail":104830020,"statfs":{"total":107374182400,"available":107345940480,"internally_reserved":0,"allocated":634880,"data_stored":307798,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":23185,"internal_metadata":27567471},"hb_peers":[0,1,2,3,4,5,6,7,9],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":43,"seq":184683593746,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":27580,"kb_used_data":620,"kb_used_omap":27,"kb_used_meta":26916,"kb_avail":104830020,"statfs":{"total":107374182400,"available":107345940480,"internally_reserved":0,"allocated":634880,"data_stored":307798,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":28387,"internal_metadata":27562269},"hb_peers":[0,1,2,3,4,5,6,8,9],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":43,"seq":184683593746,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":104857600,"kb_used":27580,"kb_used_data":620,"kb_used_omap":27,"kb_used_meta":26916,"kb_avail":104830020,"statfs":{"total":107374182400,"available":107345940480,"internally_reserved":0,"allocated":634880,"data_stored":307798,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":27742,"internal_metadata":27562914},"hb_peers":[0,1,3,4,5,6,7,8,9],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":43,"seq":184683593746,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":27580,"kb_used_data":620,"kb_used_omap":25,"kb_used_meta":26918,"kb_avail":104830020,"statfs":{"total":107374182400,"available":107345940480,"internally_reserved":0,"allocated":634880,"data_stored":307798,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":26438,"internal_metadata":27564218},"hb_peers":[0,1,2,3,4,5,7,8,9],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":43,"seq":184683593747,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":27580,"kb_used_data":620,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":104830020,"statfs":{"total":107374182400,"available":107345940480,"internally_reserved":0,"allocated":634880,"data_stored":307798,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17326,"internal_metadata":27573330},"hb_peers":[0,1,2,3,4,6,7,8,9],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":43,"seq":184683593746,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":27584,"kb_used_data":624,"kb_used_omap":28,"kb_used_meta":26915,"kb_avail":104830016,"statfs":{"total":107374182400,"available":107345936384,"internally_reserved":0,"allocated":638976,"data_stored":307817,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":29044,"internal_metadata":27561612},"hb_peers":[0,1,2,3,5,6,7,8,9],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":43,"seq":184683593747,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":27580,"kb_used_data":620,"kb_used_omap":27,"kb_used_meta":26916,"kb_avail":104830020,"statfs":{"total":107374182400,"available":107345940480,"internally_reserved":0,"allocated":634880,"data_stored":307798,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":28390,"internal_metadata":27562266},"hb_peers":[0,1,2,4,5,6,7,8,9],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":55,"seq":236223201298,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":27584,"kb_used_data":624,"kb_used_omap":25,"kb_used_meta":26918,"kb_avail":104830016,"statfs":{"total":107374182400,"available":107345936384,"internally_reserved":0,"allocated":638976,"data_stored":307817,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":25789,"internal_metadata":27564867},"hb_peers":[0,2,3,4,5,6,7,8,9],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":55,"seq":236223201298,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":27580,"kb_used_data":620,"kb_used_omap":27,"kb_used_meta":26916,"kb_avail":104830020,"statfs":{"total":107374182400,"available":107345940480,"internally_reserved":0,"allocated":634880,"data_stored":307798,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":27742,"internal_metadata":27562914},"hb_peers":[1,2,3,4,5,6,7,8,9],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-25T12:02:26.573 DEBUG:teuthology.orchestra.run.trial117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2026-02-25T12:02:26.738 INFO:teuthology.orchestra.run.trial117.stdout: 2026-02-25T12:02:26.738 INFO:teuthology.orchestra.run.trial117.stdout:{"epoch":56,"fsid":"43e63318-add1-4399-a1fc-80188594353b","created":"2026-02-25T12:00:57.865568+0000","modified":"2026-02-25T12:02:18.902190+0000","last_up_change":"2026-02-25T12:02:17.901585+0000","last_in_change":"2026-02-25T12:00:59.546090+0000","flags":"noout,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5800064,"flags_set":["noout","pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":11,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":10,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":false,"pools":[{"pool":1,"pool_name":"rbd","create_time":"2026-02-25T12:01:08.174903+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":2,"snap_epoch":20,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.5,"score_stable":2.5,"optimal_score":0.80000001192092896,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1.0000001192092896,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"a879bc90-f2aa-4692-98ac-d45e56987af0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":15,"last_clean_end":54,"up_from":55,"up_thru":34,"down_at":52,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.117:6810","nonce":2144854447},{"type":"v1","addr":"10.20.193.117:6811","nonce":2144854447}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.117:6882","nonce":2147854447},{"type":"v1","addr":"10.20.193.117:6883","nonce":2147854447}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.117:6816","nonce":2144854447},{"type":"v1","addr":"10.20.193.117:6817","nonce":2144854447}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.117:6814","nonce":2144854447},{"type":"v1","addr":"10.20.193.117:6815","nonce":2144854447}]},"public_addr":"10.20.193.117:6811/2144854447","cluster_addr":"10.20.193.117:6883/2147854447","heartbeat_back_addr":"10.20.193.117:6817/2144854447","heartbeat_front_addr":"10.20.193.117:6815/2144854447","state":["exists","up"]},{"osd":1,"uuid":"78845355-1113-42e3-9453-cbd686dd4cfb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":15,"last_clean_end":54,"up_from":55,"up_thru":55,"down_at":52,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.117:6826","nonce":491871890},{"type":"v1","addr":"10.20.193.117:6827","nonce":491871890}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.117:6860","nonce":494871890},{"type":"v1","addr":"10.20.193.117:6861","nonce":494871890}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.117:6832","nonce":491871890},{"type":"v1","addr":"10.20.193.117:6833","nonce":491871890}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.117:6830","nonce":491871890},{"type":"v1","addr":"10.20.193.117:6831","nonce":491871890}]},"public_addr":"10.20.193.117:6827/491871890","cluster_addr":"10.20.193.117:6861/494871890","heartbeat_back_addr":"10.20.193.117:6833/491871890","heartbeat_front_addr":"10.20.193.117:6831/491871890","state":["exists","up"]},{"osd":2,"uuid":"f96856b9-ffb4-4170-918a-dcb7374aebbb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":16,"last_clean_end":42,"up_from":43,"up_thru":0,"down_at":40,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.117:6858","nonce":3832000820},{"type":"v1","addr":"10.20.193.117:6859","nonce":3832000820}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.117:6804","nonce":3833000820},{"type":"v1","addr":"10.20.193.117:6805","nonce":3833000820}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.117:6864","nonce":3832000820},{"type":"v1","addr":"10.20.193.117:6865","nonce":3832000820}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.117:6862","nonce":3832000820},{"type":"v1","addr":"10.20.193.117:6863","nonce":3832000820}]},"public_addr":"10.20.193.117:6859/3832000820","cluster_addr":"10.20.193.117:6805/3833000820","heartbeat_back_addr":"10.20.193.117:6865/3832000820","heartbeat_front_addr":"10.20.193.117:6863/3832000820","state":["exists","up"]},{"osd":3,"uuid":"5d49c0b4-c47f-466c-bb15-52ef5e96fd02","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":15,"last_clean_end":42,"up_from":43,"up_thru":55,"down_at":40,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.117:6818","nonce":2181619050},{"type":"v1","addr":"10.20.193.117:6819","nonce":2181619050}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.117:6829","nonce":2182619050},{"type":"v1","addr":"10.20.193.117:6813","nonce":2182619050}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.117:6824","nonce":2181619050},{"type":"v1","addr":"10.20.193.117:6825","nonce":2181619050}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.117:6822","nonce":2181619050},{"type":"v1","addr":"10.20.193.117:6823","nonce":2181619050}]},"public_addr":"10.20.193.117:6819/2181619050","cluster_addr":"10.20.193.117:6813/2182619050","heartbeat_back_addr":"10.20.193.117:6825/2181619050","heartbeat_front_addr":"10.20.193.117:6823/2181619050","state":["exists","up"]},{"osd":4,"uuid":"8327850a-7ec7-4203-ae90-0a54b25d3a35","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":15,"last_clean_end":42,"up_from":43,"up_thru":55,"down_at":40,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.117:6843","nonce":2433940125},{"type":"v1","addr":"10.20.193.117:6845","nonce":2433940125}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.117:6821","nonce":2434940125},{"type":"v1","addr":"10.20.193.117:6846","nonce":2434940125}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.117:6855","nonce":2433940125},{"type":"v1","addr":"10.20.193.117:6857","nonce":2433940125}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.117:6851","nonce":2433940125},{"type":"v1","addr":"10.20.193.117:6853","nonce":2433940125}]},"public_addr":"10.20.193.117:6845/2433940125","cluster_addr":"10.20.193.117:6846/2434940125","heartbeat_back_addr":"10.20.193.117:6857/2433940125","heartbeat_front_addr":"10.20.193.117:6853/2433940125","state":["exists","up"]},{"osd":5,"uuid":"04975d9f-b1e5-465b-ae33-7a342cf7f976","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":15,"last_clean_end":42,"up_from":43,"up_thru":43,"down_at":40,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.117:6866","nonce":3722132383},{"type":"v1","addr":"10.20.193.117:6867","nonce":3722132383}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.117:6836","nonce":3723132383},{"type":"v1","addr":"10.20.193.117:6847","nonce":3723132383}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.117:6872","nonce":3722132383},{"type":"v1","addr":"10.20.193.117:6873","nonce":3722132383}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.117:6870","nonce":3722132383},{"type":"v1","addr":"10.20.193.117:6871","nonce":3722132383}]},"public_addr":"10.20.193.117:6867/3722132383","cluster_addr":"10.20.193.117:6847/3723132383","heartbeat_back_addr":"10.20.193.117:6873/3722132383","heartbeat_front_addr":"10.20.193.117:6871/3722132383","state":["exists","up"]},{"osd":6,"uuid":"6d6dfec7-2127-44bb-ad90-d3924e3d0902","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":15,"last_clean_end":42,"up_from":43,"up_thru":52,"down_at":40,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.117:6802","nonce":3926541475},{"type":"v1","addr":"10.20.193.117:6803","nonce":3926541475}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.117:6868","nonce":3927541475},{"type":"v1","addr":"10.20.193.117:6869","nonce":3927541475}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.117:6808","nonce":3926541475},{"type":"v1","addr":"10.20.193.117:6809","nonce":3926541475}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.117:6806","nonce":3926541475},{"type":"v1","addr":"10.20.193.117:6807","nonce":3926541475}]},"public_addr":"10.20.193.117:6803/3926541475","cluster_addr":"10.20.193.117:6869/3927541475","heartbeat_back_addr":"10.20.193.117:6809/3926541475","heartbeat_front_addr":"10.20.193.117:6807/3926541475","state":["exists","up"]},{"osd":7,"uuid":"617a797b-c363-4bde-a19e-ce36bedaba09","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":16,"last_clean_end":42,"up_from":43,"up_thru":43,"down_at":40,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.117:6842","nonce":1628495216},{"type":"v1","addr":"10.20.193.117:6844","nonce":1628495216}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.117:6820","nonce":1629495216},{"type":"v1","addr":"10.20.193.117:6812","nonce":1629495216}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.117:6854","nonce":1628495216},{"type":"v1","addr":"10.20.193.117:6856","nonce":1628495216}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.117:6850","nonce":1628495216},{"type":"v1","addr":"10.20.193.117:6852","nonce":1628495216}]},"public_addr":"10.20.193.117:6844/1628495216","cluster_addr":"10.20.193.117:6812/1629495216","heartbeat_back_addr":"10.20.193.117:6856/1628495216","heartbeat_front_addr":"10.20.193.117:6852/1628495216","state":["exists","up"]},{"osd":8,"uuid":"8a6d937f-608e-4cf8-9d91-fc6104b9ea99","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":16,"last_clean_end":42,"up_from":43,"up_thru":17,"down_at":40,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.117:6874","nonce":3404023726},{"type":"v1","addr":"10.20.193.117:6875","nonce":3404023726}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.117:6837","nonce":3406023726},{"type":"v1","addr":"10.20.193.117:6877","nonce":3406023726}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.117:6880","nonce":3404023726},{"type":"v1","addr":"10.20.193.117:6881","nonce":3404023726}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.117:6878","nonce":3404023726},{"type":"v1","addr":"10.20.193.117:6879","nonce":3404023726}]},"public_addr":"10.20.193.117:6875/3404023726","cluster_addr":"10.20.193.117:6877/3406023726","heartbeat_back_addr":"10.20.193.117:6881/3404023726","heartbeat_front_addr":"10.20.193.117:6879/3404023726","state":["exists","up"]},{"osd":9,"uuid":"5dba0073-60fa-4ce0-baab-a34df860f7b1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":16,"last_clean_end":36,"up_from":37,"up_thru":55,"down_at":34,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.117:6834","nonce":387956821},{"type":"v1","addr":"10.20.193.117:6835","nonce":387956821}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.117:6828","nonce":388956821},{"type":"v1","addr":"10.20.193.117:6876","nonce":388956821}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.117:6840","nonce":387956821},{"type":"v1","addr":"10.20.193.117:6841","nonce":387956821}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.117:6838","nonce":387956821},{"type":"v1","addr":"10.20.193.117:6839","nonce":387956821}]},"public_addr":"10.20.193.117:6835/387956821","cluster_addr":"10.20.193.117:6876/388956821","heartbeat_back_addr":"10.20.193.117:6841/387956821","heartbeat_front_addr":"10.20.193.117:6839/387956821","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"2026-02-25T12:02:14.895291+0000","laggy_probability":0.65700000524520874,"laggy_interval":1,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-25T12:01:01.784765+0000","dead_epoch":52},{"osd":1,"down_stamp":"2026-02-25T12:02:14.895291+0000","laggy_probability":0.65700000524520874,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-25T12:01:01.865864+0000","dead_epoch":54},{"osd":2,"down_stamp":"2026-02-25T12:02:02.867986+0000","laggy_probability":0.30000001192092896,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-25T12:01:01.998591+0000","dead_epoch":42},{"osd":3,"down_stamp":"2026-02-25T12:02:02.867986+0000","laggy_probability":0.30000001192092896,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-25T12:01:01.827884+0000","dead_epoch":40},{"osd":4,"down_stamp":"2026-02-25T12:02:02.867986+0000","laggy_probability":0.30000001192092896,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-25T12:01:01.900858+0000","dead_epoch":40},{"osd":5,"down_stamp":"2026-02-25T12:02:02.867986+0000","laggy_probability":0.30000001192092896,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-25T12:01:01.849251+0000","dead_epoch":40},{"osd":6,"down_stamp":"2026-02-25T12:02:02.867986+0000","laggy_probability":0.30000001192092896,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-25T12:01:01.817516+0000","dead_epoch":41},{"osd":7,"down_stamp":"2026-02-25T12:02:02.867986+0000","laggy_probability":0.30000001192092896,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-25T12:01:01.903985+0000","dead_epoch":40},{"osd":8,"down_stamp":"2026-02-25T12:02:02.867986+0000","laggy_probability":0.50999999046325684,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-25T12:01:01.938704+0000","dead_epoch":40},{"osd":9,"down_stamp":"2026-02-25T12:01:56.854651+0000","laggy_probability":0.30000001192092896,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-25T12:01:01.968765+0000","dead_epoch":35}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-25T12:02:27.751 INFO:tasks.ceph:Scrubbing osd.0 2026-02-25T12:02:27.751 DEBUG:teuthology.orchestra.run.trial117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 config set osd_debug_deep_scrub_sleep 0 2026-02-25T12:02:27.838 INFO:teuthology.orchestra.run.trial117.stdout:{ 2026-02-25T12:02:27.839 INFO:teuthology.orchestra.run.trial117.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2026-02-25T12:02:27.839 INFO:teuthology.orchestra.run.trial117.stdout:} 2026-02-25T12:02:27.848 DEBUG:teuthology.orchestra.run.trial117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 0 2026-02-25T12:02:28.002 INFO:teuthology.orchestra.run.trial117.stderr:instructed osd(s) 0 to deep-scrub 2026-02-25T12:02:28.015 INFO:tasks.ceph:Scrubbing osd.1 2026-02-25T12:02:28.015 DEBUG:teuthology.orchestra.run.trial117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 config set osd_debug_deep_scrub_sleep 0 2026-02-25T12:02:28.093 INFO:teuthology.orchestra.run.trial117.stdout:{ 2026-02-25T12:02:28.093 INFO:teuthology.orchestra.run.trial117.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2026-02-25T12:02:28.093 INFO:teuthology.orchestra.run.trial117.stdout:} 2026-02-25T12:02:28.102 DEBUG:teuthology.orchestra.run.trial117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 1 2026-02-25T12:02:28.256 INFO:teuthology.orchestra.run.trial117.stderr:instructed osd(s) 1 to deep-scrub 2026-02-25T12:02:28.270 INFO:tasks.ceph:Scrubbing osd.2 2026-02-25T12:02:28.270 DEBUG:teuthology.orchestra.run.trial117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 config set osd_debug_deep_scrub_sleep 0 2026-02-25T12:02:28.347 INFO:teuthology.orchestra.run.trial117.stdout:{ 2026-02-25T12:02:28.347 INFO:teuthology.orchestra.run.trial117.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2026-02-25T12:02:28.347 INFO:teuthology.orchestra.run.trial117.stdout:} 2026-02-25T12:02:28.356 DEBUG:teuthology.orchestra.run.trial117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 2 2026-02-25T12:02:28.513 INFO:teuthology.orchestra.run.trial117.stderr:instructed osd(s) 2 to deep-scrub 2026-02-25T12:02:28.524 INFO:tasks.ceph:Scrubbing osd.3 2026-02-25T12:02:28.525 DEBUG:teuthology.orchestra.run.trial117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 config set osd_debug_deep_scrub_sleep 0 2026-02-25T12:02:28.602 INFO:teuthology.orchestra.run.trial117.stdout:{ 2026-02-25T12:02:28.602 INFO:teuthology.orchestra.run.trial117.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2026-02-25T12:02:28.602 INFO:teuthology.orchestra.run.trial117.stdout:} 2026-02-25T12:02:28.611 DEBUG:teuthology.orchestra.run.trial117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 3 2026-02-25T12:02:28.763 INFO:tasks.ceph.mgr.x.trial117.stderr:INFO:mgr_util:scanning for idle connections.. 2026-02-25T12:02:28.763 INFO:tasks.ceph.mgr.x.trial117.stderr:INFO:mgr_util:cleaning up connections: [] 2026-02-25T12:02:28.763 INFO:teuthology.orchestra.run.trial117.stderr:instructed osd(s) 3 to deep-scrub 2026-02-25T12:02:28.766 INFO:tasks.ceph.mgr.x.trial117.stderr:INFO:mgr_util:scanning for idle connections.. 2026-02-25T12:02:28.766 INFO:tasks.ceph.mgr.x.trial117.stderr:INFO:mgr_util:cleaning up connections: [] 2026-02-25T12:02:28.767 INFO:tasks.ceph.mgr.x.trial117.stderr:INFO:mgr_util:scanning for idle connections.. 2026-02-25T12:02:28.767 INFO:tasks.ceph.mgr.x.trial117.stderr:INFO:mgr_util:cleaning up connections: [] 2026-02-25T12:02:28.776 INFO:tasks.ceph:Scrubbing osd.4 2026-02-25T12:02:28.776 DEBUG:teuthology.orchestra.run.trial117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.4 config set osd_debug_deep_scrub_sleep 0 2026-02-25T12:02:28.856 INFO:teuthology.orchestra.run.trial117.stdout:{ 2026-02-25T12:02:28.856 INFO:teuthology.orchestra.run.trial117.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2026-02-25T12:02:28.856 INFO:teuthology.orchestra.run.trial117.stdout:} 2026-02-25T12:02:28.868 DEBUG:teuthology.orchestra.run.trial117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 4 2026-02-25T12:02:29.021 INFO:teuthology.orchestra.run.trial117.stderr:instructed osd(s) 4 to deep-scrub 2026-02-25T12:02:29.032 INFO:tasks.ceph:Scrubbing osd.5 2026-02-25T12:02:29.032 DEBUG:teuthology.orchestra.run.trial117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.5 config set osd_debug_deep_scrub_sleep 0 2026-02-25T12:02:29.109 INFO:teuthology.orchestra.run.trial117.stdout:{ 2026-02-25T12:02:29.109 INFO:teuthology.orchestra.run.trial117.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2026-02-25T12:02:29.110 INFO:teuthology.orchestra.run.trial117.stdout:} 2026-02-25T12:02:29.121 DEBUG:teuthology.orchestra.run.trial117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 5 2026-02-25T12:02:29.278 INFO:teuthology.orchestra.run.trial117.stderr:instructed osd(s) 5 to deep-scrub 2026-02-25T12:02:29.290 INFO:tasks.ceph:Scrubbing osd.6 2026-02-25T12:02:29.290 DEBUG:teuthology.orchestra.run.trial117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.6 config set osd_debug_deep_scrub_sleep 0 2026-02-25T12:02:29.376 INFO:teuthology.orchestra.run.trial117.stdout:{ 2026-02-25T12:02:29.376 INFO:teuthology.orchestra.run.trial117.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2026-02-25T12:02:29.376 INFO:teuthology.orchestra.run.trial117.stdout:} 2026-02-25T12:02:29.385 DEBUG:teuthology.orchestra.run.trial117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 6 2026-02-25T12:02:29.540 INFO:teuthology.orchestra.run.trial117.stderr:instructed osd(s) 6 to deep-scrub 2026-02-25T12:02:29.551 INFO:tasks.ceph:Scrubbing osd.7 2026-02-25T12:02:29.551 DEBUG:teuthology.orchestra.run.trial117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.7 config set osd_debug_deep_scrub_sleep 0 2026-02-25T12:02:29.635 INFO:teuthology.orchestra.run.trial117.stdout:{ 2026-02-25T12:02:29.635 INFO:teuthology.orchestra.run.trial117.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2026-02-25T12:02:29.635 INFO:teuthology.orchestra.run.trial117.stdout:} 2026-02-25T12:02:29.644 DEBUG:teuthology.orchestra.run.trial117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 7 2026-02-25T12:02:29.799 INFO:teuthology.orchestra.run.trial117.stderr:instructed osd(s) 7 to deep-scrub 2026-02-25T12:02:29.810 INFO:tasks.ceph:Scrubbing osd.8 2026-02-25T12:02:29.810 DEBUG:teuthology.orchestra.run.trial117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.8 config set osd_debug_deep_scrub_sleep 0 2026-02-25T12:02:29.888 INFO:teuthology.orchestra.run.trial117.stdout:{ 2026-02-25T12:02:29.889 INFO:teuthology.orchestra.run.trial117.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2026-02-25T12:02:29.889 INFO:teuthology.orchestra.run.trial117.stdout:} 2026-02-25T12:02:29.900 DEBUG:teuthology.orchestra.run.trial117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 8 2026-02-25T12:02:30.054 INFO:teuthology.orchestra.run.trial117.stderr:instructed osd(s) 8 to deep-scrub 2026-02-25T12:02:30.066 INFO:tasks.ceph:Scrubbing osd.9 2026-02-25T12:02:30.066 DEBUG:teuthology.orchestra.run.trial117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.9 config set osd_debug_deep_scrub_sleep 0 2026-02-25T12:02:30.140 INFO:teuthology.orchestra.run.trial117.stdout:{ 2026-02-25T12:02:30.141 INFO:teuthology.orchestra.run.trial117.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2026-02-25T12:02:30.141 INFO:teuthology.orchestra.run.trial117.stdout:} 2026-02-25T12:02:30.150 DEBUG:teuthology.orchestra.run.trial117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 9 2026-02-25T12:02:30.305 INFO:teuthology.orchestra.run.trial117.stderr:instructed osd(s) 9 to deep-scrub 2026-02-25T12:02:30.317 DEBUG:teuthology.orchestra.run.trial117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-25T12:02:30.483 INFO:teuthology.orchestra.run.trial117.stdout: 2026-02-25T12:02:30.483 INFO:teuthology.orchestra.run.trial117.stderr:dumped all 2026-02-25T12:02:30.494 INFO:teuthology.orchestra.run.trial117.stdout:{"pg_ready":true,"pg_map":{"version":98,"stamp":"2026-02-25T12:02:28.736055+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":10,"num_per_pool_osds":10,"num_per_pool_omap_osds":8,"kb":1048576000,"kb_used":275808,"kb_used_data":6208,"kb_used_omap":256,"kb_used_meta":269183,"kb_avail":1048300192,"statfs":{"total":1073741824000,"available":1073459396608,"internally_reserved":0,"allocated":6356992,"data_stored":3078018,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":262441,"internal_metadata":275644119},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.001206"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":73,"reported_epoch":56,"state":"active+clean","last_fresh":"2026-02-25T12:02:18.903755+0000","last_change":"2026-02-25T12:02:06.883264+0000","last_active":"2026-02-25T12:02:18.903755+0000","last_peered":"2026-02-25T12:02:18.903755+0000","last_clean":"2026-02-25T12:02:18.903755+0000","last_became_active":"2026-02-25T12:02:06.883022+0000","last_became_peered":"2026-02-25T12:02:06.883022+0000","last_unstale":"2026-02-25T12:02:18.903755+0000","last_undegraded":"2026-02-25T12:02:18.903755+0000","last_fullsized":"2026-02-25T12:02:18.903755+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-25T12:01:08.885417+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-25T12:01:08.885417+0000","last_clean_scrub_stamp":"2026-02-25T12:01:08.885417+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-26T22:09:05.780239+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00026608399999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":91,"reported_epoch":56,"state":"active+clean","last_fresh":"2026-02-25T12:02:19.135470+0000","last_change":"2026-02-25T12:02:06.883262+0000","last_active":"2026-02-25T12:02:19.135470+0000","last_peered":"2026-02-25T12:02:19.135470+0000","last_clean":"2026-02-25T12:02:19.135470+0000","last_became_active":"2026-02-25T12:02:06.883003+0000","last_became_peered":"2026-02-25T12:02:06.883003+0000","last_unstale":"2026-02-25T12:02:19.135470+0000","last_undegraded":"2026-02-25T12:02:19.135470+0000","last_fullsized":"2026-02-25T12:02:19.135470+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-25T12:01:08.885417+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-25T12:01:08.885417+0000","last_clean_scrub_stamp":"2026-02-25T12:01:08.885417+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-26T15:40:20.663132+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00030945499999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":107,"reported_epoch":56,"state":"active+clean","last_fresh":"2026-02-25T12:02:19.227443+0000","last_change":"2026-02-25T12:02:19.227382+0000","last_active":"2026-02-25T12:02:19.227443+0000","last_peered":"2026-02-25T12:02:19.227443+0000","last_clean":"2026-02-25T12:02:19.227443+0000","last_became_active":"2026-02-25T12:02:19.227208+0000","last_became_peered":"2026-02-25T12:02:19.227208+0000","last_unstale":"2026-02-25T12:02:19.227443+0000","last_undegraded":"2026-02-25T12:02:19.227443+0000","last_fullsized":"2026-02-25T12:02:19.227443+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-25T12:01:08.885417+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-25T12:01:08.885417+0000","last_clean_scrub_stamp":"2026-02-25T12:01:08.885417+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-26T22:28:26.067831+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00017141899999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0],"acting":[9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":78,"reported_epoch":56,"state":"active+clean","last_fresh":"2026-02-25T12:02:19.136181+0000","last_change":"2026-02-25T12:02:19.136117+0000","last_active":"2026-02-25T12:02:19.136181+0000","last_peered":"2026-02-25T12:02:19.136181+0000","last_clean":"2026-02-25T12:02:19.136181+0000","last_became_active":"2026-02-25T12:02:19.135936+0000","last_became_peered":"2026-02-25T12:02:19.135936+0000","last_unstale":"2026-02-25T12:02:19.136181+0000","last_undegraded":"2026-02-25T12:02:19.136181+0000","last_fullsized":"2026-02-25T12:02:19.136181+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-25T12:01:08.885417+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-25T12:01:08.885417+0000","last_clean_scrub_stamp":"2026-02-25T12:01:08.885417+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-26T17:27:50.663072+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.3","version":"18'1","reported_seq":93,"reported_epoch":56,"state":"active+clean","last_fresh":"2026-02-25T12:02:18.905437+0000","last_change":"2026-02-25T12:02:18.905365+0000","last_active":"2026-02-25T12:02:18.905437+0000","last_peered":"2026-02-25T12:02:18.905437+0000","last_clean":"2026-02-25T12:02:18.905437+0000","last_became_active":"2026-02-25T12:02:18.905150+0000","last_became_peered":"2026-02-25T12:02:18.905150+0000","last_unstale":"2026-02-25T12:02:18.905437+0000","last_undegraded":"2026-02-25T12:02:18.905437+0000","last_fullsized":"2026-02-25T12:02:18.905437+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-25T12:01:08.885417+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-25T12:01:08.885417+0000","last_clean_scrub_stamp":"2026-02-25T12:01:08.885417+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-26T16:20:44.992590+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"20'2","reported_seq":97,"reported_epoch":56,"state":"active+clean","last_fresh":"2026-02-25T12:02:18.905682+0000","last_change":"2026-02-25T12:02:18.905622+0000","last_active":"2026-02-25T12:02:18.905682+0000","last_peered":"2026-02-25T12:02:18.905682+0000","last_clean":"2026-02-25T12:02:18.905682+0000","last_became_active":"2026-02-25T12:02:18.905430+0000","last_became_peered":"2026-02-25T12:02:18.905430+0000","last_unstale":"2026-02-25T12:02:18.905682+0000","last_undegraded":"2026-02-25T12:02:18.905682+0000","last_fullsized":"2026-02-25T12:02:18.905682+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-25T12:01:08.885417+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-25T12:01:08.885417+0000","last_clean_scrub_stamp":"2026-02-25T12:01:08.885417+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-26T22:29:58.049830+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062137499999999997,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":74,"reported_epoch":56,"state":"active+clean","last_fresh":"2026-02-25T12:02:19.135280+0000","last_change":"2026-02-25T12:02:06.883333+0000","last_active":"2026-02-25T12:02:19.135280+0000","last_peered":"2026-02-25T12:02:19.135280+0000","last_clean":"2026-02-25T12:02:19.135280+0000","last_became_active":"2026-02-25T12:02:06.883131+0000","last_became_peered":"2026-02-25T12:02:06.883131+0000","last_unstale":"2026-02-25T12:02:19.135280+0000","last_undegraded":"2026-02-25T12:02:19.135280+0000","last_fullsized":"2026-02-25T12:02:19.135280+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-25T12:01:08.885417+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-25T12:01:08.885417+0000","last_clean_scrub_stamp":"2026-02-25T12:01:08.885417+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-26T17:10:55.006863+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00015265299999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":86,"reported_epoch":56,"state":"active+clean","last_fresh":"2026-02-25T12:02:18.935442+0000","last_change":"2026-02-25T12:02:07.090070+0000","last_active":"2026-02-25T12:02:18.935442+0000","last_peered":"2026-02-25T12:02:18.935442+0000","last_clean":"2026-02-25T12:02:18.935442+0000","last_became_active":"2026-02-25T12:02:07.089906+0000","last_became_peered":"2026-02-25T12:02:07.089906+0000","last_unstale":"2026-02-25T12:02:18.935442+0000","last_undegraded":"2026-02-25T12:02:18.935442+0000","last_fullsized":"2026-02-25T12:02:18.935442+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-25T12:01:08.885417+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-25T12:01:08.885417+0000","last_clean_scrub_stamp":"2026-02-25T12:01:08.885417+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-26T23:02:05.903788+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9],"acting":[5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":9}],"osd_stats":[{"osd":9,"up_from":37,"seq":158913789970,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":27580,"kb_used_data":620,"kb_used_omap":27,"kb_used_meta":26916,"kb_avail":104830020,"statfs":{"total":107374182400,"available":107345940480,"internally_reserved":0,"allocated":634880,"data_stored":307798,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":28398,"internal_metadata":27562258},"hb_peers":[0,1,2,3,4,5,6,7,8],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":43,"seq":184683593747,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":104857600,"kb_used":27580,"kb_used_data":620,"kb_used_omap":22,"kb_used_meta":26921,"kb_avail":104830020,"statfs":{"total":107374182400,"available":107345940480,"internally_reserved":0,"allocated":634880,"data_stored":307798,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":23185,"internal_metadata":27567471},"hb_peers":[0,1,2,3,4,5,6,7,9],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":43,"seq":184683593746,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":27580,"kb_used_data":620,"kb_used_omap":27,"kb_used_meta":26916,"kb_avail":104830020,"statfs":{"total":107374182400,"available":107345940480,"internally_reserved":0,"allocated":634880,"data_stored":307798,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":28387,"internal_metadata":27562269},"hb_peers":[0,1,2,3,4,5,6,8,9],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":43,"seq":184683593746,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":104857600,"kb_used":27580,"kb_used_data":620,"kb_used_omap":27,"kb_used_meta":26916,"kb_avail":104830020,"statfs":{"total":107374182400,"available":107345940480,"internally_reserved":0,"allocated":634880,"data_stored":307798,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":27742,"internal_metadata":27562914},"hb_peers":[0,1,3,4,5,6,7,8,9],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":43,"seq":184683593747,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":27580,"kb_used_data":620,"kb_used_omap":25,"kb_used_meta":26918,"kb_avail":104830020,"statfs":{"total":107374182400,"available":107345940480,"internally_reserved":0,"allocated":634880,"data_stored":307798,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":26438,"internal_metadata":27564218},"hb_peers":[0,1,2,3,4,5,7,8,9],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":43,"seq":184683593747,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":27580,"kb_used_data":620,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":104830020,"statfs":{"total":107374182400,"available":107345940480,"internally_reserved":0,"allocated":634880,"data_stored":307798,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17326,"internal_metadata":27573330},"hb_peers":[0,1,2,3,4,6,7,8,9],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":43,"seq":184683593746,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":27584,"kb_used_data":624,"kb_used_omap":28,"kb_used_meta":26915,"kb_avail":104830016,"statfs":{"total":107374182400,"available":107345936384,"internally_reserved":0,"allocated":638976,"data_stored":307817,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":29044,"internal_metadata":27561612},"hb_peers":[0,1,2,3,5,6,7,8,9],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":43,"seq":184683593748,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":27580,"kb_used_data":620,"kb_used_omap":27,"kb_used_meta":26916,"kb_avail":104830020,"statfs":{"total":107374182400,"available":107345940480,"internally_reserved":0,"allocated":634880,"data_stored":307798,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":28390,"internal_metadata":27562266},"hb_peers":[0,1,2,4,5,6,7,8,9],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":55,"seq":236223201299,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":27584,"kb_used_data":624,"kb_used_omap":25,"kb_used_meta":26918,"kb_avail":104830016,"statfs":{"total":107374182400,"available":107345936384,"internally_reserved":0,"allocated":638976,"data_stored":307817,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":25789,"internal_metadata":27564867},"hb_peers":[0,2,3,4,5,6,7,8,9],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":55,"seq":236223201299,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":27580,"kb_used_data":620,"kb_used_omap":27,"kb_used_meta":26916,"kb_avail":104830020,"statfs":{"total":107374182400,"available":107345940480,"internally_reserved":0,"allocated":634880,"data_stored":307798,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":27742,"internal_metadata":27562914},"hb_peers":[1,2,3,4,5,6,7,8,9],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-25T12:02:30.499 INFO:tasks.ceph:pgid 1.7 last_scrub_stamp 2026-02-25T12:01:08.885417+0000 time.struct_time(tm_year=2026, tm_mon=2, tm_mday=25, tm_hour=12, tm_min=1, tm_sec=8, tm_wday=2, tm_yday=56, tm_isdst=-1) <= time.struct_time(tm_year=2026, tm_mon=2, tm_mday=25, tm_hour=12, tm_min=2, tm_sec=26, tm_wday=2, tm_yday=56, tm_isdst=0) 2026-02-25T12:02:30.500 INFO:tasks.ceph:pgid 1.6 last_scrub_stamp 2026-02-25T12:01:08.885417+0000 time.struct_time(tm_year=2026, tm_mon=2, tm_mday=25, tm_hour=12, tm_min=1, tm_sec=8, tm_wday=2, tm_yday=56, tm_isdst=-1) <= time.struct_time(tm_year=2026, tm_mon=2, tm_mday=25, tm_hour=12, tm_min=2, tm_sec=26, tm_wday=2, tm_yday=56, tm_isdst=0) 2026-02-25T12:02:30.500 INFO:tasks.ceph:pgid 1.5 last_scrub_stamp 2026-02-25T12:01:08.885417+0000 time.struct_time(tm_year=2026, tm_mon=2, tm_mday=25, tm_hour=12, tm_min=1, tm_sec=8, tm_wday=2, tm_yday=56, tm_isdst=-1) <= time.struct_time(tm_year=2026, tm_mon=2, tm_mday=25, tm_hour=12, tm_min=2, tm_sec=26, tm_wday=2, tm_yday=56, tm_isdst=0) 2026-02-25T12:02:30.500 INFO:tasks.ceph:pgid 1.4 last_scrub_stamp 2026-02-25T12:01:08.885417+0000 time.struct_time(tm_year=2026, tm_mon=2, tm_mday=25, tm_hour=12, tm_min=1, tm_sec=8, tm_wday=2, tm_yday=56, tm_isdst=-1) <= time.struct_time(tm_year=2026, tm_mon=2, tm_mday=25, tm_hour=12, tm_min=2, tm_sec=26, tm_wday=2, tm_yday=56, tm_isdst=0) 2026-02-25T12:02:30.500 INFO:tasks.ceph:pgid 1.3 last_scrub_stamp 2026-02-25T12:01:08.885417+0000 time.struct_time(tm_year=2026, tm_mon=2, tm_mday=25, tm_hour=12, tm_min=1, tm_sec=8, tm_wday=2, tm_yday=56, tm_isdst=-1) <= time.struct_time(tm_year=2026, tm_mon=2, tm_mday=25, tm_hour=12, tm_min=2, tm_sec=26, tm_wday=2, tm_yday=56, tm_isdst=0) 2026-02-25T12:02:30.500 INFO:tasks.ceph:pgid 1.2 last_scrub_stamp 2026-02-25T12:01:08.885417+0000 time.struct_time(tm_year=2026, tm_mon=2, tm_mday=25, tm_hour=12, tm_min=1, tm_sec=8, tm_wday=2, tm_yday=56, tm_isdst=-1) <= time.struct_time(tm_year=2026, tm_mon=2, tm_mday=25, tm_hour=12, tm_min=2, tm_sec=26, tm_wday=2, tm_yday=56, tm_isdst=0) 2026-02-25T12:02:30.500 INFO:tasks.ceph:pgid 1.1 last_scrub_stamp 2026-02-25T12:01:08.885417+0000 time.struct_time(tm_year=2026, tm_mon=2, tm_mday=25, tm_hour=12, tm_min=1, tm_sec=8, tm_wday=2, tm_yday=56, tm_isdst=-1) <= time.struct_time(tm_year=2026, tm_mon=2, tm_mday=25, tm_hour=12, tm_min=2, tm_sec=26, tm_wday=2, tm_yday=56, tm_isdst=0) 2026-02-25T12:02:30.501 INFO:tasks.ceph:pgid 1.0 last_scrub_stamp 2026-02-25T12:01:08.885417+0000 time.struct_time(tm_year=2026, tm_mon=2, tm_mday=25, tm_hour=12, tm_min=1, tm_sec=8, tm_wday=2, tm_yday=56, tm_isdst=-1) <= time.struct_time(tm_year=2026, tm_mon=2, tm_mday=25, tm_hour=12, tm_min=2, tm_sec=26, tm_wday=2, tm_yday=56, tm_isdst=0) 2026-02-25T12:02:30.501 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2026-02-25T12:02:50.502 DEBUG:teuthology.orchestra.run.trial117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-25T12:02:50.659 INFO:teuthology.orchestra.run.trial117.stdout: 2026-02-25T12:02:50.659 INFO:teuthology.orchestra.run.trial117.stderr:dumped all 2026-02-25T12:02:50.670 INFO:teuthology.orchestra.run.trial117.stdout:{"pg_ready":true,"pg_map":{"version":108,"stamp":"2026-02-25T12:02:48.738157+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":5,"ondisk_log_size":5,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":10,"num_per_pool_osds":10,"num_per_pool_omap_osds":8,"kb":1048576000,"kb_used":275808,"kb_used_data":6208,"kb_used_omap":256,"kb_used_meta":269183,"kb_avail":1048300192,"statfs":{"total":1073741824000,"available":1073459396608,"internally_reserved":0,"allocated":6356992,"data_stored":3078018,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":262441,"internal_metadata":275644119},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001163"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":81,"reported_epoch":56,"state":"active+clean","last_fresh":"2026-02-25T12:02:28.837211+0000","last_change":"2026-02-25T12:02:28.837211+0000","last_active":"2026-02-25T12:02:28.837211+0000","last_peered":"2026-02-25T12:02:28.837211+0000","last_clean":"2026-02-25T12:02:28.837211+0000","last_became_active":"2026-02-25T12:02:06.883022+0000","last_became_peered":"2026-02-25T12:02:06.883022+0000","last_unstale":"2026-02-25T12:02:28.837211+0000","last_undegraded":"2026-02-25T12:02:28.837211+0000","last_fullsized":"2026-02-25T12:02:28.837211+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-25T12:02:28.837162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-25T12:02:28.837162+0000","last_clean_scrub_stamp":"2026-02-25T12:02:28.837162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-26T17:24:49.569059+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00026608399999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":99,"reported_epoch":56,"state":"active+clean","last_fresh":"2026-02-25T12:02:29.650999+0000","last_change":"2026-02-25T12:02:29.650999+0000","last_active":"2026-02-25T12:02:29.650999+0000","last_peered":"2026-02-25T12:02:29.650999+0000","last_clean":"2026-02-25T12:02:29.650999+0000","last_became_active":"2026-02-25T12:02:06.883003+0000","last_became_peered":"2026-02-25T12:02:06.883003+0000","last_unstale":"2026-02-25T12:02:29.650999+0000","last_undegraded":"2026-02-25T12:02:29.650999+0000","last_fullsized":"2026-02-25T12:02:29.650999+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-25T12:02:29.650951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-25T12:02:29.650951+0000","last_clean_scrub_stamp":"2026-02-25T12:02:29.650951+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-26T15:06:01.726169+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00030945499999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":115,"reported_epoch":56,"state":"active+clean","last_fresh":"2026-02-25T12:02:31.124754+0000","last_change":"2026-02-25T12:02:31.124754+0000","last_active":"2026-02-25T12:02:31.124754+0000","last_peered":"2026-02-25T12:02:31.124754+0000","last_clean":"2026-02-25T12:02:31.124754+0000","last_became_active":"2026-02-25T12:02:19.227208+0000","last_became_peered":"2026-02-25T12:02:19.227208+0000","last_unstale":"2026-02-25T12:02:31.124754+0000","last_undegraded":"2026-02-25T12:02:31.124754+0000","last_fullsized":"2026-02-25T12:02:31.124754+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-25T12:02:31.124714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-25T12:02:31.124714+0000","last_clean_scrub_stamp":"2026-02-25T12:02:31.124714+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-26T16:22:07.231887+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00017141899999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0],"acting":[9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":86,"reported_epoch":56,"state":"active+clean","last_fresh":"2026-02-25T12:02:28.432723+0000","last_change":"2026-02-25T12:02:28.432723+0000","last_active":"2026-02-25T12:02:28.432723+0000","last_peered":"2026-02-25T12:02:28.432723+0000","last_clean":"2026-02-25T12:02:28.432723+0000","last_became_active":"2026-02-25T12:02:19.135936+0000","last_became_peered":"2026-02-25T12:02:19.135936+0000","last_unstale":"2026-02-25T12:02:28.432723+0000","last_undegraded":"2026-02-25T12:02:28.432723+0000","last_fullsized":"2026-02-25T12:02:28.432723+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-25T12:02:28.432663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-25T12:02:28.432663+0000","last_clean_scrub_stamp":"2026-02-25T12:02:28.432663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2026-02-26T21:49:15.138678+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.3","version":"56'2","reported_seq":102,"reported_epoch":56,"state":"active+clean","last_fresh":"2026-02-25T12:02:29.797510+0000","last_change":"2026-02-25T12:02:29.797510+0000","last_active":"2026-02-25T12:02:29.797510+0000","last_peered":"2026-02-25T12:02:29.797510+0000","last_clean":"2026-02-25T12:02:29.797510+0000","last_became_active":"2026-02-25T12:02:18.905150+0000","last_became_peered":"2026-02-25T12:02:18.905150+0000","last_unstale":"2026-02-25T12:02:29.797510+0000","last_undegraded":"2026-02-25T12:02:29.797510+0000","last_fullsized":"2026-02-25T12:02:29.797510+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"56'2","last_scrub_stamp":"2026-02-25T12:02:29.797470+0000","last_deep_scrub":"56'2","last_deep_scrub_stamp":"2026-02-25T12:02:29.797470+0000","last_clean_scrub_stamp":"2026-02-25T12:02:29.797470+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2026-02-26T13:26:03.085790+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"56'3","reported_seq":106,"reported_epoch":56,"state":"active+clean","last_fresh":"2026-02-25T12:02:29.471388+0000","last_change":"2026-02-25T12:02:29.471388+0000","last_active":"2026-02-25T12:02:29.471388+0000","last_peered":"2026-02-25T12:02:29.471388+0000","last_clean":"2026-02-25T12:02:29.471388+0000","last_became_active":"2026-02-25T12:02:18.905430+0000","last_became_peered":"2026-02-25T12:02:18.905430+0000","last_unstale":"2026-02-25T12:02:29.471388+0000","last_undegraded":"2026-02-25T12:02:29.471388+0000","last_fullsized":"2026-02-25T12:02:29.471388+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"56'3","last_scrub_stamp":"2026-02-25T12:02:29.471344+0000","last_deep_scrub":"56'3","last_deep_scrub_stamp":"2026-02-25T12:02:29.471344+0000","last_clean_scrub_stamp":"2026-02-25T12:02:29.471344+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-26T23:36:25.811287+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062137499999999997,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":82,"reported_epoch":56,"state":"active+clean","last_fresh":"2026-02-25T12:02:30.169577+0000","last_change":"2026-02-25T12:02:30.169577+0000","last_active":"2026-02-25T12:02:30.169577+0000","last_peered":"2026-02-25T12:02:30.169577+0000","last_clean":"2026-02-25T12:02:30.169577+0000","last_became_active":"2026-02-25T12:02:06.883131+0000","last_became_peered":"2026-02-25T12:02:06.883131+0000","last_unstale":"2026-02-25T12:02:30.169577+0000","last_undegraded":"2026-02-25T12:02:30.169577+0000","last_fullsized":"2026-02-25T12:02:30.169577+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-25T12:02:30.169525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-25T12:02:30.169525+0000","last_clean_scrub_stamp":"2026-02-25T12:02:30.169525+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-26T15:43:27.977966+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00015265299999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":94,"reported_epoch":56,"state":"active+clean","last_fresh":"2026-02-25T12:02:29.893396+0000","last_change":"2026-02-25T12:02:29.893396+0000","last_active":"2026-02-25T12:02:29.893396+0000","last_peered":"2026-02-25T12:02:29.893396+0000","last_clean":"2026-02-25T12:02:29.893396+0000","last_became_active":"2026-02-25T12:02:07.089906+0000","last_became_peered":"2026-02-25T12:02:07.089906+0000","last_unstale":"2026-02-25T12:02:29.893396+0000","last_undegraded":"2026-02-25T12:02:29.893396+0000","last_fullsized":"2026-02-25T12:02:29.893396+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-25T12:02:29.893325+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-25T12:02:29.893325+0000","last_clean_scrub_stamp":"2026-02-25T12:02:29.893325+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-26T17:31:45.858838+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9],"acting":[5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":5,"ondisk_log_size":5,"up":16,"acting":16,"num_store_stats":9}],"osd_stats":[{"osd":9,"up_from":37,"seq":158913789974,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":27580,"kb_used_data":620,"kb_used_omap":27,"kb_used_meta":26916,"kb_avail":104830020,"statfs":{"total":107374182400,"available":107345940480,"internally_reserved":0,"allocated":634880,"data_stored":307798,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":28398,"internal_metadata":27562258},"hb_peers":[0,1,2,3,4,5,6,7,8],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":43,"seq":184683593751,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":104857600,"kb_used":27580,"kb_used_data":620,"kb_used_omap":22,"kb_used_meta":26921,"kb_avail":104830020,"statfs":{"total":107374182400,"available":107345940480,"internally_reserved":0,"allocated":634880,"data_stored":307798,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":23185,"internal_metadata":27567471},"hb_peers":[0,1,2,3,4,5,6,7,9],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":43,"seq":184683593750,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":27580,"kb_used_data":620,"kb_used_omap":27,"kb_used_meta":26916,"kb_avail":104830020,"statfs":{"total":107374182400,"available":107345940480,"internally_reserved":0,"allocated":634880,"data_stored":307798,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":28387,"internal_metadata":27562269},"hb_peers":[0,1,2,3,4,5,6,8,9],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":43,"seq":184683593750,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":104857600,"kb_used":27580,"kb_used_data":620,"kb_used_omap":27,"kb_used_meta":26916,"kb_avail":104830020,"statfs":{"total":107374182400,"available":107345940480,"internally_reserved":0,"allocated":634880,"data_stored":307798,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":27742,"internal_metadata":27562914},"hb_peers":[0,1,3,4,5,6,7,8,9],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":43,"seq":184683593751,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":27580,"kb_used_data":620,"kb_used_omap":25,"kb_used_meta":26918,"kb_avail":104830020,"statfs":{"total":107374182400,"available":107345940480,"internally_reserved":0,"allocated":634880,"data_stored":307798,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":26438,"internal_metadata":27564218},"hb_peers":[0,1,2,3,4,5,7,8,9],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":43,"seq":184683593751,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":27580,"kb_used_data":620,"kb_used_omap":16,"kb_used_meta":26927,"kb_avail":104830020,"statfs":{"total":107374182400,"available":107345940480,"internally_reserved":0,"allocated":634880,"data_stored":307798,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17326,"internal_metadata":27573330},"hb_peers":[0,1,2,3,4,6,7,8,9],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":43,"seq":184683593750,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":27584,"kb_used_data":624,"kb_used_omap":28,"kb_used_meta":26915,"kb_avail":104830016,"statfs":{"total":107374182400,"available":107345936384,"internally_reserved":0,"allocated":638976,"data_stored":307817,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":29044,"internal_metadata":27561612},"hb_peers":[0,1,2,3,5,6,7,8,9],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":43,"seq":184683593752,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":27580,"kb_used_data":620,"kb_used_omap":27,"kb_used_meta":26916,"kb_avail":104830020,"statfs":{"total":107374182400,"available":107345940480,"internally_reserved":0,"allocated":634880,"data_stored":307798,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":28390,"internal_metadata":27562266},"hb_peers":[0,1,2,4,5,6,7,8,9],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":55,"seq":236223201303,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":27584,"kb_used_data":624,"kb_used_omap":25,"kb_used_meta":26918,"kb_avail":104830016,"statfs":{"total":107374182400,"available":107345936384,"internally_reserved":0,"allocated":638976,"data_stored":307817,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":25789,"internal_metadata":27564867},"hb_peers":[0,2,3,4,5,6,7,8,9],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":55,"seq":236223201303,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":27580,"kb_used_data":620,"kb_used_omap":27,"kb_used_meta":26916,"kb_avail":104830020,"statfs":{"total":107374182400,"available":107345940480,"internally_reserved":0,"allocated":634880,"data_stored":307798,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":27742,"internal_metadata":27562914},"hb_peers":[1,2,3,4,5,6,7,8,9],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-25T12:02:50.671 DEBUG:teuthology.orchestra.run.trial117:> sudo ceph --cluster ceph config set global mon_health_to_clog false 2026-02-25T12:02:50.837 INFO:teuthology.misc:Shutting down mds daemons... 2026-02-25T12:02:50.837 INFO:teuthology.misc:Shutting down osd daemons... 2026-02-25T12:02:50.837 DEBUG:tasks.ceph.osd.0:waiting for process to exit 2026-02-25T12:02:50.837 INFO:teuthology.orchestra.run:waiting for 300 2026-02-25T12:02:50.909 INFO:tasks.ceph.osd.0:Stopped 2026-02-25T12:02:50.910 DEBUG:tasks.ceph.osd.1:waiting for process to exit 2026-02-25T12:02:50.910 INFO:teuthology.orchestra.run:waiting for 300 2026-02-25T12:02:50.993 INFO:tasks.ceph.osd.1:Stopped 2026-02-25T12:02:50.994 DEBUG:tasks.ceph.osd.2:waiting for process to exit 2026-02-25T12:02:50.994 INFO:teuthology.orchestra.run:waiting for 300 2026-02-25T12:02:51.061 INFO:tasks.ceph.osd.2:Stopped 2026-02-25T12:02:51.061 DEBUG:tasks.ceph.osd.3:waiting for process to exit 2026-02-25T12:02:51.061 INFO:teuthology.orchestra.run:waiting for 300 2026-02-25T12:02:51.141 INFO:tasks.ceph.osd.3:Stopped 2026-02-25T12:02:51.141 DEBUG:tasks.ceph.osd.4:waiting for process to exit 2026-02-25T12:02:51.141 INFO:teuthology.orchestra.run:waiting for 300 2026-02-25T12:02:51.213 INFO:tasks.ceph.osd.4:Stopped 2026-02-25T12:02:51.213 DEBUG:tasks.ceph.osd.5:waiting for process to exit 2026-02-25T12:02:51.213 INFO:teuthology.orchestra.run:waiting for 300 2026-02-25T12:02:51.293 INFO:tasks.ceph.osd.5:Stopped 2026-02-25T12:02:51.293 DEBUG:tasks.ceph.osd.6:waiting for process to exit 2026-02-25T12:02:51.293 INFO:teuthology.orchestra.run:waiting for 300 2026-02-25T12:02:51.372 INFO:tasks.ceph.osd.6:Stopped 2026-02-25T12:02:51.372 DEBUG:tasks.ceph.osd.7:waiting for process to exit 2026-02-25T12:02:51.372 INFO:teuthology.orchestra.run:waiting for 300 2026-02-25T12:02:51.464 INFO:tasks.ceph.osd.7:Stopped 2026-02-25T12:02:51.465 DEBUG:tasks.ceph.osd.8:waiting for process to exit 2026-02-25T12:02:51.465 INFO:teuthology.orchestra.run:waiting for 300 2026-02-25T12:02:51.532 INFO:tasks.ceph.osd.8:Stopped 2026-02-25T12:02:51.533 DEBUG:tasks.ceph.osd.9:waiting for process to exit 2026-02-25T12:02:51.533 INFO:teuthology.orchestra.run:waiting for 300 2026-02-25T12:02:51.632 INFO:tasks.ceph.osd.9:Stopped 2026-02-25T12:02:51.632 INFO:teuthology.misc:Shutting down mgr daemons... 2026-02-25T12:02:51.632 DEBUG:tasks.ceph.mgr.x:waiting for process to exit 2026-02-25T12:02:51.632 INFO:teuthology.orchestra.run:waiting for 300 2026-02-25T12:02:51.647 INFO:tasks.ceph.mgr.x:Stopped 2026-02-25T12:02:51.647 INFO:teuthology.misc:Shutting down mon daemons... 2026-02-25T12:02:51.647 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2026-02-25T12:02:51.647 INFO:teuthology.orchestra.run:waiting for 300 2026-02-25T12:02:51.704 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.4 is failed for ~0s 2026-02-25T12:02:51.705 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.5 is failed for ~0s 2026-02-25T12:02:51.705 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.6 is failed for ~0s 2026-02-25T12:02:51.705 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.7 is failed for ~0s 2026-02-25T12:02:51.705 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.8 is failed for ~0s 2026-02-25T12:02:51.705 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.9 is failed for ~0s 2026-02-25T12:02:51.705 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.a is failed for ~0s 2026-02-25T12:02:51.705 INFO:tasks.ceph.mon.a:Stopped 2026-02-25T12:02:51.705 INFO:tasks.ceph:Checking cluster log for badness... 2026-02-25T12:02:51.705 DEBUG:teuthology.orchestra.run.trial117:> sudo grep -E '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/ceph.log | grep -E -v 'but it is still running' | grep -E -v 'overall HEALTH_' | grep -E -v '\(OSDMAP_FLAGS\)' | grep -E -v '\(OSD_' | grep -E -v '\(PG_' | grep -E -v '\(POOL_APP_NOT_ENABLED\)' | grep -E -v '\(MDS_ALL_DOWN\)' | grep -E -v '\(MDS_UP_LESS_THAN_MAX\)' | head -n 1 2026-02-25T12:02:51.758 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-0 on ubuntu@trial117.front.sepia.ceph.com 2026-02-25T12:02:51.759 DEBUG:teuthology.orchestra.run.trial117:> sync && sudo umount -f /var/lib/ceph/osd/ceph-0 2026-02-25T12:02:51.812 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-1 on ubuntu@trial117.front.sepia.ceph.com 2026-02-25T12:02:51.812 DEBUG:teuthology.orchestra.run.trial117:> sync && sudo umount -f /var/lib/ceph/osd/ceph-1 2026-02-25T12:02:51.863 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-2 on ubuntu@trial117.front.sepia.ceph.com 2026-02-25T12:02:51.863 DEBUG:teuthology.orchestra.run.trial117:> sync && sudo umount -f /var/lib/ceph/osd/ceph-2 2026-02-25T12:02:51.914 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-3 on ubuntu@trial117.front.sepia.ceph.com 2026-02-25T12:02:51.914 DEBUG:teuthology.orchestra.run.trial117:> sync && sudo umount -f /var/lib/ceph/osd/ceph-3 2026-02-25T12:02:51.967 INFO:tasks.ceph:Archiving mon data... 2026-02-25T12:02:51.968 DEBUG:teuthology.misc:Transferring archived files from trial117:/var/lib/ceph/mon/ceph-a to /home/teuthworker/mnt/teuthology/nmordech-2026-02-25_11:36:23-rados-wip-rocky10-branch-of-the-day-2026-02-24-1771941190-distro-default-trial/69872/data/mon.a.tgz 2026-02-25T12:02:51.968 DEBUG:teuthology.orchestra.run.trial117:> mktemp 2026-02-25T12:02:52.012 INFO:teuthology.orchestra.run.trial117.stdout:/tmp/tmp.1PN87sbKZE 2026-02-25T12:02:52.013 DEBUG:teuthology.orchestra.run.trial117:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-a -- . > /tmp/tmp.1PN87sbKZE 2026-02-25T12:02:52.087 DEBUG:teuthology.orchestra.run.trial117:> sudo chmod 0666 /tmp/tmp.1PN87sbKZE 2026-02-25T12:02:52.185 DEBUG:teuthology.orchestra.remote:trial117:/tmp/tmp.1PN87sbKZE is 213KB 2026-02-25T12:02:52.234 DEBUG:teuthology.orchestra.run.trial117:> rm -fr /tmp/tmp.1PN87sbKZE 2026-02-25T12:02:52.237 INFO:tasks.ceph:Cleaning ceph cluster... 2026-02-25T12:02:52.237 DEBUG:teuthology.orchestra.run.trial117:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2026-02-25T12:02:52.333 INFO:teuthology.util.scanner:summary_data or yaml_file is empty! 2026-02-25T12:02:52.333 INFO:tasks.ceph:Archiving crash dumps... 2026-02-25T12:02:52.335 DEBUG:teuthology.misc:Transferring archived files from trial117:/var/lib/ceph/crash to /home/teuthworker/mnt/teuthology/nmordech-2026-02-25_11:36:23-rados-wip-rocky10-branch-of-the-day-2026-02-24-1771941190-distro-default-trial/69872/remote/trial117/crash 2026-02-25T12:02:52.336 DEBUG:teuthology.orchestra.run.trial117:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2026-02-25T12:02:52.385 INFO:tasks.ceph:Compressing logs... 2026-02-25T12:02:52.385 DEBUG:teuthology.orchestra.run.trial117:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2026-02-25T12:02:52.392 INFO:teuthology.orchestra.run.trial117.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26146.log 2026-02-25T12:02:52.393 INFO:teuthology.orchestra.run.trial117.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26220.log 2026-02-25T12:02:52.393 INFO:teuthology.orchestra.run.trial117.stderr:/var/log/ceph/ceph-client.admin.26146.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27327.log 2026-02-25T12:02:52.393 INFO:teuthology.orchestra.run.trial117.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.26146.log.gz 2026-02-25T12:02:52.393 INFO:teuthology.orchestra.run.trial117.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26394.log 2026-02-25T12:02:52.394 INFO:teuthology.orchestra.run.trial117.stderr:/var/log/ceph/ceph-client.admin.26220.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.24661.log 2026-02-25T12:02:52.394 INFO:teuthology.orchestra.run.trial117.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.26220.log.gz 2026-02-25T12:02:52.394 INFO:teuthology.orchestra.run.trial117.stderr:/var/log/ceph/ceph-client.admin.27327.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.27327.log.gz 2026-02-25T12:02:52.394 INFO:teuthology.orchestra.run.trial117.stderr:/var/log/ceph/ceph-client.admin.26394.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.19586.log 2026-02-25T12:02:52.394 INFO:teuthology.orchestra.run.trial117.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.26394.log.gz 2026-02-25T12:02:52.394 INFO:teuthology.orchestra.run.trial117.stderr:/var/log/ceph/ceph-client.admin.24661.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.24661.log.gz 2026-02-25T12:02:52.394 INFO:teuthology.orchestra.run.trial117.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.25625.log 2026-02-25T12:02:52.395 INFO:teuthology.orchestra.run.trial117.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26244.log 2026-02-25T12:02:52.395 INFO:teuthology.orchestra.run.trial117.stderr:/var/log/ceph/ceph-client.admin.19586.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.19586.log.gz 2026-02-25T12:02:52.395 INFO:teuthology.orchestra.run.trial117.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.19735.log 2026-02-25T12:02:52.395 INFO:teuthology.orchestra.run.trial117.stderr:/var/log/ceph/ceph-client.admin.25625.log: /var/log/ceph/ceph-client.admin.26244.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.25625.log.gz 2026-02-25T12:02:52.395 INFO:teuthology.orchestra.run.trial117.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.26244.log.gz 2026-02-25T12:02:52.395 INFO:teuthology.orchestra.run.trial117.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26521.log 2026-02-25T12:02:52.396 INFO:teuthology.orchestra.run.trial117.stderr:/var/log/ceph/ceph-client.admin.19735.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.25846.log 2026-02-25T12:02:52.396 INFO:teuthology.orchestra.run.trial117.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.19735.log.gz 2026-02-25T12:02:52.396 INFO:teuthology.orchestra.run.trial117.stderr:/var/log/ceph/ceph-client.admin.26521.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.26521.log.gz 2026-02-25T12:02:52.396 INFO:teuthology.orchestra.run.trial117.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.19831.log 2026-02-25T12:02:52.396 INFO:teuthology.orchestra.run.trial117.stderr:/var/log/ceph/ceph-client.admin.25846.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26989.log 2026-02-25T12:02:52.396 INFO:teuthology.orchestra.run.trial117.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.25846.log.gz 2026-02-25T12:02:52.396 INFO:teuthology.orchestra.run.trial117.stderr:/var/log/ceph/ceph-client.admin.19831.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.19831.log.gz 2026-02-25T12:02:52.396 INFO:teuthology.orchestra.run.trial117.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27433.log 2026-02-25T12:02:52.397 INFO:teuthology.orchestra.run.trial117.stderr:/var/log/ceph/ceph-client.admin.26989.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26729.log 2026-02-25T12:02:52.397 INFO:teuthology.orchestra.run.trial117.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.26989.log.gz 2026-02-25T12:02:52.397 INFO:teuthology.orchestra.run.trial117.stderr:/var/log/ceph/ceph-client.admin.27433.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.27433.log.gz 2026-02-25T12:02:52.397 INFO:teuthology.orchestra.run.trial117.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.25895.log 2026-02-25T12:02:52.397 INFO:teuthology.orchestra.run.trial117.stderr:/var/log/ceph/ceph-client.admin.26729.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27355.log 2026-02-25T12:02:52.397 INFO:teuthology.orchestra.run.trial117.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.26729.log.gz 2026-02-25T12:02:52.397 INFO:teuthology.orchestra.run.trial117.stderr:/var/log/ceph/ceph-client.admin.25895.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.25895.log.gz 2026-02-25T12:02:52.398 INFO:teuthology.orchestra.run.trial117.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26595.log 2026-02-25T12:02:52.398 INFO:teuthology.orchestra.run.trial117.stderr:/var/log/ceph/ceph-client.admin.27355.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26807.log 2026-02-25T12:02:52.398 INFO:teuthology.orchestra.run.trial117.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.27355.log.gz 2026-02-25T12:02:52.398 INFO:teuthology.orchestra.run.trial117.stderr:/var/log/ceph/ceph-client.admin.26595.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27563.log 2026-02-25T12:02:52.398 INFO:teuthology.orchestra.run.trial117.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.26595.log.gz 2026-02-25T12:02:52.398 INFO:teuthology.orchestra.run.trial117.stderr:/var/log/ceph/ceph-client.admin.26807.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.26807.log.gz 2026-02-25T12:02:52.398 INFO:teuthology.orchestra.run.trial117.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.19936.log 2026-02-25T12:02:52.399 INFO:teuthology.orchestra.run.trial117.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27485.log 2026-02-25T12:02:52.399 INFO:teuthology.orchestra.run.trial117.stderr:/var/log/ceph/ceph-client.admin.27563.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.27563.log.gz 2026-02-25T12:02:52.399 INFO:teuthology.orchestra.run.trial117.stderr:/var/log/ceph/ceph-client.admin.19936.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27093.log 2026-02-25T12:02:52.399 INFO:teuthology.orchestra.run.trial117.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.19936.log.gz 2026-02-25T12:02:52.399 INFO:teuthology.orchestra.run.trial117.stderr:/var/log/ceph/ceph-client.admin.27485.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.25062.log 2026-02-25T12:02:52.400 INFO:teuthology.orchestra.run.trial117.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.27485.log.gz 2026-02-25T12:02:52.400 INFO:teuthology.orchestra.run.trial117.stderr:/var/log/ceph/ceph-client.admin.27093.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.3.log 2026-02-25T12:02:52.400 INFO:teuthology.orchestra.run.trial117.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.27093.log.gz 2026-02-25T12:02:52.400 INFO:teuthology.orchestra.run.trial117.stderr:/var/log/ceph/ceph-client.admin.25062.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.25797.log 2026-02-25T12:02:52.400 INFO:teuthology.orchestra.run.trial117.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.25062.log.gz 2026-02-25T12:02:52.400 INFO:teuthology.orchestra.run.trial117.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27067.log 2026-02-25T12:02:52.401 INFO:teuthology.orchestra.run.trial117.stderr:/var/log/ceph/ceph-osd.3.log: /var/log/ceph/ceph-client.admin.25797.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.24641.log 2026-02-25T12:02:52.401 INFO:teuthology.orchestra.run.trial117.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.25797.log.gz 2026-02-25T12:02:52.401 INFO:teuthology.orchestra.run.trial117.stderr:/var/log/ceph/ceph-client.admin.27067.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.27067.log.gz 2026-02-25T12:02:52.401 INFO:teuthology.orchestra.run.trial117.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26859.log 2026-02-25T12:02:52.401 INFO:teuthology.orchestra.run.trial117.stderr:/var/log/ceph/ceph-client.admin.24641.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.24788.log 2026-02-25T12:02:52.401 INFO:teuthology.orchestra.run.trial117.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.24641.log.gz 2026-02-25T12:02:52.402 INFO:teuthology.orchestra.run.trial117.stderr:/var/log/ceph/ceph-client.admin.26859.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.24799.log 2026-02-25T12:02:52.402 INFO:teuthology.orchestra.run.trial117.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.26859.log.gz 2026-02-25T12:02:52.402 INFO:teuthology.orchestra.run.trial117.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27015.log 2026-02-25T12:02:52.402 INFO:teuthology.orchestra.run.trial117.stderr:/var/log/ceph/ceph-client.admin.24788.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.24788.log.gz 2026-02-25T12:02:52.402 INFO:teuthology.orchestra.run.trial117.stderr:/var/log/ceph/ceph-client.admin.24799.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.24799.log.gz 2026-02-25T12:02:52.402 INFO:teuthology.orchestra.run.trial117.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26833.log 2026-02-25T12:02:52.402 INFO:teuthology.orchestra.run.trial117.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.19879.log 2026-02-25T12:02:52.403 INFO:teuthology.orchestra.run.trial117.stderr:/var/log/ceph/ceph-client.admin.27015.log: /var/log/ceph/ceph-client.admin.26833.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.27015.log.gz 2026-02-25T12:02:52.403 INFO:teuthology.orchestra.run.trial117.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.26833.log.gz 2026-02-25T12:02:52.403 INFO:teuthology.orchestra.run.trial117.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.5.log 2026-02-25T12:02:52.403 INFO:teuthology.orchestra.run.trial117.stderr:/var/log/ceph/ceph-client.admin.19879.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26570.log 2026-02-25T12:02:52.403 INFO:teuthology.orchestra.run.trial117.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.19879.log.gz 2026-02-25T12:02:52.403 INFO:teuthology.orchestra.run.trial117.stderr:/var/log/ceph/ceph-osd.5.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.19807.log 2026-02-25T12:02:52.404 INFO:teuthology.orchestra.run.trial117.stderr:gzip/var/log/ceph/ceph-client.admin.26570.log: -5 --verbose -- /var/log/ceph/ceph-mgr.x.log 2026-02-25T12:02:52.404 INFO:teuthology.orchestra.run.trial117.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.26570.log.gz 2026-02-25T12:02:52.404 INFO:teuthology.orchestra.run.trial117.stderr:/var/log/ceph/ceph-client.admin.19807.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.19807.log.gz 2026-02-25T12:02:52.404 INFO:teuthology.orchestra.run.trial117.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.9.log 2026-02-25T12:02:52.404 INFO:teuthology.orchestra.run.trial117.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.6.log 2026-02-25T12:02:52.404 INFO:teuthology.orchestra.run.trial117.stderr:/var/log/ceph/ceph-mgr.x.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.25044.log 2026-02-25T12:02:52.405 INFO:teuthology.orchestra.run.trial117.stderr:/var/log/ceph/ceph-osd.9.log: /var/log/ceph/ceph-osd.6.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.25297.log 2026-02-25T12:02:52.405 INFO:teuthology.orchestra.run.trial117.stderr:/var/log/ceph/ceph-client.admin.25044.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.19633.log 2026-02-25T12:02:52.405 INFO:teuthology.orchestra.run.trial117.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.25044.log.gz 2026-02-25T12:02:52.405 INFO:teuthology.orchestra.run.trial117.stderr:/var/log/ceph/ceph-client.admin.25297.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26911.log 2026-02-25T12:02:52.406 INFO:teuthology.orchestra.run.trial117.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.25297.log.gz 2026-02-25T12:02:52.406 INFO:teuthology.orchestra.run.trial117.stderr:/var/log/ceph/ceph-client.admin.19633.log: gzip -5 --verbose -- /var/log/ceph/ceph.tmp-client.admin.11462.log 2026-02-25T12:02:52.406 INFO:teuthology.orchestra.run.trial117.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.19633.log.gz 2026-02-25T12:02:52.406 INFO:teuthology.orchestra.run.trial117.stderr:/var/log/ceph/ceph-client.admin.26911.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27407.log 2026-02-25T12:02:52.406 INFO:teuthology.orchestra.run.trial117.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.26911.log.gz 2026-02-25T12:02:52.407 INFO:teuthology.orchestra.run.trial117.stderr:/var/log/ceph/ceph.tmp-client.admin.11462.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.25649.log 2026-02-25T12:02:52.407 INFO:teuthology.orchestra.run.trial117.stderr: 0.0% -- replaced with /var/log/ceph/ceph.tmp-client.admin.11462.log.gz 2026-02-25T12:02:52.407 INFO:teuthology.orchestra.run.trial117.stderr:/var/log/ceph/ceph-client.admin.27407.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.27407.log.gz 2026-02-25T12:02:52.407 INFO:teuthology.orchestra.run.trial117.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mon.a.log 2026-02-25T12:02:52.407 INFO:teuthology.orchestra.run.trial117.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26196.log 2026-02-25T12:02:52.408 INFO:teuthology.orchestra.run.trial117.stderr:/var/log/ceph/ceph-client.admin.25649.log: 0.0%/var/log/ceph/ceph-mon.a.log: -- replaced with /var/log/ceph/ceph-client.admin.25649.log.gz 2026-02-25T12:02:52.408 INFO:teuthology.orchestra.run.trial117.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.8.log 2026-02-25T12:02:52.408 INFO:teuthology.orchestra.run.trial117.stderr:/var/log/ceph/ceph-client.admin.26196.log: gzip -5 --verbose -- 0.0% /var/log/ceph/ceph-client.admin.25415.log -- replaced with /var/log/ceph/ceph-client.admin.26196.log.gz 2026-02-25T12:02:52.408 INFO:teuthology.orchestra.run.trial117.stderr: 2026-02-25T12:02:52.408 INFO:teuthology.orchestra.run.trial117.stderr:/var/log/ceph/ceph-osd.8.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27041.log 2026-02-25T12:02:52.408 INFO:teuthology.orchestra.run.trial117.stderr:/var/log/ceph/ceph-client.admin.25415.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.25415.log.gz 2026-02-25T12:02:52.409 INFO:teuthology.orchestra.run.trial117.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.24797.log 2026-02-25T12:02:52.409 INFO:teuthology.orchestra.run.trial117.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27249.log 2026-02-25T12:02:52.409 INFO:teuthology.orchestra.run.trial117.stderr:/var/log/ceph/ceph-client.admin.27041.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.27041.log.gz 2026-02-25T12:02:52.409 INFO:teuthology.orchestra.run.trial117.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26121.log 2026-02-25T12:02:52.409 INFO:teuthology.orchestra.run.trial117.stderr:/var/log/ceph/ceph-client.admin.24797.log: /var/log/ceph/ceph-client.admin.27249.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.24797.log.gz 2026-02-25T12:02:52.409 INFO:teuthology.orchestra.run.trial117.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.27249.log.gz 2026-02-25T12:02:52.409 INFO:teuthology.orchestra.run.trial117.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.19403.log 2026-02-25T12:02:52.410 INFO:teuthology.orchestra.run.trial117.stderr:/var/log/ceph/ceph-client.admin.26121.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.26121.log.gz 2026-02-25T12:02:52.410 INFO:teuthology.orchestra.run.trial117.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.19469.log 2026-02-25T12:02:52.410 INFO:teuthology.orchestra.run.trial117.stderr:/var/log/ceph/ceph-client.admin.19403.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27613.log 2026-02-25T12:02:52.410 INFO:teuthology.orchestra.run.trial117.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.19403.log.gz 2026-02-25T12:02:52.410 INFO:teuthology.orchestra.run.trial117.stderr:/var/log/ceph/ceph-client.admin.19469.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.24739.log 2026-02-25T12:02:52.410 INFO:teuthology.orchestra.run.trial117.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.19469.log.gz 2026-02-25T12:02:52.410 INFO:teuthology.orchestra.run.trial117.stderr:/var/log/ceph/ceph-client.admin.27613.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.27613.log.gz 2026-02-25T12:02:52.410 INFO:teuthology.orchestra.run.trial117.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.25549.log 2026-02-25T12:02:52.411 INFO:teuthology.orchestra.run.trial117.stderr:/var/log/ceph/ceph-client.admin.24739.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.25521.log 2026-02-25T12:02:52.411 INFO:teuthology.orchestra.run.trial117.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.24739.log.gz 2026-02-25T12:02:52.411 INFO:teuthology.orchestra.run.trial117.stderr:/var/log/ceph/ceph-client.admin.25549.log: gzip 0.0% -- replaced with /var/log/ceph/ceph-client.admin.25549.log.gz -5 2026-02-25T12:02:52.411 INFO:teuthology.orchestra.run.trial117.stderr: --verbose -- /var/log/ceph/ceph-client.admin.25674.log 2026-02-25T12:02:52.411 INFO:teuthology.orchestra.run.trial117.stderr:/var/log/ceph/ceph-client.admin.25521.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.25521.log.gz 2026-02-25T12:02:52.411 INFO:teuthology.orchestra.run.trial117.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.25285.log 2026-02-25T12:02:52.412 INFO:teuthology.orchestra.run.trial117.stderr:/var/log/ceph/ceph-client.admin.25674.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.19855.log 2026-02-25T12:02:52.412 INFO:teuthology.orchestra.run.trial117.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.25674.log.gz 2026-02-25T12:02:52.412 INFO:teuthology.orchestra.run.trial117.stderr:/var/log/ceph/ceph-client.admin.25285.log: 0.0%gzip -- replaced with /var/log/ceph/ceph-client.admin.25285.log.gz 2026-02-25T12:02:52.412 INFO:teuthology.orchestra.run.trial117.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.26294.log 2026-02-25T12:02:52.412 INFO:teuthology.orchestra.run.trial117.stderr:/var/log/ceph/ceph-client.admin.19855.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.19855.log.gz 2026-02-25T12:02:52.412 INFO:teuthology.orchestra.run.trial117.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.25298.log 2026-02-25T12:02:52.413 INFO:teuthology.orchestra.run.trial117.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.25871.log 2026-02-25T12:02:52.413 INFO:teuthology.orchestra.run.trial117.stderr:/var/log/ceph/ceph-client.admin.26294.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.26294.log.gz 2026-02-25T12:02:52.413 INFO:teuthology.orchestra.run.trial117.stderr:/var/log/ceph/ceph-client.admin.25298.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.25298.log.gz 2026-02-25T12:02:52.413 INFO:teuthology.orchestra.run.trial117.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.7.log 2026-02-25T12:02:52.413 INFO:teuthology.orchestra.run.trial117.stderr:/var/log/ceph/ceph-client.admin.25871.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.25871.log.gz 2026-02-25T12:02:52.413 INFO:teuthology.orchestra.run.trial117.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26497.log 2026-02-25T12:02:52.413 INFO:teuthology.orchestra.run.trial117.stderr:/var/log/ceph/ceph-osd.7.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26171.log 2026-02-25T12:02:52.414 INFO:teuthology.orchestra.run.trial117.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.25573.log 2026-02-25T12:02:52.414 INFO:teuthology.orchestra.run.trial117.stderr:/var/log/ceph/ceph-client.admin.26497.log: /var/log/ceph/ceph-client.admin.26171.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.26497.log.gz 2026-02-25T12:02:52.414 INFO:teuthology.orchestra.run.trial117.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.26171.log.gz 2026-02-25T12:02:52.414 INFO:teuthology.orchestra.run.trial117.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.24616.log 2026-02-25T12:02:52.414 INFO:teuthology.orchestra.run.trial117.stderr:/var/log/ceph/ceph-client.admin.25573.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.19517.log 2026-02-25T12:02:52.415 INFO:teuthology.orchestra.run.trial117.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.25573.log.gz 2026-02-25T12:02:52.415 INFO:teuthology.orchestra.run.trial117.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.25441.log 2026-02-25T12:02:52.415 INFO:teuthology.orchestra.run.trial117.stderr:/var/log/ceph/ceph-client.admin.24616.log: /var/log/ceph/ceph-client.admin.19517.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.24616.log.gz 2026-02-25T12:02:52.415 INFO:teuthology.orchestra.run.trial117.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.19517.log.gz 2026-02-25T12:02:52.415 INFO:teuthology.orchestra.run.trial117.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26781.log 2026-02-25T12:02:52.415 INFO:teuthology.orchestra.run.trial117.stderr:/var/log/ceph/ceph-client.admin.25441.log: 0.0%gzip -- replaced with /var/log/ceph/ceph-client.admin.25441.log.gz 2026-02-25T12:02:52.415 INFO:teuthology.orchestra.run.trial117.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.25294.log 2026-02-25T12:02:52.416 INFO:teuthology.orchestra.run.trial117.stderr:/var/log/ceph/ceph-client.admin.26781.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27197.log 2026-02-25T12:02:52.416 INFO:teuthology.orchestra.run.trial117.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.26781.log.gz 2026-02-25T12:02:52.416 INFO:teuthology.orchestra.run.trial117.stderr:/var/log/ceph/ceph-client.admin.25294.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.25294.log.gz 2026-02-25T12:02:52.416 INFO:teuthology.orchestra.run.trial117.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.25058.log 2026-02-25T12:02:52.416 INFO:teuthology.orchestra.run.trial117.stderr:/var/log/ceph/ceph-client.admin.27197.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.27197.log.gz 2026-02-25T12:02:52.416 INFO:teuthology.orchestra.run.trial117.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.24789.log 2026-02-25T12:02:52.416 INFO:teuthology.orchestra.run.trial117.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27171.log 2026-02-25T12:02:52.417 INFO:teuthology.orchestra.run.trial117.stderr:/var/log/ceph/ceph-client.admin.25058.log: /var/log/ceph/ceph-client.admin.24789.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.25058.log.gz 2026-02-25T12:02:52.417 INFO:teuthology.orchestra.run.trial117.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.24789.log.gz 2026-02-25T12:02:52.417 INFO:teuthology.orchestra.run.trial117.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.19783.log 2026-02-25T12:02:52.417 INFO:teuthology.orchestra.run.trial117.stderr: 91.5% -- replaced with /var/log/ceph/ceph-mgr.x.log.gz 2026-02-25T12:02:52.417 INFO:teuthology.orchestra.run.trial117.stderr:/var/log/ceph/ceph-client.admin.27171.log: gzip 0.0% -- replaced with /var/log/ceph/ceph-client.admin.27171.log.gz -5 2026-02-25T12:02:52.417 INFO:teuthology.orchestra.run.trial117.stderr: --verbose -- /var/log/ceph/ceph-client.admin.26670.log 2026-02-25T12:02:52.417 INFO:teuthology.orchestra.run.trial117.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.25698.log 2026-02-25T12:02:52.418 INFO:teuthology.orchestra.run.trial117.stderr:/var/log/ceph/ceph-client.admin.19783.log: /var/log/ceph/ceph-client.admin.26670.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.19783.log.gz 2026-02-25T12:02:52.418 INFO:teuthology.orchestra.run.trial117.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.26670.log.gz 2026-02-25T12:02:52.418 INFO:teuthology.orchestra.run.trial117.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26620.log 2026-02-25T12:02:52.418 INFO:teuthology.orchestra.run.trial117.stderr:/var/log/ceph/ceph-client.admin.25698.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.25698.log.gz 2026-02-25T12:02:52.418 INFO:teuthology.orchestra.run.trial117.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26755.log 2026-02-25T12:02:52.418 INFO:teuthology.orchestra.run.trial117.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.25747.log 2026-02-25T12:02:52.419 INFO:teuthology.orchestra.run.trial117.stderr:/var/log/ceph/ceph-client.admin.26620.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.26620.log.gz 2026-02-25T12:02:52.419 INFO:teuthology.orchestra.run.trial117.stderr:/var/log/ceph/ceph-client.admin.26755.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.24591.log 0.0% 2026-02-25T12:02:52.419 INFO:teuthology.orchestra.run.trial117.stderr: -- replaced with /var/log/ceph/ceph-client.admin.26755.log.gz 2026-02-25T12:02:52.419 INFO:teuthology.orchestra.run.trial117.stderr:/var/log/ceph/ceph-client.admin.25747.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.25747.log.gz 2026-02-25T12:02:52.419 INFO:teuthology.orchestra.run.trial117.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26368.log 2026-02-25T12:02:52.419 INFO:teuthology.orchestra.run.trial117.stderr:/var/log/ceph/ceph-client.admin.24591.log: gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2026-02-25T12:02:52.419 INFO:teuthology.orchestra.run.trial117.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.24591.log.gz 2026-02-25T12:02:52.420 INFO:teuthology.orchestra.run.trial117.stderr:/var/log/ceph/ceph-client.admin.26368.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26937.log 2026-02-25T12:02:52.420 INFO:teuthology.orchestra.run.trial117.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.26368.log.gz 2026-02-25T12:02:52.420 INFO:teuthology.orchestra.run.trial117.stderr:/var/log/ceph/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.2.log 2026-02-25T12:02:52.420 INFO:teuthology.orchestra.run.trial117.stderr:/var/log/ceph/ceph-client.admin.26937.log: 90.3% -- replaced with /var/log/ceph/ceph.audit.log.gz 2026-02-25T12:02:52.420 INFO:teuthology.orchestra.run.trial117.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26545.log 2026-02-25T12:02:52.420 INFO:teuthology.orchestra.run.trial117.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.26937.log.gz 2026-02-25T12:02:52.420 INFO:teuthology.orchestra.run.trial117.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.24787.log 2026-02-25T12:02:52.421 INFO:teuthology.orchestra.run.trial117.stderr:/var/log/ceph/ceph-osd.2.log: /var/log/ceph/ceph-client.admin.26545.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.26545.log.gz 2026-02-25T12:02:52.421 INFO:teuthology.orchestra.run.trial117.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.24798.log 2026-02-25T12:02:52.421 INFO:teuthology.orchestra.run.trial117.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.log 2026-02-25T12:02:52.421 INFO:teuthology.orchestra.run.trial117.stderr:/var/log/ceph/ceph-client.admin.24787.log: /var/log/ceph/ceph-client.admin.24798.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.24787.log.gz 2026-02-25T12:02:52.421 INFO:teuthology.orchestra.run.trial117.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.24798.log.gz 2026-02-25T12:02:52.421 INFO:teuthology.orchestra.run.trial117.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26645.log 2026-02-25T12:02:52.422 INFO:teuthology.orchestra.run.trial117.stderr:/var/log/ceph/ceph.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26446.log 2026-02-25T12:02:52.422 INFO:teuthology.orchestra.run.trial117.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27537.log 2026-02-25T12:02:52.422 INFO:teuthology.orchestra.run.trial117.stderr:/var/log/ceph/ceph-client.admin.26645.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.26645.log.gz 2026-02-25T12:02:52.422 INFO:teuthology.orchestra.run.trial117.stderr:/var/log/ceph/ceph-client.admin.26446.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.26446.log.gz 2026-02-25T12:02:52.422 INFO:teuthology.orchestra.run.trial117.stderr: 85.8% -- replaced with /var/log/ceph/ceph.log.gz 2026-02-25T12:02:52.422 INFO:teuthology.orchestra.run.trial117.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27589.log 2026-02-25T12:02:52.423 INFO:teuthology.orchestra.run.trial117.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.4.log 2026-02-25T12:02:52.423 INFO:teuthology.orchestra.run.trial117.stderr:/var/log/ceph/ceph-client.admin.27537.log: /var/log/ceph/ceph-client.admin.27589.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.27537.log.gz 2026-02-25T12:02:52.423 INFO:teuthology.orchestra.run.trial117.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.27589.log.gz 2026-02-25T12:02:52.423 INFO:teuthology.orchestra.run.trial117.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.25389.log 2026-02-25T12:02:52.423 INFO:teuthology.orchestra.run.trial117.stderr:/var/log/ceph/ceph-osd.4.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26045.log 2026-02-25T12:02:52.424 INFO:teuthology.orchestra.run.trial117.stderr:/var/log/ceph/ceph-client.admin.25389.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.19759.log 2026-02-25T12:02:52.424 INFO:teuthology.orchestra.run.trial117.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.25389.log.gz 2026-02-25T12:02:52.424 INFO:teuthology.orchestra.run.trial117.stderr:/var/log/ceph/ceph-client.admin.26045.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.26045.log.gz 2026-02-25T12:02:52.424 INFO:teuthology.orchestra.run.trial117.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27223.log 2026-02-25T12:02:52.424 INFO:teuthology.orchestra.run.trial117.stderr:/var/log/ceph/ceph-client.admin.19759.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.19759.log.gz 2026-02-25T12:02:52.424 INFO:teuthology.orchestra.run.trial117.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27119.log 2026-02-25T12:02:52.424 INFO:teuthology.orchestra.run.trial117.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26885.log 2026-02-25T12:02:52.425 INFO:teuthology.orchestra.run.trial117.stderr:/var/log/ceph/ceph-client.admin.27223.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.27223.log.gz 2026-02-25T12:02:52.425 INFO:teuthology.orchestra.run.trial117.stderr:/var/log/ceph/ceph-client.admin.27119.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.24687.log 2026-02-25T12:02:52.425 INFO:teuthology.orchestra.run.trial117.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.27119.log.gz 2026-02-25T12:02:52.425 INFO:teuthology.orchestra.run.trial117.stderr:/var/log/ceph/ceph-client.admin.26885.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26963.log 0.0% 2026-02-25T12:02:52.425 INFO:teuthology.orchestra.run.trial117.stderr: -- replaced with /var/log/ceph/ceph-client.admin.26885.log.gz 2026-02-25T12:02:52.425 INFO:teuthology.orchestra.run.trial117.stderr:/var/log/ceph/ceph-client.admin.24687.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.25043.log 2026-02-25T12:02:52.425 INFO:teuthology.orchestra.run.trial117.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.24687.log.gz 2026-02-25T12:02:52.426 INFO:teuthology.orchestra.run.trial117.stderr:/var/log/ceph/ceph-client.admin.26963.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.26963.log.gz 2026-02-25T12:02:52.426 INFO:teuthology.orchestra.run.trial117.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27301.log 2026-02-25T12:02:52.426 INFO:teuthology.orchestra.run.trial117.stderr:/var/log/ceph/ceph-client.admin.25043.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.25772.log 2026-02-25T12:02:52.426 INFO:teuthology.orchestra.run.trial117.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.25043.log.gz 2026-02-25T12:02:52.426 INFO:teuthology.orchestra.run.trial117.stderr:/var/log/ceph/ceph-client.admin.27301.log: gzip 0.0% -5 -- replaced with /var/log/ceph/ceph-client.admin.27301.log.gz --verbose 2026-02-25T12:02:52.426 INFO:teuthology.orchestra.run.trial117.stderr: -- /var/log/ceph/ceph-client.admin.27511.log 2026-02-25T12:02:52.427 INFO:teuthology.orchestra.run.trial117.stderr:/var/log/ceph/ceph-client.admin.25772.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.25921.log 2026-02-25T12:02:52.427 INFO:teuthology.orchestra.run.trial117.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.25772.log.gz 2026-02-25T12:02:52.427 INFO:teuthology.orchestra.run.trial117.stderr:/var/log/ceph/ceph-client.admin.27511.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.25995.log 2026-02-25T12:02:52.427 INFO:teuthology.orchestra.run.trial117.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.27511.log.gz 2026-02-25T12:02:52.428 INFO:teuthology.orchestra.run.trial117.stderr:/var/log/ceph/ceph-client.admin.25921.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26269.log 2026-02-25T12:02:52.428 INFO:teuthology.orchestra.run.trial117.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.25921.log.gz 2026-02-25T12:02:52.428 INFO:teuthology.orchestra.run.trial117.stderr:/var/log/ceph/ceph-client.admin.25995.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.25995.log.gz 2026-02-25T12:02:52.428 INFO:teuthology.orchestra.run.trial117.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.24713.log 2026-02-25T12:02:52.428 INFO:teuthology.orchestra.run.trial117.stderr:/var/log/ceph/ceph-client.admin.26269.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.24800.log 2026-02-25T12:02:52.428 INFO:teuthology.orchestra.run.trial117.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.26269.log.gz 2026-02-25T12:02:52.429 INFO:teuthology.orchestra.run.trial117.stderr:/var/log/ceph/ceph-client.admin.24713.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26421.log 2026-02-25T12:02:52.429 INFO:teuthology.orchestra.run.trial117.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.24713.log.gz 2026-02-25T12:02:52.429 INFO:teuthology.orchestra.run.trial117.stderr:/var/log/ceph/ceph-client.admin.24800.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.24800.log.gz 2026-02-25T12:02:52.429 INFO:teuthology.orchestra.run.trial117.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26470.log 2026-02-25T12:02:52.429 INFO:teuthology.orchestra.run.trial117.stderr:/var/log/ceph/ceph-client.admin.26421.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26694.log 2026-02-25T12:02:52.429 INFO:teuthology.orchestra.run.trial117.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.26421.log.gz 2026-02-25T12:02:52.430 INFO:teuthology.orchestra.run.trial117.stderr:/var/log/ceph/ceph-client.admin.26470.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.0.log 0.0% 2026-02-25T12:02:52.430 INFO:teuthology.orchestra.run.trial117.stderr: -- replaced with /var/log/ceph/ceph-client.admin.26470.log.gz 2026-02-25T12:02:52.430 INFO:teuthology.orchestra.run.trial117.stderr:/var/log/ceph/ceph-client.admin.26694.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26095.log 2026-02-25T12:02:52.430 INFO:teuthology.orchestra.run.trial117.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.26694.log.gz 2026-02-25T12:02:52.430 INFO:teuthology.orchestra.run.trial117.stderr:/var/log/ceph/ceph-osd.0.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.25970.log 2026-02-25T12:02:52.431 INFO:teuthology.orchestra.run.trial117.stderr:/var/log/ceph/ceph-client.admin.26095.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27459.log 2026-02-25T12:02:52.431 INFO:teuthology.orchestra.run.trial117.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.26095.log.gz 2026-02-25T12:02:52.431 INFO:teuthology.orchestra.run.trial117.stderr: 92.5% -- replaced with /var/log/ceph/ceph-osd.8.log.gz 2026-02-25T12:02:52.431 INFO:teuthology.orchestra.run.trial117.stderr: 92.3% -- replaced with /var/log/ceph/ceph-osd.3.log.gz 2026-02-25T12:02:52.431 INFO:teuthology.orchestra.run.trial117.stderr:/var/log/ceph/ceph-client.admin.25970.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.25970.log.gz 2026-02-25T12:02:52.431 INFO:teuthology.orchestra.run.trial117.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.24801.log 2026-02-25T12:02:52.431 INFO:teuthology.orchestra.run.trial117.stderr:/var/log/ceph/ceph-client.admin.27459.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.27459.log.gz 2026-02-25T12:02:52.431 INFO:teuthology.orchestra.run.trial117.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26070.log 2026-02-25T12:02:52.432 INFO:teuthology.orchestra.run.trial117.stderr:/var/log/ceph/ceph-client.admin.24801.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27145.log 2026-02-25T12:02:52.432 INFO:teuthology.orchestra.run.trial117.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.24801.log.gz 2026-02-25T12:02:52.432 INFO:teuthology.orchestra.run.trial117.stderr:/var/log/ceph/ceph-client.admin.26070.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.26070.log.gz 2026-02-25T12:02:52.432 INFO:teuthology.orchestra.run.trial117.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.25945.log 2026-02-25T12:02:52.432 INFO:teuthology.orchestra.run.trial117.stderr:/var/log/ceph/ceph-client.admin.27145.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.27145.log.gz 2026-02-25T12:02:52.432 INFO:teuthology.orchestra.run.trial117.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.24795.log 2026-02-25T12:02:52.433 INFO:teuthology.orchestra.run.trial117.stderr:/var/log/ceph/ceph-client.admin.25945.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.25945.log.gz 2026-02-25T12:02:52.433 INFO:teuthology.orchestra.run.trial117.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.25063.log 2026-02-25T12:02:52.433 INFO:teuthology.orchestra.run.trial117.stderr:/var/log/ceph/ceph-client.admin.24795.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.24795.log.gz 2026-02-25T12:02:52.433 INFO:teuthology.orchestra.run.trial117.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27275.log 2026-02-25T12:02:52.434 INFO:teuthology.orchestra.run.trial117.stderr:/var/log/ceph/ceph-client.admin.25063.log: 0.0%gzip -- replaced with /var/log/ceph/ceph-client.admin.25063.log.gz 2026-02-25T12:02:52.434 INFO:teuthology.orchestra.run.trial117.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.25059.log 2026-02-25T12:02:52.434 INFO:teuthology.orchestra.run.trial117.stderr:/var/log/ceph/ceph-client.admin.27275.log: gzip -5 --verbose -- 0.0% -- replaced with /var/log/ceph/ceph-client.admin.27275.log.gz /var/log/ceph/ceph-client.admin.26020.log 2026-02-25T12:02:52.434 INFO:teuthology.orchestra.run.trial117.stderr: 2026-02-25T12:02:52.434 INFO:teuthology.orchestra.run.trial117.stderr:/var/log/ceph/ceph-client.admin.25059.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.25059.log.gz 2026-02-25T12:02:52.434 INFO:teuthology.orchestra.run.trial117.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.25822.log 2026-02-25T12:02:52.435 INFO:teuthology.orchestra.run.trial117.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.19562.log 2026-02-25T12:02:52.435 INFO:teuthology.orchestra.run.trial117.stderr:/var/log/ceph/ceph-client.admin.26020.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.26020.log.gz 2026-02-25T12:02:52.435 INFO:teuthology.orchestra.run.trial117.stderr:/var/log/ceph/ceph-client.admin.25822.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.25822.log.gz 2026-02-25T12:02:52.435 INFO:teuthology.orchestra.run.trial117.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.25045.log 2026-02-25T12:02:52.435 INFO:teuthology.orchestra.run.trial117.stderr:/var/log/ceph/ceph-client.admin.19562.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.1.log 2026-02-25T12:02:52.435 INFO:teuthology.orchestra.run.trial117.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.19562.log.gz 2026-02-25T12:02:52.436 INFO:teuthology.orchestra.run.trial117.stderr:/var/log/ceph/ceph-client.admin.25045.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.25045.log.gz 2026-02-25T12:02:52.436 INFO:teuthology.orchestra.run.trial117.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27381.log 2026-02-25T12:02:52.436 INFO:teuthology.orchestra.run.trial117.stderr:/var/log/ceph/ceph-osd.1.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.25040.log 2026-02-25T12:02:52.436 INFO:teuthology.orchestra.run.trial117.stderr:/var/log/ceph/ceph-client.admin.27381.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.27381.log.gz 2026-02-25T12:02:52.436 INFO:teuthology.orchestra.run.trial117.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.20447.log 2026-02-25T12:02:52.437 INFO:teuthology.orchestra.run.trial117.stderr:/var/log/ceph/ceph-client.admin.25040.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.25040.log.gz 2026-02-25T12:02:52.437 INFO:teuthology.orchestra.run.trial117.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.25041.log 2026-02-25T12:02:52.437 INFO:teuthology.orchestra.run.trial117.stderr:/var/log/ceph/ceph-client.admin.20447.log: 92.2% -- replaced with /var/log/ceph/ceph-osd.5.log.gz 2026-02-25T12:02:52.437 INFO:teuthology.orchestra.run.trial117.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.20447.log.gz 2026-02-25T12:02:52.437 INFO:teuthology.orchestra.run.trial117.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.25042.log 2026-02-25T12:02:52.437 INFO:teuthology.orchestra.run.trial117.stderr:/var/log/ceph/ceph-client.admin.25041.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.24785.log 2026-02-25T12:02:52.437 INFO:teuthology.orchestra.run.trial117.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.25041.log.gz 2026-02-25T12:02:52.438 INFO:teuthology.orchestra.run.trial117.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26344.log 2026-02-25T12:02:52.438 INFO:teuthology.orchestra.run.trial117.stderr:/var/log/ceph/ceph-client.admin.25042.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.25042.log.gz 2026-02-25T12:02:52.438 INFO:teuthology.orchestra.run.trial117.stderr:/var/log/ceph/ceph-client.admin.24785.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.24785.log.gz 2026-02-25T12:02:52.438 INFO:teuthology.orchestra.run.trial117.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.25722.log 2026-02-25T12:02:52.438 INFO:teuthology.orchestra.run.trial117.stderr:gzip/var/log/ceph/ceph-client.admin.26344.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.26319.log 2026-02-25T12:02:52.438 INFO:teuthology.orchestra.run.trial117.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.26344.log.gz 2026-02-25T12:02:52.438 INFO:teuthology.orchestra.run.trial117.stderr:/var/log/ceph/ceph-client.admin.25722.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.25722.log.gz 2026-02-25T12:02:52.439 INFO:teuthology.orchestra.run.trial117.stderr:/var/log/ceph/ceph-client.admin.26319.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.26319.log.gz 2026-02-25T12:02:52.441 INFO:teuthology.orchestra.run.trial117.stderr: 92.3% -- replaced with /var/log/ceph/ceph-osd.7.log.gz 2026-02-25T12:02:52.442 INFO:teuthology.orchestra.run.trial117.stderr: 92.2% -- replaced with /var/log/ceph/ceph-osd.9.log.gz 2026-02-25T12:02:52.443 INFO:teuthology.orchestra.run.trial117.stderr: 92.2% -- replaced with /var/log/ceph/ceph-osd.6.log.gz 2026-02-25T12:02:52.445 INFO:teuthology.orchestra.run.trial117.stderr: 92.5% -- replaced with /var/log/ceph/ceph-osd.2.log.gz 2026-02-25T12:02:52.462 INFO:teuthology.orchestra.run.trial117.stderr: 92.3% -- replaced with /var/log/ceph/ceph-osd.4.log.gz 2026-02-25T12:02:52.463 INFO:teuthology.orchestra.run.trial117.stderr: 92.1% -- replaced with /var/log/ceph/ceph-osd.0.log.gz 2026-02-25T12:02:52.470 INFO:teuthology.orchestra.run.trial117.stderr: 92.3% -- replaced with /var/log/ceph/ceph-osd.1.log.gz 2026-02-25T12:02:52.493 INFO:teuthology.orchestra.run.trial117.stderr: 88.8% -- replaced with /var/log/ceph/ceph-mon.a.log.gz 2026-02-25T12:02:52.494 INFO:teuthology.orchestra.run.trial117.stderr: 2026-02-25T12:02:52.494 INFO:teuthology.orchestra.run.trial117.stderr:real 0m0.107s 2026-02-25T12:02:52.494 INFO:teuthology.orchestra.run.trial117.stderr:user 0m0.477s 2026-02-25T12:02:52.495 INFO:teuthology.orchestra.run.trial117.stderr:sys 0m0.064s 2026-02-25T12:02:52.495 INFO:tasks.ceph:Archiving logs... 2026-02-25T12:02:52.495 DEBUG:teuthology.misc:Transferring archived files from trial117:/var/log/ceph to /home/teuthworker/mnt/teuthology/nmordech-2026-02-25_11:36:23-rados-wip-rocky10-branch-of-the-day-2026-02-24-1771941190-distro-default-trial/69872/remote/trial117/log 2026-02-25T12:02:52.498 DEBUG:teuthology.orchestra.run.trial117:> sudo tar c -f - -C /var/log/ceph -- . 2026-02-25T12:02:52.780 DEBUG:teuthology.run_tasks:Unwinding manager install 2026-02-25T12:02:52.791 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2026-02-25T12:02:52.792 DEBUG:teuthology.orchestra.run.trial117:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2026-02-25T12:02:52.880 DEBUG:teuthology.run_tasks:Unwinding manager clock 2026-02-25T12:02:52.892 INFO:teuthology.task.clock:Checking final clock skew... 2026-02-25T12:02:52.892 DEBUG:teuthology.orchestra.run.trial117:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2026-02-25T12:02:52.917 INFO:teuthology.orchestra.run.trial117.stdout: remote refid st t when poll reach delay offset jitter 2026-02-25T12:02:52.917 INFO:teuthology.orchestra.run.trial117.stdout:============================================================================== 2026-02-25T12:02:52.917 INFO:teuthology.orchestra.run.trial117.stdout: 69-10-208-170.r 128.9.176.30 2 u 36 64 7 65.298 -1.190 0.241 2026-02-25T12:02:52.917 INFO:teuthology.orchestra.run.trial117.stdout: B1-66ER.matrix. 129.6.15.28 2 u 37 64 7 8.908 +0.325 0.124 2026-02-25T12:02:52.917 INFO:teuthology.orchestra.run.trial117.stdout: 38.45.64.130 129.134.25.123 2 u 42 64 7 21.742 -1.071 1.478 2026-02-25T12:02:52.917 INFO:teuthology.orchestra.run.trial117.stdout: 170-187-142-180 82.43.52.28 2 u 37 64 7 23.675 +9.530 0.321 2026-02-25T12:02:52.918 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2026-02-25T12:02:52.930 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2026-02-25T12:02:52.931 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2026-02-25T12:02:52.944 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2026-02-25T12:02:52.959 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2026-02-25T12:02:52.982 INFO:teuthology.task.internal:Duration was 320.691773 seconds 2026-02-25T12:02:52.982 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2026-02-25T12:02:52.997 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2026-02-25T12:02:52.997 DEBUG:teuthology.orchestra.run.trial117:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2026-02-25T12:02:53.023 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2026-02-25T12:02:53.023 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@trial117.front.sepia.ceph.com 2026-02-25T12:02:53.023 DEBUG:teuthology.orchestra.run.trial117:> grep -E --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | grep -E -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | grep -E -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2026-02-25T12:02:53.071 INFO:teuthology.task.internal.syslog:Gathering journactl... 2026-02-25T12:02:53.071 DEBUG:teuthology.orchestra.run.trial117:> sudo journalctl > /home/ubuntu/cephtest/archive/syslog/journalctl.log 2026-02-25T12:02:53.339 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2026-02-25T12:02:53.339 DEBUG:teuthology.orchestra.run.trial117:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --max-args=1 --max-procs=0 --verbose --no-run-if-empty -- gzip -5 --verbose -- 2026-02-25T12:02:53.348 INFO:teuthology.orchestra.run.trial117.stderr:gzip -5 --verbose -- /home/ubuntu/cephtest/archive/syslog/misc.log 2026-02-25T12:02:53.348 INFO:teuthology.orchestra.run.trial117.stderr:gzip -5 --verbose -- /home/ubuntu/cephtest/archive/syslog/journalctl.log 2026-02-25T12:02:53.349 INFO:teuthology.orchestra.run.trial117.stderr:/home/ubuntu/cephtest/archive/syslog/misc.log: 0.0% -- replaced with /home/ubuntu/cephtest/archive/syslog/misc.log.gz 2026-02-25T12:02:53.349 INFO:teuthology.orchestra.run.trial117.stderr:gzip -5 --verbose -- /home/ubuntu/cephtest/archive/syslog/kern.log 2026-02-25T12:02:53.349 INFO:teuthology.orchestra.run.trial117.stderr:/home/ubuntu/cephtest/archive/syslog/journalctl.log: /home/ubuntu/cephtest/archive/syslog/kern.log: 0.0% -- replaced with /home/ubuntu/cephtest/archive/syslog/kern.log.gz 2026-02-25T12:02:53.420 INFO:teuthology.orchestra.run.trial117.stderr: 81.5% -- replaced with /home/ubuntu/cephtest/archive/syslog/journalctl.log.gz 2026-02-25T12:02:53.422 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2026-02-25T12:02:53.438 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2026-02-25T12:02:53.439 DEBUG:teuthology.orchestra.run.trial117:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2026-02-25T12:02:53.473 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2026-02-25T12:02:53.493 DEBUG:teuthology.orchestra.run.trial117:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2026-02-25T12:02:53.524 INFO:teuthology.orchestra.run.trial117.stdout:kernel.core_pattern = core 2026-02-25T12:02:53.532 DEBUG:teuthology.orchestra.run.trial117:> test -e /home/ubuntu/cephtest/archive/coredump 2026-02-25T12:02:53.576 DEBUG:teuthology.orchestra.run:got remote process result: 1 2026-02-25T12:02:53.576 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2026-02-25T12:02:53.591 INFO:teuthology.task.internal:Transferring archived files... 2026-02-25T12:02:53.592 DEBUG:teuthology.misc:Transferring archived files from trial117:/home/ubuntu/cephtest/archive to /home/teuthworker/mnt/teuthology/nmordech-2026-02-25_11:36:23-rados-wip-rocky10-branch-of-the-day-2026-02-24-1771941190-distro-default-trial/69872/remote/trial117 2026-02-25T12:02:53.592 DEBUG:teuthology.orchestra.run.trial117:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2026-02-25T12:02:53.658 INFO:teuthology.task.internal:Removing archive directory... 2026-02-25T12:02:53.658 DEBUG:teuthology.orchestra.run.trial117:> rm -rf -- /home/ubuntu/cephtest/archive 2026-02-25T12:02:53.685 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2026-02-25T12:02:53.703 INFO:teuthology.task.internal:Not uploading archives. 2026-02-25T12:02:53.703 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2026-02-25T12:02:53.719 INFO:teuthology.task.internal:Tidying up after the test... 2026-02-25T12:02:53.719 DEBUG:teuthology.orchestra.run.trial117:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2026-02-25T12:02:53.733 INFO:teuthology.orchestra.run.trial117.stdout: 1311264 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Feb 25 12:02 /home/ubuntu/cephtest 2026-02-25T12:02:53.735 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2026-02-25T12:02:53.752 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2026-02-25T12:02:53.790 INFO:teuthology.run:Summary data: description: rados/singleton-nomsgr/{all/health-warnings mon_election/connectivity rados supported-random-distro$/{ubuntu_latest}} duration: 320.6917726993561 flavor: default owner: scheduled_nmordech@soko04.front.sepia.ceph.com success: true 2026-02-25T12:02:53.791 DEBUG:teuthology.report:Pushing job info to https://paddles-paddles.apps.pok.os.sepia.ceph.com 2026-02-25T12:02:53.910 INFO:teuthology.run:pass