2026-02-26T02:54:01.317 INFO:root:teuthology version: 1.2.4.dev11+g6e3d0419c 2026-02-26T02:54:01.332 DEBUG:teuthology.report:Pushing job info to https://paddles-paddles.apps.pok.os.sepia.ceph.com 2026-02-26T02:54:01.412 INFO:teuthology.run:Config: archive_path: /home/teuthworker/mnt/teuthology/yuriw-2026-02-26_01:51:16-rados-wip-yuri9-testing-2026-02-25-1600-squid-distro-default-trial/71117 branch: wip-yuri9-testing-2026-02-25-1600-squid description: rados/singleton/{all/divergent_priors2 mon_election/classic msgr-failures/few msgr/async objectstore/bluestore-comp-snappy rados supported-random-distro$/{centos_latest}} email: yweinste@redhat.com first_in_suite: false flavor: default job_id: '71117' kernel: branch: distro kdb: true sha1: distro last_in_suite: false machine_type: trial name: yuriw-2026-02-26_01:51:16-rados-wip-yuri9-testing-2026-02-25-1600-squid-distro-default-trial no_nested_subset: false openstack: - volumes: count: 3 size: 10 os_type: centos os_version: 9.stream overrides: admin_socket: branch: wip-yuri9-testing-2026-02-25-1600-squid ceph: conf: global: mon client directed command retry: 5 mon election default strategy: 1 ms inject socket failures: 5000 ms type: async mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon scrub interval: 300 osd: bluestore block size: 96636764160 bluestore compression algorithm: snappy bluestore compression mode: aggressive bluestore fsck on mount: true bluestore zero block detection: true debug bluefs: 20 debug bluestore: 20 debug ms: 1 debug osd: 20 debug rocksdb: 10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd debug verify cached snaps: true osd debug verify missing on start: true osd failsafe full ratio: 0.95 osd mclock override recovery settings: true osd mclock profile: high_recovery_ops osd objectstore: bluestore osd op queue: debug_random osd op queue cut off: debug_random flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - overall HEALTH_ - \(OSDMAP_FLAGS\) - \(OSD_ - \(PG_ - \(OBJECT_ - \(POOL_APP_NOT_ENABLED\) - \(OSD_SLOW_PING_TIME - \(MON_DOWN\) sha1: 8d724284d6d767af9ab8bf4bc6bd4e45f8502cb5 ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} install: ceph: flavor: default sha1: 8d724284d6d767af9ab8bf4bc6bd4e45f8502cb5 selinux: allowlist: - scontext=system_u:system_r:getty_t:s0 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: wip-yuri9-testing-2026-02-25-1600-squid sha1: 8d724284d6d767af9ab8bf4bc6bd4e45f8502cb5 owner: scheduled_yuriw@soko04.front.sepia.ceph.com priority: 69 repo: https://github.com/ceph/ceph-ci.git roles: - - mon.a - mgr.x - osd.0 - osd.1 - osd.2 - client.0 seed: 1525 sha1: 8d724284d6d767af9ab8bf4bc6bd4e45f8502cb5 sleep_before_teardown: 0 subset: 111/120000 suite: rados suite_branch: wip-yuri9-testing-2026-02-25-1600-squid suite_path: /home/teuthworker/src/github.com_ceph_ceph-c_8d724284d6d767af9ab8bf4bc6bd4e45f8502cb5/qa suite_relpath: qa suite_repo: https://github.com/ceph/ceph-ci.git suite_sha1: 8d724284d6d767af9ab8bf4bc6bd4e45f8502cb5 targets: trial197.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBP7CMkwbyz7vDJuG2HaLTRkZgbc4OUrKmvLXOk/PM1MD4mX2CZNKLwxpycgmJydkkoUmssKmhi6XrZLW06Lwwxs= tasks: - install: null - ceph: pre-mgr-commands: - sudo ceph config set mgr mgr_pool false --force - divergent_priors2: null teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: 6e3d0419ccca0e75a03e08feb708946e198236f4 timestamp: 2026-02-26_01:51:16 tube: trial user: yuriw verbose: true worker_log: /home/teuthworker/mnt/teuthology/worker_logs/dispatcher.trial.904699 2026-02-26T02:54:01.413 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/github.com_ceph_ceph-c_8d724284d6d767af9ab8bf4bc6bd4e45f8502cb5/qa; will attempt to use it 2026-02-26T02:54:01.413 INFO:teuthology.run:Found tasks at /home/teuthworker/src/github.com_ceph_ceph-c_8d724284d6d767af9ab8bf4bc6bd4e45f8502cb5/qa/tasks 2026-02-26T02:54:01.413 INFO:teuthology.run_tasks:Running task internal.check_packages... 2026-02-26T02:54:01.415 INFO:teuthology.task.internal:Checking packages... 2026-02-26T02:54:01.415 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash '8d724284d6d767af9ab8bf4bc6bd4e45f8502cb5' 2026-02-26T02:54:01.415 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2026-02-26T02:54:01.415 INFO:teuthology.packaging:ref: None 2026-02-26T02:54:01.415 INFO:teuthology.packaging:tag: None 2026-02-26T02:54:01.415 INFO:teuthology.packaging:branch: wip-yuri9-testing-2026-02-25-1600-squid 2026-02-26T02:54:01.415 INFO:teuthology.packaging:sha1: 8d724284d6d767af9ab8bf4bc6bd4e45f8502cb5 2026-02-26T02:54:01.415 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=wip-yuri9-testing-2026-02-25-1600-squid 2026-02-26T02:54:01.538 INFO:teuthology.task.internal:Found packages for ceph version 19.2.3-687.g8d724284 2026-02-26T02:54:01.539 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2026-02-26T02:54:01.554 INFO:teuthology.task.internal:no buildpackages task found 2026-02-26T02:54:01.555 INFO:teuthology.run_tasks:Running task internal.save_config... 2026-02-26T02:54:01.568 INFO:teuthology.task.internal:Saving configuration 2026-02-26T02:54:01.580 INFO:teuthology.run_tasks:Running task internal.check_lock... 2026-02-26T02:54:01.593 INFO:teuthology.task.internal.check_lock:Checking locks... 2026-02-26T02:54:01.671 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'trial197.front.sepia.ceph.com', 'description': '/home/teuthworker/mnt/teuthology/yuriw-2026-02-26_01:51:16-rados-wip-yuri9-testing-2026-02-25-1600-squid-distro-default-trial/71117', 'up': True, 'machine_type': 'trial', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9.stream', 'arch': 'x86_64', 'locked': True, 'locked_since': '2026-02-26 02:50:33.025576', 'locked_by': 'scheduled_yuriw@soko04.front.sepia.ceph.com', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBP7CMkwbyz7vDJuG2HaLTRkZgbc4OUrKmvLXOk/PM1MD4mX2CZNKLwxpycgmJydkkoUmssKmhi6XrZLW06Lwwxs='} 2026-02-26T02:54:01.672 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2026-02-26T02:54:01.683 INFO:teuthology.task.internal:roles: ubuntu@trial197.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'osd.0', 'osd.1', 'osd.2', 'client.0'] 2026-02-26T02:54:01.683 INFO:teuthology.run_tasks:Running task console_log... 2026-02-26T02:54:01.779 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7396278ee660>, signals=[15]) 2026-02-26T02:54:01.779 INFO:teuthology.run_tasks:Running task internal.connect... 2026-02-26T02:54:01.794 INFO:teuthology.task.internal:Opening connections... 2026-02-26T02:54:01.794 DEBUG:teuthology.task.internal:connecting to ubuntu@trial197.front.sepia.ceph.com 2026-02-26T02:54:01.795 DEBUG:teuthology.orchestra.connection:{'hostname': 'trial197.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2026-02-26T02:54:01.856 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2026-02-26T02:54:01.870 DEBUG:teuthology.orchestra.run.trial197:> uname -m 2026-02-26T02:54:01.927 INFO:teuthology.orchestra.run.trial197.stdout:x86_64 2026-02-26T02:54:01.927 DEBUG:teuthology.orchestra.run.trial197:> cat /etc/os-release 2026-02-26T02:54:01.983 INFO:teuthology.orchestra.run.trial197.stdout:NAME="CentOS Stream" 2026-02-26T02:54:01.983 INFO:teuthology.orchestra.run.trial197.stdout:VERSION="9" 2026-02-26T02:54:01.983 INFO:teuthology.orchestra.run.trial197.stdout:ID="centos" 2026-02-26T02:54:01.983 INFO:teuthology.orchestra.run.trial197.stdout:ID_LIKE="rhel fedora" 2026-02-26T02:54:01.983 INFO:teuthology.orchestra.run.trial197.stdout:VERSION_ID="9" 2026-02-26T02:54:01.983 INFO:teuthology.orchestra.run.trial197.stdout:PLATFORM_ID="platform:el9" 2026-02-26T02:54:01.983 INFO:teuthology.orchestra.run.trial197.stdout:PRETTY_NAME="CentOS Stream 9" 2026-02-26T02:54:01.983 INFO:teuthology.orchestra.run.trial197.stdout:ANSI_COLOR="0;31" 2026-02-26T02:54:01.983 INFO:teuthology.orchestra.run.trial197.stdout:LOGO="fedora-logo-icon" 2026-02-26T02:54:01.983 INFO:teuthology.orchestra.run.trial197.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2026-02-26T02:54:01.983 INFO:teuthology.orchestra.run.trial197.stdout:HOME_URL="https://centos.org/" 2026-02-26T02:54:01.983 INFO:teuthology.orchestra.run.trial197.stdout:BUG_REPORT_URL="https://issues.redhat.com/" 2026-02-26T02:54:01.983 INFO:teuthology.orchestra.run.trial197.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2026-02-26T02:54:01.984 INFO:teuthology.orchestra.run.trial197.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2026-02-26T02:54:01.984 INFO:teuthology.lock.ops:Updating trial197.front.sepia.ceph.com on lock server 2026-02-26T02:54:02.057 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2026-02-26T02:54:02.072 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2026-02-26T02:54:02.083 INFO:teuthology.task.internal:Checking for old test directory... 2026-02-26T02:54:02.084 DEBUG:teuthology.orchestra.run.trial197:> test '!' -e /home/ubuntu/cephtest 2026-02-26T02:54:02.099 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2026-02-26T02:54:02.114 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2026-02-26T02:54:02.114 DEBUG:teuthology.orchestra.run.trial197:> test -z $(ls -A /var/lib/ceph) 2026-02-26T02:54:02.156 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2026-02-26T02:54:02.234 INFO:teuthology.run_tasks:Running task kernel... 2026-02-26T02:54:02.254 INFO:teuthology.task.kernel:normalize config orig: {'branch': 'distro', 'kdb': True, 'sha1': 'distro'} 2026-02-26T02:54:02.254 INFO:teuthology.task.kernel:config {'mon.a': {'branch': 'distro', 'kdb': True, 'sha1': 'distro'}}, timeout 300 2026-02-26T02:54:02.254 DEBUG:teuthology.orchestra.run.trial197:> test -f /run/.containerenv -o -f /.dockerenv 2026-02-26T02:54:02.269 DEBUG:teuthology.orchestra.run:got remote process result: 1 2026-02-26T02:54:02.269 DEBUG:teuthology.orchestra.run.trial197:> uname -r 2026-02-26T02:54:02.324 INFO:teuthology.orchestra.run.trial197.stdout:5.14.0-681.el9.x86_64 2026-02-26T02:54:02.324 INFO:teuthology.task.kernel:Running kernel on trial197: 5.14.0-681.el9.x86_64 2026-02-26T02:54:02.324 DEBUG:teuthology.orchestra.run.trial197:> sudo yum install -y kernel 2026-02-26T02:54:03.376 INFO:teuthology.orchestra.run.trial197.stdout:CentOS Stream 9 - BaseOS 13 MB/s | 8.9 MB 00:00 2026-02-26T02:54:31.077 INFO:teuthology.orchestra.run.trial197.stdout:CentOS Stream 9 - AppStream 1.0 MB/s | 27 MB 00:27 2026-02-26T02:54:34.501 INFO:teuthology.orchestra.run.trial197.stdout:CentOS Stream 9 - CRB 7.7 MB/s | 8.0 MB 00:01 2026-02-26T02:54:35.721 INFO:teuthology.orchestra.run.trial197.stdout:CentOS Stream 9 - Extras packages 42 kB/s | 20 kB 00:00 2026-02-26T02:54:36.715 INFO:teuthology.orchestra.run.trial197.stdout:Extra Packages for Enterprise Linux 22 MB/s | 20 MB 00:00 2026-02-26T02:54:39.913 INFO:teuthology.orchestra.run.trial197.stdout:lab-extras 1.8 MB/s | 50 kB 00:00 2026-02-26T02:54:40.945 INFO:teuthology.orchestra.run.trial197.stdout:Package kernel-5.14.0-645.el9.x86_64 is already installed. 2026-02-26T02:54:40.945 INFO:teuthology.orchestra.run.trial197.stdout:Package kernel-5.14.0-681.el9.x86_64 is already installed. 2026-02-26T02:54:40.965 INFO:teuthology.orchestra.run.trial197.stdout:Dependencies resolved. 2026-02-26T02:54:40.966 INFO:teuthology.orchestra.run.trial197.stdout:================================================================================ 2026-02-26T02:54:40.966 INFO:teuthology.orchestra.run.trial197.stdout: Package Architecture Version Repository Size 2026-02-26T02:54:40.966 INFO:teuthology.orchestra.run.trial197.stdout:================================================================================ 2026-02-26T02:54:40.966 INFO:teuthology.orchestra.run.trial197.stdout:Installing: 2026-02-26T02:54:40.966 INFO:teuthology.orchestra.run.trial197.stdout: kernel x86_64 5.14.0-686.el9 baseos 901 k 2026-02-26T02:54:40.966 INFO:teuthology.orchestra.run.trial197.stdout:Installing dependencies: 2026-02-26T02:54:40.966 INFO:teuthology.orchestra.run.trial197.stdout: kernel-core x86_64 5.14.0-686.el9 baseos 17 M 2026-02-26T02:54:40.966 INFO:teuthology.orchestra.run.trial197.stdout: kernel-modules x86_64 5.14.0-686.el9 baseos 40 M 2026-02-26T02:54:40.966 INFO:teuthology.orchestra.run.trial197.stdout: kernel-modules-core x86_64 5.14.0-686.el9 baseos 31 M 2026-02-26T02:54:40.966 INFO:teuthology.orchestra.run.trial197.stdout: 2026-02-26T02:54:40.966 INFO:teuthology.orchestra.run.trial197.stdout:Transaction Summary 2026-02-26T02:54:40.966 INFO:teuthology.orchestra.run.trial197.stdout:================================================================================ 2026-02-26T02:54:40.966 INFO:teuthology.orchestra.run.trial197.stdout:Install 4 Packages 2026-02-26T02:54:40.966 INFO:teuthology.orchestra.run.trial197.stdout: 2026-02-26T02:54:40.966 INFO:teuthology.orchestra.run.trial197.stdout:Total download size: 89 M 2026-02-26T02:54:40.966 INFO:teuthology.orchestra.run.trial197.stdout:Installed size: 135 M 2026-02-26T02:54:40.967 INFO:teuthology.orchestra.run.trial197.stdout:Downloading Packages: 2026-02-26T02:54:42.023 INFO:teuthology.orchestra.run.trial197.stdout:(1/4): kernel-5.14.0-686.el9.x86_64.rpm 933 kB/s | 901 kB 00:00 2026-02-26T02:54:42.480 INFO:teuthology.orchestra.run.trial197.stdout:(2/4): kernel-core-5.14.0-686.el9.x86_64.rpm 12 MB/s | 17 MB 00:01 2026-02-26T02:54:43.024 INFO:teuthology.orchestra.run.trial197.stdout:(3/4): kernel-modules-5.14.0-686.el9.x86_64.rpm 20 MB/s | 40 MB 00:01 2026-02-26T02:54:43.877 INFO:teuthology.orchestra.run.trial197.stdout:(4/4): kernel-modules-core-5.14.0-686.el9.x86_6 17 MB/s | 31 MB 00:01 2026-02-26T02:54:43.878 INFO:teuthology.orchestra.run.trial197.stdout:-------------------------------------------------------------------------------- 2026-02-26T02:54:43.878 INFO:teuthology.orchestra.run.trial197.stdout:Total 31 MB/s | 89 MB 00:02 2026-02-26T02:54:44.071 INFO:teuthology.orchestra.run.trial197.stdout:Running transaction check 2026-02-26T02:54:44.283 INFO:teuthology.orchestra.run.trial197.stdout:Transaction check succeeded. 2026-02-26T02:54:44.283 INFO:teuthology.orchestra.run.trial197.stdout:Running transaction test 2026-02-26T02:54:44.667 INFO:teuthology.orchestra.run.trial197.stdout:Transaction test succeeded. 2026-02-26T02:54:44.667 INFO:teuthology.orchestra.run.trial197.stdout:Running transaction 2026-02-26T02:54:45.780 INFO:teuthology.orchestra.run.trial197.stdout: Preparing : 1/1 2026-02-26T02:54:46.074 INFO:teuthology.orchestra.run.trial197.stdout: Installing : kernel-modules-core-5.14.0-686.el9.x86_64 1/4 2026-02-26T02:54:46.096 INFO:teuthology.orchestra.run.trial197.stdout: Installing : kernel-core-5.14.0-686.el9.x86_64 2/4 2026-02-26T02:54:48.666 INFO:teuthology.orchestra.run.trial197.stdout: Running scriptlet: kernel-core-5.14.0-686.el9.x86_64 2/4 2026-02-26T02:54:48.700 INFO:teuthology.orchestra.run.trial197.stdout: Installing : kernel-modules-5.14.0-686.el9.x86_64 3/4 2026-02-26T02:54:51.808 INFO:teuthology.orchestra.run.trial197.stdout: Running scriptlet: kernel-modules-5.14.0-686.el9.x86_64 3/4 2026-02-26T02:54:51.814 INFO:teuthology.orchestra.run.trial197.stdout: Installing : kernel-5.14.0-686.el9.x86_64 4/4 2026-02-26T02:54:54.867 INFO:teuthology.orchestra.run.trial197.stdout: Running scriptlet: kernel-modules-core-5.14.0-686.el9.x86_64 4/4 2026-02-26T02:55:12.309 INFO:teuthology.orchestra.run.trial197.stdout: Running scriptlet: kernel-core-5.14.0-686.el9.x86_64 4/4 2026-02-26T02:55:12.330 INFO:teuthology.orchestra.run.trial197.stdout: Running scriptlet: kernel-modules-5.14.0-686.el9.x86_64 4/4 2026-02-26T02:55:12.513 INFO:teuthology.orchestra.run.trial197.stdout: Running scriptlet: kernel-5.14.0-686.el9.x86_64 4/4 2026-02-26T02:55:12.513 INFO:teuthology.orchestra.run.trial197.stdout: Verifying : kernel-5.14.0-686.el9.x86_64 1/4 2026-02-26T02:55:12.513 INFO:teuthology.orchestra.run.trial197.stdout: Verifying : kernel-core-5.14.0-686.el9.x86_64 2/4 2026-02-26T02:55:12.514 INFO:teuthology.orchestra.run.trial197.stdout: Verifying : kernel-modules-5.14.0-686.el9.x86_64 3/4 2026-02-26T02:55:12.570 INFO:teuthology.orchestra.run.trial197.stdout: Verifying : kernel-modules-core-5.14.0-686.el9.x86_64 4/4 2026-02-26T02:55:12.570 INFO:teuthology.orchestra.run.trial197.stdout: 2026-02-26T02:55:12.570 INFO:teuthology.orchestra.run.trial197.stdout:Installed: 2026-02-26T02:55:12.570 INFO:teuthology.orchestra.run.trial197.stdout: kernel-5.14.0-686.el9.x86_64 2026-02-26T02:55:12.570 INFO:teuthology.orchestra.run.trial197.stdout: kernel-core-5.14.0-686.el9.x86_64 2026-02-26T02:55:12.570 INFO:teuthology.orchestra.run.trial197.stdout: kernel-modules-5.14.0-686.el9.x86_64 2026-02-26T02:55:12.570 INFO:teuthology.orchestra.run.trial197.stdout: kernel-modules-core-5.14.0-686.el9.x86_64 2026-02-26T02:55:12.570 INFO:teuthology.orchestra.run.trial197.stdout: 2026-02-26T02:55:12.571 INFO:teuthology.orchestra.run.trial197.stdout:Complete! 2026-02-26T02:55:12.630 DEBUG:teuthology.orchestra.run.trial197:> rpm -q kernel | sort -rV | head -n 1 2026-02-26T02:55:12.657 INFO:teuthology.orchestra.run.trial197.stdout:kernel-5.14.0-686.el9.x86_64 2026-02-26T02:55:12.659 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-686.el9.x86_64 2026-02-26T02:55:12.659 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-681.el9.x86_64 Expected: 5.14.0-686.el9.x86_64 2026-02-26T02:55:12.659 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2026-02-26T02:55:12.659 DEBUG:teuthology.task.kernel:src is distro, skipping download 2026-02-26T02:55:12.659 INFO:teuthology.task.kernel:Installing distro kernel on mon.a... 2026-02-26T02:55:12.659 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@trial197.front.sepia.ceph.com, path=None, version=distro) 2026-02-26T02:55:12.659 DEBUG:teuthology.orchestra.run.trial197:> rpm -q kernel | sort -rV | head -n 1 2026-02-26T02:55:12.724 INFO:teuthology.orchestra.run.trial197.stdout:kernel-5.14.0-686.el9.x86_64 2026-02-26T02:55:12.725 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-686.el9.x86_64 2026-02-26T02:55:12.725 DEBUG:teuthology.orchestra.run.trial197:> sudo rpm -qi grub2-tools 2026-02-26T02:55:12.792 INFO:teuthology.orchestra.run.trial197.stdout:Name : grub2-tools 2026-02-26T02:55:12.792 INFO:teuthology.orchestra.run.trial197.stdout:Epoch : 1 2026-02-26T02:55:12.793 INFO:teuthology.orchestra.run.trial197.stdout:Version : 2.06 2026-02-26T02:55:12.793 INFO:teuthology.orchestra.run.trial197.stdout:Release : 120.el9 2026-02-26T02:55:12.793 INFO:teuthology.orchestra.run.trial197.stdout:Architecture: x86_64 2026-02-26T02:55:12.793 INFO:teuthology.orchestra.run.trial197.stdout:Install Date: Wed 18 Feb 2026 05:18:42 PM UTC 2026-02-26T02:55:12.793 INFO:teuthology.orchestra.run.trial197.stdout:Group : Unspecified 2026-02-26T02:55:12.793 INFO:teuthology.orchestra.run.trial197.stdout:Size : 9108611 2026-02-26T02:55:12.793 INFO:teuthology.orchestra.run.trial197.stdout:License : GPLv3+ 2026-02-26T02:55:12.793 INFO:teuthology.orchestra.run.trial197.stdout:Signature : RSA/SHA256, Thu 04 Dec 2025 09:42:59 AM UTC, Key ID 05b555b38483c65d 2026-02-26T02:55:12.793 INFO:teuthology.orchestra.run.trial197.stdout:Source RPM : grub2-2.06-120.el9.src.rpm 2026-02-26T02:55:12.793 INFO:teuthology.orchestra.run.trial197.stdout:Build Date : Wed 03 Dec 2025 12:10:38 PM UTC 2026-02-26T02:55:12.793 INFO:teuthology.orchestra.run.trial197.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2026-02-26T02:55:12.793 INFO:teuthology.orchestra.run.trial197.stdout:Packager : builder@centos.org 2026-02-26T02:55:12.794 INFO:teuthology.orchestra.run.trial197.stdout:Vendor : CentOS 2026-02-26T02:55:12.794 INFO:teuthology.orchestra.run.trial197.stdout:URL : http://www.gnu.org/software/grub/ 2026-02-26T02:55:12.794 INFO:teuthology.orchestra.run.trial197.stdout:Summary : Support tools for GRUB. 2026-02-26T02:55:12.794 INFO:teuthology.orchestra.run.trial197.stdout:Description : 2026-02-26T02:55:12.794 INFO:teuthology.orchestra.run.trial197.stdout: 2026-02-26T02:55:12.794 INFO:teuthology.orchestra.run.trial197.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2026-02-26T02:55:12.794 INFO:teuthology.orchestra.run.trial197.stdout:customizable bootloader with modular architecture. It supports a rich 2026-02-26T02:55:12.794 INFO:teuthology.orchestra.run.trial197.stdout:variety of kernel formats, file systems, computer architectures and 2026-02-26T02:55:12.794 INFO:teuthology.orchestra.run.trial197.stdout:hardware devices. 2026-02-26T02:55:12.794 INFO:teuthology.orchestra.run.trial197.stdout: 2026-02-26T02:55:12.794 INFO:teuthology.orchestra.run.trial197.stdout:This subpackage provides tools for support of all platforms. 2026-02-26T02:55:12.796 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2026-02-26T02:55:12.796 INFO:teuthology.task.kernel:Updating grub on trial197 to boot 5.14.0-686.el9.x86_64 2026-02-26T02:55:12.796 DEBUG:teuthology.orchestra.run.trial197:> sudo find /boot/loader/entries -maxdepth 1 -name "*.conf" -type f -print -quit 2026-02-26T02:55:12.860 INFO:teuthology.orchestra.run.trial197.stdout:/boot/loader/entries/b0373be789df49cdafedf423703d2566-5.14.0-681.el9.x86_64.conf 2026-02-26T02:55:12.862 DEBUG:teuthology.orchestra.run.trial197:> sudo command -v grubby && echo yes || echo no 2026-02-26T02:55:12.924 INFO:teuthology.orchestra.run.trial197.stdout:/sbin/grubby 2026-02-26T02:55:12.926 INFO:teuthology.orchestra.run.trial197.stdout:yes 2026-02-26T02:55:12.926 DEBUG:teuthology.orchestra.run.trial197:> sudo find /boot/loader/entries -maxdepth 1 -name "*.conf" -type f | grep -F "5.14.0-686.el9.x86_64" | head -n 1 2026-02-26T02:55:12.990 INFO:teuthology.orchestra.run.trial197.stdout:/boot/loader/entries/b0373be789df49cdafedf423703d2566-5.14.0-686.el9.x86_64.conf 2026-02-26T02:55:12.991 DEBUG:teuthology.orchestra.run.trial197:> sudo grub2-set-default b0373be789df49cdafedf423703d2566-5.14.0-686.el9.x86_64 2026-02-26T02:55:13.064 DEBUG:teuthology.orchestra.run.trial197:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2026-02-26T02:55:13.273 INFO:teuthology.orchestra.run.trial197.stderr:Generating grub configuration file ... 2026-02-26T02:55:13.340 INFO:teuthology.orchestra.run.trial197.stderr:Found linux image: /boot/vmlinuz-5.14.0-686.el9.x86_64 2026-02-26T02:55:13.346 INFO:teuthology.orchestra.run.trial197.stderr:Found initrd image: /boot/initramfs-5.14.0-686.el9.x86_64.img 2026-02-26T02:55:13.598 INFO:teuthology.orchestra.run.trial197.stderr:Found linux image: /boot/vmlinuz-5.14.0-681.el9.x86_64 2026-02-26T02:55:13.605 INFO:teuthology.orchestra.run.trial197.stderr:Found initrd image: /boot/initramfs-5.14.0-681.el9.x86_64.img 2026-02-26T02:55:13.640 INFO:teuthology.orchestra.run.trial197.stderr:Found linux image: /boot/vmlinuz-5.14.0-645.el9.x86_64 2026-02-26T02:55:13.646 INFO:teuthology.orchestra.run.trial197.stderr:Found initrd image: /boot/initramfs-5.14.0-645.el9.x86_64.img 2026-02-26T02:55:13.677 INFO:teuthology.orchestra.run.trial197.stderr:Found linux image: /boot/vmlinuz-0-rescue-b0373be789df49cdafedf423703d2566 2026-02-26T02:55:13.684 INFO:teuthology.orchestra.run.trial197.stderr:Found initrd image: /boot/initramfs-0-rescue-b0373be789df49cdafedf423703d2566.img 2026-02-26T02:55:13.707 INFO:teuthology.orchestra.run.trial197.stderr:Found linux image: /boot/vmlinuz-0-rescue-2734456005d743dca7f3dc77c35a2d0d 2026-02-26T02:55:13.714 INFO:teuthology.orchestra.run.trial197.stderr:Found initrd image: /boot/initramfs-0-rescue-2734456005d743dca7f3dc77c35a2d0d.img 2026-02-26T02:55:13.800 INFO:teuthology.orchestra.run.trial197.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2026-02-26T02:55:13.808 INFO:teuthology.orchestra.run.trial197.stderr:done 2026-02-26T02:55:13.810 DEBUG:teuthology.orchestra.run.trial197:> test -d /sys/firmware/efi 2026-02-26T02:55:13.827 DEBUG:teuthology.orchestra.run.trial197:> sudo find /boot/efi/EFI -mindepth 1 -maxdepth 1 -type d || true 2026-02-26T02:55:13.894 INFO:teuthology.orchestra.run.trial197.stdout:/boot/efi/EFI/BOOT 2026-02-26T02:55:13.894 INFO:teuthology.orchestra.run.trial197.stdout:/boot/efi/EFI/centos 2026-02-26T02:55:13.896 DEBUG:teuthology.orchestra.run.trial197:> sudo test -f /boot/efi/EFI/BOOT/grub.cfg && echo yes || echo no 2026-02-26T02:55:13.962 INFO:teuthology.orchestra.run.trial197.stdout:no 2026-02-26T02:55:13.962 DEBUG:teuthology.orchestra.run.trial197:> sudo test -f /boot/efi/EFI/centos/grub.cfg && echo yes || echo no 2026-02-26T02:55:14.026 INFO:teuthology.orchestra.run.trial197.stdout:yes 2026-02-26T02:55:14.027 DEBUG:teuthology.orchestra.run.trial197:> sudo cp -f /boot/grub2/grub.cfg /boot/efi/EFI/centos/grub.cfg 2026-02-26T02:55:14.091 INFO:teuthology.orchestra.remote:Triggering safe hard reboot via SysRq on trial197 2026-02-26T02:55:14.091 DEBUG:teuthology.orchestra.run.trial197:> sudo bash -c ' 2026-02-26T02:55:14.091 DEBUG:teuthology.orchestra.run.trial197:> echo pci > /sys/kernel/reboot/type && echo 1 > /sys/kernel/reboot/force && sync && echo s > /proc/sysrq-trigger && echo u > /proc/sysrq-trigger && sleep 2 && echo b > /proc/sysrq-trigger 2026-02-26T02:55:14.091 DEBUG:teuthology.orchestra.run.trial197:> ' 2026-02-26T02:55:44.133 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version... 2026-02-26T02:55:44.133 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@trial197.front.sepia.ceph.com' 2026-02-26T02:55:44.133 DEBUG:teuthology.orchestra.connection:{'hostname': 'trial197.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2026-02-26T02:56:44.193 DEBUG:teuthology.orchestra.remote:timed out 2026-02-26T02:56:53.202 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@trial197.front.sepia.ceph.com' 2026-02-26T02:56:53.203 DEBUG:teuthology.orchestra.connection:{'hostname': 'trial197.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2026-02-26T02:56:53.299 DEBUG:teuthology.orchestra.run.trial197:> true 2026-02-26T02:56:53.452 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@trial197.front.sepia.ceph.com' 2026-02-26T02:56:53.452 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "5.14.0-686.el9.x86_64"... 2026-02-26T02:56:53.452 DEBUG:teuthology.orchestra.run.trial197:> uname -r 2026-02-26T02:56:53.508 INFO:teuthology.orchestra.run.trial197.stdout:5.14.0-686.el9.x86_64 2026-02-26T02:56:53.508 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-686.el9.x86_64 vs 5.14.0-686.el9.x86_64 2026-02-26T02:56:53.509 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2026-02-26T02:56:53.509 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2026-02-26T02:56:54.510 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2026-02-26T02:56:54.510 DEBUG:teuthology.orchestra.run.trial197:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2026-02-26T02:56:54.539 INFO:teuthology.orchestra.run.trial197.stdout:ttyS1 2026-02-26T02:56:54.559 DEBUG:teuthology.parallel:result is None 2026-02-26T02:56:54.559 INFO:teuthology.run_tasks:Running task internal.base... 2026-02-26T02:56:54.606 INFO:teuthology.task.internal:Creating test directory... 2026-02-26T02:56:54.607 DEBUG:teuthology.orchestra.run.trial197:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2026-02-26T02:56:54.623 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2026-02-26T02:56:54.634 INFO:teuthology.run_tasks:Running task internal.archive... 2026-02-26T02:56:54.648 INFO:teuthology.task.internal:Creating archive directory... 2026-02-26T02:56:54.649 DEBUG:teuthology.orchestra.run.trial197:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2026-02-26T02:56:54.683 INFO:teuthology.run_tasks:Running task internal.coredump... 2026-02-26T02:56:54.694 INFO:teuthology.task.internal:Enabling coredump saving... 2026-02-26T02:56:54.694 DEBUG:teuthology.orchestra.run.trial197:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2026-02-26T02:56:54.741 INFO:teuthology.orchestra.run.trial197.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-26T02:56:54.749 INFO:teuthology.orchestra.run.trial197.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-26T02:56:54.750 INFO:teuthology.run_tasks:Running task internal.sudo... 2026-02-26T02:56:54.762 INFO:teuthology.task.internal:Configuring sudo... 2026-02-26T02:56:54.762 DEBUG:teuthology.orchestra.run.trial197:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2026-02-26T02:56:54.813 INFO:teuthology.run_tasks:Running task internal.syslog... 2026-02-26T02:56:54.830 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2026-02-26T02:56:54.830 DEBUG:teuthology.orchestra.run.trial197:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2026-02-26T02:56:54.870 DEBUG:teuthology.orchestra.run.trial197:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2026-02-26T02:56:55.003 DEBUG:teuthology.orchestra.run.trial197:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2026-02-26T02:56:55.028 DEBUG:teuthology.orchestra.run.trial197:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2026-02-26T02:56:55.147 DEBUG:teuthology.orchestra.run.trial197:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2026-02-26T02:56:55.169 DEBUG:teuthology.orchestra.run.trial197:> set -ex 2026-02-26T02:56:55.169 DEBUG:teuthology.orchestra.run.trial197:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2026-02-26T02:56:55.232 DEBUG:teuthology.orchestra.run.trial197:> sudo service rsyslog restart 2026-02-26T02:56:55.299 INFO:teuthology.orchestra.run.trial197.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2026-02-26T02:56:55.692 INFO:teuthology.run_tasks:Running task internal.timer... 2026-02-26T02:56:55.704 INFO:teuthology.task.internal:Starting timer... 2026-02-26T02:56:55.704 INFO:teuthology.run_tasks:Running task pcp... 2026-02-26T02:56:55.726 INFO:teuthology.run_tasks:Running task selinux... 2026-02-26T02:56:55.745 DEBUG:teuthology.task:Applying overrides for task selinux: {'allowlist': ['scontext=system_u:system_r:getty_t:s0']} 2026-02-26T02:56:55.745 DEBUG:teuthology.orchestra.run.trial197:> sudo service auditd rotate 2026-02-26T02:56:55.773 INFO:teuthology.orchestra.run.trial197.stdout:Rotating logs: 2026-02-26T02:56:55.775 DEBUG:teuthology.task.selinux:Getting current SELinux state 2026-02-26T02:56:55.775 DEBUG:teuthology.orchestra.run.trial197:> /usr/sbin/getenforce 2026-02-26T02:56:55.831 INFO:teuthology.orchestra.run.trial197.stdout:Permissive 2026-02-26T02:56:55.831 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@trial197.front.sepia.ceph.com': 'permissive'} 2026-02-26T02:56:55.831 DEBUG:teuthology.orchestra.run.trial197:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:getty_t:s0 2026-02-26T02:56:55.894 DEBUG:teuthology.orchestra.run:got remote process result: 1 2026-02-26T02:56:55.894 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2026-02-26T02:56:55.895 DEBUG:teuthology.orchestra.run.trial197:> sudo /usr/sbin/setenforce permissive 2026-02-26T02:56:55.957 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2026-02-26T02:56:55.973 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2026-02-26T02:56:55.982 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2026-02-26T02:56:55.983 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit trial197.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2026-02-26T02:59:40.388 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@trial197.front.sepia.ceph.com')] 2026-02-26T02:59:40.389 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@trial197.front.sepia.ceph.com' 2026-02-26T02:59:40.390 DEBUG:teuthology.orchestra.connection:{'hostname': 'trial197.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2026-02-26T02:59:40.506 DEBUG:teuthology.orchestra.run.trial197:> true 2026-02-26T02:59:40.581 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@trial197.front.sepia.ceph.com' 2026-02-26T02:59:40.582 INFO:teuthology.run_tasks:Running task clock... 2026-02-26T02:59:40.595 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2026-02-26T02:59:40.595 INFO:teuthology.orchestra.run:Running command with timeout 360 2026-02-26T02:59:40.596 DEBUG:teuthology.orchestra.run.trial197:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2026-02-26T02:59:40.647 INFO:teuthology.orchestra.run.trial197.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2026-02-26T02:59:40.659 INFO:teuthology.orchestra.run.trial197.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2026-02-26T02:59:40.680 INFO:teuthology.orchestra.run.trial197.stderr:sudo: ntpd: command not found 2026-02-26T02:59:40.690 INFO:teuthology.orchestra.run.trial197.stdout:506 Cannot talk to daemon 2026-02-26T02:59:40.702 INFO:teuthology.orchestra.run.trial197.stderr:Failed to start ntp.service: Unit ntp.service not found. 2026-02-26T02:59:40.713 INFO:teuthology.orchestra.run.trial197.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2026-02-26T02:59:40.759 INFO:teuthology.orchestra.run.trial197.stderr:bash: line 1: ntpq: command not found 2026-02-26T02:59:40.781 INFO:teuthology.orchestra.run.trial197.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2026-02-26T02:59:40.781 INFO:teuthology.orchestra.run.trial197.stdout:=============================================================================== 2026-02-26T02:59:40.781 INFO:teuthology.orchestra.run.trial197.stdout:^? 108.61.73.243 0 6 0 - +0ns[ +0ns] +/- 0ns 2026-02-26T02:59:40.781 INFO:teuthology.orchestra.run.trial197.stdout:^? ntp.shastacoe.net 0 6 0 - +0ns[ +0ns] +/- 0ns 2026-02-26T02:59:40.781 INFO:teuthology.orchestra.run.trial197.stdout:^? time.cloudflare.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2026-02-26T02:59:40.781 INFO:teuthology.orchestra.run.trial197.stdout:^? 149.28.200.179.vultruser> 0 6 0 - +0ns[ +0ns] +/- 0ns 2026-02-26T02:59:40.782 INFO:teuthology.run_tasks:Running task install... 2026-02-26T02:59:40.794 DEBUG:teuthology.task.install:project ceph 2026-02-26T02:59:40.794 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': '8d724284d6d767af9ab8bf4bc6bd4e45f8502cb5'}} 2026-02-26T02:59:40.794 DEBUG:teuthology.task.install:config {'flavor': 'default', 'sha1': '8d724284d6d767af9ab8bf4bc6bd4e45f8502cb5'} 2026-02-26T02:59:40.794 INFO:teuthology.task.install:Using flavor: default 2026-02-26T02:59:40.796 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2026-02-26T02:59:40.796 INFO:teuthology.task.install:extra packages: [] 2026-02-26T02:59:40.797 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '8d724284d6d767af9ab8bf4bc6bd4e45f8502cb5', 'tag': None, 'wait_for_package': False} 2026-02-26T02:59:40.797 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=8d724284d6d767af9ab8bf4bc6bd4e45f8502cb5 2026-02-26T02:59:40.919 INFO:teuthology.task.install.rpm:Pulling from https://2.chacra.ceph.com/r/ceph/wip-yuri9-testing-2026-02-25-1600-squid/8d724284d6d767af9ab8bf4bc6bd4e45f8502cb5/centos/9/flavors/default/ 2026-02-26T02:59:40.919 INFO:teuthology.task.install.rpm:Package version is 19.2.3-687.g8d724284 2026-02-26T02:59:41.037 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://2.chacra.ceph.com/r/ceph/wip-yuri9-testing-2026-02-25-1600-squid/8d724284d6d767af9ab8bf4bc6bd4e45f8502cb5/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-yuri9-testing-2026-02-25-1600-squid/8d724284d6d767af9ab8bf4bc6bd4e45f8502cb5/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-yuri9-testing-2026-02-25-1600-squid/8d724284d6d767af9ab8bf4bc6bd4e45f8502cb5/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2026-02-26T02:59:41.037 DEBUG:teuthology.orchestra.run.trial197:> set -ex 2026-02-26T02:59:41.037 DEBUG:teuthology.orchestra.run.trial197:> sudo dd of=/etc/yum.repos.d/ceph.repo 2026-02-26T02:59:41.064 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2026-02-26T02:59:41.064 DEBUG:teuthology.orchestra.run.trial197:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' /etc/yum.repos.d/ceph.repo ; fi 2026-02-26T02:59:41.128 DEBUG:teuthology.orchestra.run.trial197:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2026-02-26T02:59:41.201 DEBUG:teuthology.orchestra.run.trial197:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2026-02-26T02:59:41.265 INFO:teuthology.orchestra.run.trial197.stdout:check_obsoletes = 1 2026-02-26T02:59:41.267 DEBUG:teuthology.orchestra.run.trial197:> sudo yum clean all 2026-02-26T02:59:41.414 INFO:teuthology.orchestra.run.trial197.stdout:41 files removed 2026-02-26T02:59:41.427 DEBUG:teuthology.orchestra.run.trial197:> sudo yum -y install ceph-radosgw 2026-02-26T02:59:41.706 INFO:teuthology.orchestra.run.trial197.stdout:ceph packages for x86_64 658 kB/s | 86 kB 00:00 2026-02-26T02:59:41.843 INFO:teuthology.orchestra.run.trial197.stdout:ceph noarch packages 112 kB/s | 12 kB 00:00 2026-02-26T02:59:41.972 INFO:teuthology.orchestra.run.trial197.stdout:ceph source packages 18 kB/s | 1.9 kB 00:00 2026-02-26T02:59:42.654 INFO:teuthology.orchestra.run.trial197.stdout:CentOS Stream 9 - BaseOS 13 MB/s | 8.9 MB 00:00 2026-02-26T02:59:45.710 INFO:teuthology.orchestra.run.trial197.stdout:CentOS Stream 9 - AppStream 11 MB/s | 27 MB 00:02 2026-02-26T02:59:48.977 INFO:teuthology.orchestra.run.trial197.stdout:CentOS Stream 9 - CRB 9.0 MB/s | 8.0 MB 00:00 2026-02-26T02:59:50.177 INFO:teuthology.orchestra.run.trial197.stdout:CentOS Stream 9 - Extras packages 44 kB/s | 20 kB 00:00 2026-02-26T02:59:50.713 INFO:teuthology.orchestra.run.trial197.stdout:Extra Packages for Enterprise Linux 43 MB/s | 20 MB 00:00 2026-02-26T02:59:53.872 INFO:teuthology.orchestra.run.trial197.stdout:lab-extras 3.0 MB/s | 50 kB 00:00 2026-02-26T02:59:55.115 INFO:teuthology.orchestra.run.trial197.stdout:Dependencies resolved. 2026-02-26T02:59:55.116 INFO:teuthology.orchestra.run.trial197.stdout:================================================================================ 2026-02-26T02:59:55.116 INFO:teuthology.orchestra.run.trial197.stdout: Package Arch Version Repository Size 2026-02-26T02:59:55.116 INFO:teuthology.orchestra.run.trial197.stdout:================================================================================ 2026-02-26T02:59:55.117 INFO:teuthology.orchestra.run.trial197.stdout:Installing: 2026-02-26T02:59:55.117 INFO:teuthology.orchestra.run.trial197.stdout: ceph-radosgw x86_64 2:19.2.3-687.g8d724284.el9 ceph 11 M 2026-02-26T02:59:55.117 INFO:teuthology.orchestra.run.trial197.stdout:Upgrading: 2026-02-26T02:59:55.117 INFO:teuthology.orchestra.run.trial197.stdout: librados2 x86_64 2:19.2.3-687.g8d724284.el9 ceph 3.4 M 2026-02-26T02:59:55.117 INFO:teuthology.orchestra.run.trial197.stdout: librbd1 x86_64 2:19.2.3-687.g8d724284.el9 ceph 3.2 M 2026-02-26T02:59:55.117 INFO:teuthology.orchestra.run.trial197.stdout:Installing dependencies: 2026-02-26T02:59:55.117 INFO:teuthology.orchestra.run.trial197.stdout: boost-program-options x86_64 1.75.0-13.el9 appstream 104 k 2026-02-26T02:59:55.117 INFO:teuthology.orchestra.run.trial197.stdout: ceph-base x86_64 2:19.2.3-687.g8d724284.el9 ceph 5.5 M 2026-02-26T02:59:55.117 INFO:teuthology.orchestra.run.trial197.stdout: ceph-common x86_64 2:19.2.3-687.g8d724284.el9 ceph 22 M 2026-02-26T02:59:55.117 INFO:teuthology.orchestra.run.trial197.stdout: ceph-selinux x86_64 2:19.2.3-687.g8d724284.el9 ceph 25 k 2026-02-26T02:59:55.117 INFO:teuthology.orchestra.run.trial197.stdout: gperftools-libs x86_64 2.9.1-3.el9 epel 308 k 2026-02-26T02:59:55.118 INFO:teuthology.orchestra.run.trial197.stdout: libarrow x86_64 9.0.0-15.el9 epel 4.4 M 2026-02-26T02:59:55.118 INFO:teuthology.orchestra.run.trial197.stdout: libarrow-doc noarch 9.0.0-15.el9 epel 25 k 2026-02-26T02:59:55.118 INFO:teuthology.orchestra.run.trial197.stdout: libcephfs2 x86_64 2:19.2.3-687.g8d724284.el9 ceph 999 k 2026-02-26T02:59:55.118 INFO:teuthology.orchestra.run.trial197.stdout: libnbd x86_64 1.20.3-4.el9 appstream 164 k 2026-02-26T02:59:55.118 INFO:teuthology.orchestra.run.trial197.stdout: liboath x86_64 2.6.12-1.el9 epel 49 k 2026-02-26T02:59:55.118 INFO:teuthology.orchestra.run.trial197.stdout: libpmemobj x86_64 1.12.1-1.el9 appstream 160 k 2026-02-26T02:59:55.118 INFO:teuthology.orchestra.run.trial197.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2026-02-26T02:59:55.118 INFO:teuthology.orchestra.run.trial197.stdout: libradosstriper1 x86_64 2:19.2.3-687.g8d724284.el9 ceph 504 k 2026-02-26T02:59:55.118 INFO:teuthology.orchestra.run.trial197.stdout: librdkafka x86_64 1.6.1-102.el9 appstream 662 k 2026-02-26T02:59:55.118 INFO:teuthology.orchestra.run.trial197.stdout: librgw2 x86_64 2:19.2.3-687.g8d724284.el9 ceph 5.4 M 2026-02-26T02:59:55.118 INFO:teuthology.orchestra.run.trial197.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2026-02-26T02:59:55.119 INFO:teuthology.orchestra.run.trial197.stdout: lttng-ust x86_64 2.12.0-6.el9 appstream 292 k 2026-02-26T02:59:55.119 INFO:teuthology.orchestra.run.trial197.stdout: mailcap noarch 2.1.49-5.el9 baseos 33 k 2026-02-26T02:59:55.119 INFO:teuthology.orchestra.run.trial197.stdout: parquet-libs x86_64 9.0.0-15.el9 epel 838 k 2026-02-26T02:59:55.119 INFO:teuthology.orchestra.run.trial197.stdout: pciutils x86_64 3.7.0-7.el9 baseos 93 k 2026-02-26T02:59:55.119 INFO:teuthology.orchestra.run.trial197.stdout: python3-ceph-argparse x86_64 2:19.2.3-687.g8d724284.el9 ceph 45 k 2026-02-26T02:59:55.119 INFO:teuthology.orchestra.run.trial197.stdout: python3-ceph-common x86_64 2:19.2.3-687.g8d724284.el9 ceph 142 k 2026-02-26T02:59:55.119 INFO:teuthology.orchestra.run.trial197.stdout: python3-cephfs x86_64 2:19.2.3-687.g8d724284.el9 ceph 165 k 2026-02-26T02:59:55.119 INFO:teuthology.orchestra.run.trial197.stdout: python3-rados x86_64 2:19.2.3-687.g8d724284.el9 ceph 321 k 2026-02-26T02:59:55.119 INFO:teuthology.orchestra.run.trial197.stdout: python3-rbd x86_64 2:19.2.3-687.g8d724284.el9 ceph 303 k 2026-02-26T02:59:55.119 INFO:teuthology.orchestra.run.trial197.stdout: python3-rgw x86_64 2:19.2.3-687.g8d724284.el9 ceph 100 k 2026-02-26T02:59:55.119 INFO:teuthology.orchestra.run.trial197.stdout: qatlib x86_64 25.08.0-2.el9 appstream 240 k 2026-02-26T02:59:55.119 INFO:teuthology.orchestra.run.trial197.stdout: qatzip-libs x86_64 1.3.1-1.el9 appstream 66 k 2026-02-26T02:59:55.120 INFO:teuthology.orchestra.run.trial197.stdout: re2 x86_64 1:20211101-20.el9 epel 191 k 2026-02-26T02:59:55.120 INFO:teuthology.orchestra.run.trial197.stdout: thrift x86_64 0.15.0-4.el9 epel 1.6 M 2026-02-26T02:59:55.120 INFO:teuthology.orchestra.run.trial197.stdout:Installing weak dependencies: 2026-02-26T02:59:55.120 INFO:teuthology.orchestra.run.trial197.stdout: nvme-cli x86_64 2.16-1.el9 baseos 1.2 M 2026-02-26T02:59:55.120 INFO:teuthology.orchestra.run.trial197.stdout: qatlib-service x86_64 25.08.0-2.el9 appstream 37 k 2026-02-26T02:59:55.120 INFO:teuthology.orchestra.run.trial197.stdout: 2026-02-26T02:59:55.120 INFO:teuthology.orchestra.run.trial197.stdout:Transaction Summary 2026-02-26T02:59:55.120 INFO:teuthology.orchestra.run.trial197.stdout:================================================================================ 2026-02-26T02:59:55.120 INFO:teuthology.orchestra.run.trial197.stdout:Install 33 Packages 2026-02-26T02:59:55.120 INFO:teuthology.orchestra.run.trial197.stdout:Upgrade 2 Packages 2026-02-26T02:59:55.120 INFO:teuthology.orchestra.run.trial197.stdout: 2026-02-26T02:59:55.121 INFO:teuthology.orchestra.run.trial197.stdout:Total download size: 63 M 2026-02-26T02:59:55.121 INFO:teuthology.orchestra.run.trial197.stdout:Downloading Packages: 2026-02-26T02:59:56.314 INFO:teuthology.orchestra.run.trial197.stdout:(1/35): ceph-base-19.2.3-687.g8d724284.el9.x86_ 7.8 MB/s | 5.5 MB 00:00 2026-02-26T02:59:56.352 INFO:teuthology.orchestra.run.trial197.stdout:(2/35): ceph-selinux-19.2.3-687.g8d724284.el9.x 652 kB/s | 25 kB 00:00 2026-02-26T02:59:56.409 INFO:teuthology.orchestra.run.trial197.stdout:(3/35): libcephfs2-19.2.3-687.g8d724284.el9.x86 17 MB/s | 999 kB 00:00 2026-02-26T02:59:56.457 INFO:teuthology.orchestra.run.trial197.stdout:(4/35): libradosstriper1-19.2.3-687.g8d724284.e 10 MB/s | 504 kB 00:00 2026-02-26T02:59:56.621 INFO:teuthology.orchestra.run.trial197.stdout:(5/35): ceph-radosgw-19.2.3-687.g8d724284.el9.x 11 MB/s | 11 MB 00:01 2026-02-26T02:59:56.668 INFO:teuthology.orchestra.run.trial197.stdout:(6/35): librgw2-19.2.3-687.g8d724284.el9.x86_64 26 MB/s | 5.4 MB 00:00 2026-02-26T02:59:56.668 INFO:teuthology.orchestra.run.trial197.stdout:(7/35): python3-ceph-argparse-19.2.3-687.g8d724 948 kB/s | 45 kB 00:00 2026-02-26T02:59:56.703 INFO:teuthology.orchestra.run.trial197.stdout:(8/35): python3-ceph-common-19.2.3-687.g8d72428 3.9 MB/s | 142 kB 00:00 2026-02-26T02:59:56.704 INFO:teuthology.orchestra.run.trial197.stdout:(9/35): python3-cephfs-19.2.3-687.g8d724284.el9 4.5 MB/s | 165 kB 00:00 2026-02-26T02:59:56.754 INFO:teuthology.orchestra.run.trial197.stdout:(10/35): python3-rados-19.2.3-687.g8d724284.el9 6.2 MB/s | 321 kB 00:00 2026-02-26T02:59:56.757 INFO:teuthology.orchestra.run.trial197.stdout:(11/35): python3-rbd-19.2.3-687.g8d724284.el9.x 5.6 MB/s | 303 kB 00:00 2026-02-26T02:59:56.805 INFO:teuthology.orchestra.run.trial197.stdout:(12/35): python3-rgw-19.2.3-687.g8d724284.el9.x 1.9 MB/s | 100 kB 00:00 2026-02-26T02:59:57.069 INFO:teuthology.orchestra.run.trial197.stdout:(13/35): ceph-common-19.2.3-687.g8d724284.el9.x 15 MB/s | 22 MB 00:01 2026-02-26T02:59:57.182 INFO:teuthology.orchestra.run.trial197.stdout:(14/35): mailcap-2.1.49-5.el9.noarch.rpm 78 kB/s | 33 kB 00:00 2026-02-26T02:59:57.343 INFO:teuthology.orchestra.run.trial197.stdout:(15/35): boost-program-options-1.75.0-13.el9.x8 646 kB/s | 104 kB 00:00 2026-02-26T02:59:57.504 INFO:teuthology.orchestra.run.trial197.stdout:(16/35): libnbd-1.20.3-4.el9.x86_64.rpm 1.0 MB/s | 164 kB 00:00 2026-02-26T02:59:57.550 INFO:teuthology.orchestra.run.trial197.stdout:(17/35): pciutils-3.7.0-7.el9.x86_64.rpm 193 kB/s | 93 kB 00:00 2026-02-26T02:59:57.588 INFO:teuthology.orchestra.run.trial197.stdout:(18/35): libpmemobj-1.12.1-1.el9.x86_64.rpm 1.9 MB/s | 160 kB 00:00 2026-02-26T02:59:57.616 INFO:teuthology.orchestra.run.trial197.stdout:(19/35): nvme-cli-2.16-1.el9.x86_64.rpm 1.4 MB/s | 1.2 MB 00:00 2026-02-26T02:59:57.634 INFO:teuthology.orchestra.run.trial197.stdout:(20/35): librabbitmq-0.11.0-7.el9.x86_64.rpm 540 kB/s | 45 kB 00:00 2026-02-26T02:59:57.699 INFO:teuthology.orchestra.run.trial197.stdout:(21/35): lttng-ust-2.12.0-6.el9.x86_64.rpm 3.4 MB/s | 292 kB 00:00 2026-02-26T02:59:57.750 INFO:teuthology.orchestra.run.trial197.stdout:(22/35): librdkafka-1.6.1-102.el9.x86_64.rpm 4.0 MB/s | 662 kB 00:00 2026-02-26T02:59:57.781 INFO:teuthology.orchestra.run.trial197.stdout:(23/35): qatlib-service-25.08.0-2.el9.x86_64.rp 452 kB/s | 37 kB 00:00 2026-02-26T02:59:57.798 INFO:teuthology.orchestra.run.trial197.stdout:(24/35): qatlib-25.08.0-2.el9.x86_64.rpm 1.4 MB/s | 240 kB 00:00 2026-02-26T02:59:57.833 INFO:teuthology.orchestra.run.trial197.stdout:(25/35): qatzip-libs-1.3.1-1.el9.x86_64.rpm 802 kB/s | 66 kB 00:00 2026-02-26T02:59:57.857 INFO:teuthology.orchestra.run.trial197.stdout:(26/35): gperftools-libs-2.9.1-3.el9.x86_64.rpm 4.0 MB/s | 308 kB 00:00 2026-02-26T02:59:57.862 INFO:teuthology.orchestra.run.trial197.stdout:(27/35): libarrow-doc-9.0.0-15.el9.noarch.rpm 834 kB/s | 25 kB 00:00 2026-02-26T02:59:57.867 INFO:teuthology.orchestra.run.trial197.stdout:(28/35): liboath-2.6.12-1.el9.x86_64.rpm 4.6 MB/s | 49 kB 00:00 2026-02-26T02:59:57.886 INFO:teuthology.orchestra.run.trial197.stdout:(29/35): libunwind-1.6.2-1.el9.x86_64.rpm 2.8 MB/s | 67 kB 00:00 2026-02-26T02:59:57.892 INFO:teuthology.orchestra.run.trial197.stdout:(30/35): libarrow-9.0.0-15.el9.x86_64.rpm 47 MB/s | 4.4 MB 00:00 2026-02-26T02:59:57.896 INFO:teuthology.orchestra.run.trial197.stdout:(31/35): parquet-libs-9.0.0-15.el9.x86_64.rpm 29 MB/s | 838 kB 00:00 2026-02-26T02:59:57.907 INFO:teuthology.orchestra.run.trial197.stdout:(32/35): thrift-0.15.0-4.el9.x86_64.rpm 102 MB/s | 1.6 MB 00:00 2026-02-26T02:59:57.911 INFO:teuthology.orchestra.run.trial197.stdout:(33/35): re2-20211101-20.el9.x86_64.rpm 7.5 MB/s | 191 kB 00:00 2026-02-26T02:59:58.007 INFO:teuthology.orchestra.run.trial197.stdout:(34/35): librados2-19.2.3-687.g8d724284.el9.x86 31 MB/s | 3.4 MB 00:00 2026-02-26T02:59:58.035 INFO:teuthology.orchestra.run.trial197.stdout:(35/35): librbd1-19.2.3-687.g8d724284.el9.x86_6 25 MB/s | 3.2 MB 00:00 2026-02-26T02:59:58.038 INFO:teuthology.orchestra.run.trial197.stdout:-------------------------------------------------------------------------------- 2026-02-26T02:59:58.038 INFO:teuthology.orchestra.run.trial197.stdout:Total 22 MB/s | 63 MB 00:02 2026-02-26T02:59:58.140 INFO:teuthology.orchestra.run.trial197.stdout:Running transaction check 2026-02-26T02:59:58.158 INFO:teuthology.orchestra.run.trial197.stdout:Transaction check succeeded. 2026-02-26T02:59:58.158 INFO:teuthology.orchestra.run.trial197.stdout:Running transaction test 2026-02-26T02:59:58.409 INFO:teuthology.orchestra.run.trial197.stdout:Transaction test succeeded. 2026-02-26T02:59:58.409 INFO:teuthology.orchestra.run.trial197.stdout:Running transaction 2026-02-26T02:59:58.708 INFO:teuthology.orchestra.run.trial197.stdout: Preparing : 1/1 2026-02-26T02:59:58.717 INFO:teuthology.orchestra.run.trial197.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/37 2026-02-26T02:59:58.719 INFO:teuthology.orchestra.run.trial197.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 2/37 2026-02-26T02:59:58.731 INFO:teuthology.orchestra.run.trial197.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 3/37 2026-02-26T02:59:58.889 INFO:teuthology.orchestra.run.trial197.stdout: Installing : thrift-0.15.0-4.el9.x86_64 4/37 2026-02-26T02:59:58.893 INFO:teuthology.orchestra.run.trial197.stdout: Upgrading : librados2-2:19.2.3-687.g8d724284.el9.x86_64 5/37 2026-02-26T02:59:58.922 INFO:teuthology.orchestra.run.trial197.stdout: Running scriptlet: librados2-2:19.2.3-687.g8d724284.el9.x86_64 5/37 2026-02-26T02:59:58.968 INFO:teuthology.orchestra.run.trial197.stdout: Installing : python3-rados-2:19.2.3-687.g8d724284.el9.x86_64 6/37 2026-02-26T02:59:58.969 INFO:teuthology.orchestra.run.trial197.stdout: Installing : libcephfs2-2:19.2.3-687.g8d724284.el9.x86_64 7/37 2026-02-26T02:59:58.979 INFO:teuthology.orchestra.run.trial197.stdout: Running scriptlet: libcephfs2-2:19.2.3-687.g8d724284.el9.x86_64 7/37 2026-02-26T02:59:58.983 INFO:teuthology.orchestra.run.trial197.stdout: Installing : liboath-2.6.12-1.el9.x86_64 8/37 2026-02-26T02:59:58.992 INFO:teuthology.orchestra.run.trial197.stdout: Installing : python3-ceph-argparse-2:19.2.3-687.g8d724284.el9.x 9/37 2026-02-26T02:59:59.014 INFO:teuthology.orchestra.run.trial197.stdout: Installing : python3-cephfs-2:19.2.3-687.g8d724284.el9.x86_64 10/37 2026-02-26T02:59:59.015 INFO:teuthology.orchestra.run.trial197.stdout: Installing : libradosstriper1-2:19.2.3-687.g8d724284.el9.x86_64 11/37 2026-02-26T02:59:59.027 INFO:teuthology.orchestra.run.trial197.stdout: Running scriptlet: libradosstriper1-2:19.2.3-687.g8d724284.el9.x86_64 11/37 2026-02-26T02:59:59.059 INFO:teuthology.orchestra.run.trial197.stdout: Installing : re2-1:20211101-20.el9.x86_64 12/37 2026-02-26T02:59:59.063 INFO:teuthology.orchestra.run.trial197.stdout: Installing : libarrow-9.0.0-15.el9.x86_64 13/37 2026-02-26T02:59:59.070 INFO:teuthology.orchestra.run.trial197.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 14/37 2026-02-26T02:59:59.072 INFO:teuthology.orchestra.run.trial197.stdout: Installing : gperftools-libs-2.9.1-3.el9.x86_64 15/37 2026-02-26T02:59:59.075 INFO:teuthology.orchestra.run.trial197.stdout: Installing : libarrow-doc-9.0.0-15.el9.noarch 16/37 2026-02-26T02:59:59.079 INFO:teuthology.orchestra.run.trial197.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 17/37 2026-02-26T02:59:59.222 INFO:teuthology.orchestra.run.trial197.stdout: Installing : libnbd-1.20.3-4.el9.x86_64 18/37 2026-02-26T02:59:59.223 INFO:teuthology.orchestra.run.trial197.stdout: Upgrading : librbd1-2:19.2.3-687.g8d724284.el9.x86_64 19/37 2026-02-26T02:59:59.249 INFO:teuthology.orchestra.run.trial197.stdout: Running scriptlet: librbd1-2:19.2.3-687.g8d724284.el9.x86_64 19/37 2026-02-26T02:59:59.251 INFO:teuthology.orchestra.run.trial197.stdout: Installing : python3-rbd-2:19.2.3-687.g8d724284.el9.x86_64 20/37 2026-02-26T02:59:59.257 INFO:teuthology.orchestra.run.trial197.stdout: Installing : boost-program-options-1.75.0-13.el9.x86_64 21/37 2026-02-26T02:59:59.492 INFO:teuthology.orchestra.run.trial197.stdout: Installing : parquet-libs-9.0.0-15.el9.x86_64 22/37 2026-02-26T02:59:59.493 INFO:teuthology.orchestra.run.trial197.stdout: Installing : librgw2-2:19.2.3-687.g8d724284.el9.x86_64 23/37 2026-02-26T02:59:59.511 INFO:teuthology.orchestra.run.trial197.stdout: Running scriptlet: librgw2-2:19.2.3-687.g8d724284.el9.x86_64 23/37 2026-02-26T02:59:59.515 INFO:teuthology.orchestra.run.trial197.stdout: Installing : python3-rgw-2:19.2.3-687.g8d724284.el9.x86_64 24/37 2026-02-26T02:59:59.520 INFO:teuthology.orchestra.run.trial197.stdout: Installing : pciutils-3.7.0-7.el9.x86_64 25/37 2026-02-26T02:59:59.523 INFO:teuthology.orchestra.run.trial197.stdout: Installing : qatlib-25.08.0-2.el9.x86_64 26/37 2026-02-26T02:59:59.523 INFO:teuthology.orchestra.run.trial197.stdout: Installing : qatlib-service-25.08.0-2.el9.x86_64 27/37 2026-02-26T02:59:59.543 INFO:teuthology.orchestra.run.trial197.stdout: Running scriptlet: qatlib-service-25.08.0-2.el9.x86_64 27/37 2026-02-26T02:59:59.659 INFO:teuthology.orchestra.run.trial197.stdout: Installing : qatzip-libs-1.3.1-1.el9.x86_64 28/37 2026-02-26T02:59:59.661 INFO:teuthology.orchestra.run.trial197.stdout: Installing : nvme-cli-2.16-1.el9.x86_64 29/37 2026-02-26T02:59:59.954 INFO:teuthology.orchestra.run.trial197.stdout: Running scriptlet: nvme-cli-2.16-1.el9.x86_64 29/37 2026-02-26T02:59:59.954 INFO:teuthology.orchestra.run.trial197.stdout:Created symlink /etc/systemd/system/default.target.wants/nvmefc-boot-connections.service → /usr/lib/systemd/system/nvmefc-boot-connections.service. 2026-02-26T02:59:59.954 INFO:teuthology.orchestra.run.trial197.stdout: 2026-02-26T02:59:59.990 INFO:teuthology.orchestra.run.trial197.stdout: Installing : mailcap-2.1.49-5.el9.noarch 30/37 2026-02-26T02:59:59.992 INFO:teuthology.orchestra.run.trial197.stdout: Installing : python3-ceph-common-2:19.2.3-687.g8d724284.el9.x86 31/37 2026-02-26T03:00:01.025 INFO:teuthology.orchestra.run.trial197.stdout: Running scriptlet: ceph-common-2:19.2.3-687.g8d724284.el9.x86_64 32/37 2026-02-26T03:00:01.027 INFO:teuthology.orchestra.run.trial197.stdout: Installing : ceph-common-2:19.2.3-687.g8d724284.el9.x86_64 32/37 2026-02-26T03:00:01.317 INFO:teuthology.orchestra.run.trial197.stdout: Running scriptlet: ceph-common-2:19.2.3-687.g8d724284.el9.x86_64 32/37 2026-02-26T03:00:01.320 INFO:teuthology.orchestra.run.trial197.stdout: Installing : ceph-base-2:19.2.3-687.g8d724284.el9.x86_64 33/37 2026-02-26T03:00:01.363 INFO:teuthology.orchestra.run.trial197.stdout: Running scriptlet: ceph-base-2:19.2.3-687.g8d724284.el9.x86_64 33/37 2026-02-26T03:00:01.363 INFO:teuthology.orchestra.run.trial197.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2026-02-26T03:00:01.363 INFO:teuthology.orchestra.run.trial197.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2026-02-26T03:00:01.363 INFO:teuthology.orchestra.run.trial197.stdout: 2026-02-26T03:00:01.369 INFO:teuthology.orchestra.run.trial197.stdout: Installing : ceph-selinux-2:19.2.3-687.g8d724284.el9.x86_64 34/37 2026-02-26T03:00:05.920 INFO:teuthology.orchestra.run.trial197.stdout: Running scriptlet: ceph-selinux-2:19.2.3-687.g8d724284.el9.x86_64 34/37 2026-02-26T03:00:05.920 INFO:teuthology.orchestra.run.trial197.stdout:skipping the directory /sys 2026-02-26T03:00:05.920 INFO:teuthology.orchestra.run.trial197.stdout:skipping the directory /proc 2026-02-26T03:00:05.920 INFO:teuthology.orchestra.run.trial197.stdout:skipping the directory /mnt 2026-02-26T03:00:05.920 INFO:teuthology.orchestra.run.trial197.stdout:skipping the directory /var/tmp 2026-02-26T03:00:05.920 INFO:teuthology.orchestra.run.trial197.stdout:skipping the directory /home 2026-02-26T03:00:05.920 INFO:teuthology.orchestra.run.trial197.stdout:skipping the directory /root 2026-02-26T03:00:05.920 INFO:teuthology.orchestra.run.trial197.stdout:skipping the directory /tmp 2026-02-26T03:00:05.920 INFO:teuthology.orchestra.run.trial197.stdout: 2026-02-26T03:00:06.392 INFO:teuthology.orchestra.run.trial197.stdout: Installing : ceph-radosgw-2:19.2.3-687.g8d724284.el9.x86_64 35/37 2026-02-26T03:00:06.416 INFO:teuthology.orchestra.run.trial197.stdout: Running scriptlet: ceph-radosgw-2:19.2.3-687.g8d724284.el9.x86_64 35/37 2026-02-26T03:00:06.416 INFO:teuthology.orchestra.run.trial197.stdout:Glob pattern passed to enable, but globs are not supported for this. 2026-02-26T03:00:06.416 INFO:teuthology.orchestra.run.trial197.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2026-02-26T03:00:06.416 INFO:teuthology.orchestra.run.trial197.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2026-02-26T03:00:06.416 INFO:teuthology.orchestra.run.trial197.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2026-02-26T03:00:06.417 INFO:teuthology.orchestra.run.trial197.stdout: 2026-02-26T03:00:06.418 INFO:teuthology.orchestra.run.trial197.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 36/37 2026-02-26T03:00:06.431 INFO:teuthology.orchestra.run.trial197.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 36/37 2026-02-26T03:00:06.431 INFO:teuthology.orchestra.run.trial197.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 37/37 2026-02-26T03:00:06.488 INFO:teuthology.orchestra.run.trial197.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 37/37 2026-02-26T03:00:06.489 INFO:teuthology.orchestra.run.trial197.stdout:Creating group 'qat' with GID 992. 2026-02-26T03:00:06.489 INFO:teuthology.orchestra.run.trial197.stdout: 2026-02-26T03:00:06.773 INFO:teuthology.orchestra.run.trial197.stdout: Verifying : ceph-base-2:19.2.3-687.g8d724284.el9.x86_64 1/37 2026-02-26T03:00:06.773 INFO:teuthology.orchestra.run.trial197.stdout: Verifying : ceph-common-2:19.2.3-687.g8d724284.el9.x86_64 2/37 2026-02-26T03:00:06.773 INFO:teuthology.orchestra.run.trial197.stdout: Verifying : ceph-radosgw-2:19.2.3-687.g8d724284.el9.x86_64 3/37 2026-02-26T03:00:06.773 INFO:teuthology.orchestra.run.trial197.stdout: Verifying : ceph-selinux-2:19.2.3-687.g8d724284.el9.x86_64 4/37 2026-02-26T03:00:06.773 INFO:teuthology.orchestra.run.trial197.stdout: Verifying : libcephfs2-2:19.2.3-687.g8d724284.el9.x86_64 5/37 2026-02-26T03:00:06.774 INFO:teuthology.orchestra.run.trial197.stdout: Verifying : libradosstriper1-2:19.2.3-687.g8d724284.el9.x86_64 6/37 2026-02-26T03:00:06.774 INFO:teuthology.orchestra.run.trial197.stdout: Verifying : librgw2-2:19.2.3-687.g8d724284.el9.x86_64 7/37 2026-02-26T03:00:06.774 INFO:teuthology.orchestra.run.trial197.stdout: Verifying : python3-ceph-argparse-2:19.2.3-687.g8d724284.el9.x 8/37 2026-02-26T03:00:06.774 INFO:teuthology.orchestra.run.trial197.stdout: Verifying : python3-ceph-common-2:19.2.3-687.g8d724284.el9.x86 9/37 2026-02-26T03:00:06.774 INFO:teuthology.orchestra.run.trial197.stdout: Verifying : python3-cephfs-2:19.2.3-687.g8d724284.el9.x86_64 10/37 2026-02-26T03:00:06.774 INFO:teuthology.orchestra.run.trial197.stdout: Verifying : python3-rados-2:19.2.3-687.g8d724284.el9.x86_64 11/37 2026-02-26T03:00:06.774 INFO:teuthology.orchestra.run.trial197.stdout: Verifying : python3-rbd-2:19.2.3-687.g8d724284.el9.x86_64 12/37 2026-02-26T03:00:06.774 INFO:teuthology.orchestra.run.trial197.stdout: Verifying : python3-rgw-2:19.2.3-687.g8d724284.el9.x86_64 13/37 2026-02-26T03:00:06.775 INFO:teuthology.orchestra.run.trial197.stdout: Verifying : mailcap-2.1.49-5.el9.noarch 14/37 2026-02-26T03:00:06.775 INFO:teuthology.orchestra.run.trial197.stdout: Verifying : nvme-cli-2.16-1.el9.x86_64 15/37 2026-02-26T03:00:06.775 INFO:teuthology.orchestra.run.trial197.stdout: Verifying : pciutils-3.7.0-7.el9.x86_64 16/37 2026-02-26T03:00:06.775 INFO:teuthology.orchestra.run.trial197.stdout: Verifying : boost-program-options-1.75.0-13.el9.x86_64 17/37 2026-02-26T03:00:06.775 INFO:teuthology.orchestra.run.trial197.stdout: Verifying : libnbd-1.20.3-4.el9.x86_64 18/37 2026-02-26T03:00:06.775 INFO:teuthology.orchestra.run.trial197.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 19/37 2026-02-26T03:00:06.775 INFO:teuthology.orchestra.run.trial197.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 20/37 2026-02-26T03:00:06.775 INFO:teuthology.orchestra.run.trial197.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 21/37 2026-02-26T03:00:06.776 INFO:teuthology.orchestra.run.trial197.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 22/37 2026-02-26T03:00:06.776 INFO:teuthology.orchestra.run.trial197.stdout: Verifying : qatlib-25.08.0-2.el9.x86_64 23/37 2026-02-26T03:00:06.776 INFO:teuthology.orchestra.run.trial197.stdout: Verifying : qatlib-service-25.08.0-2.el9.x86_64 24/37 2026-02-26T03:00:06.776 INFO:teuthology.orchestra.run.trial197.stdout: Verifying : qatzip-libs-1.3.1-1.el9.x86_64 25/37 2026-02-26T03:00:06.776 INFO:teuthology.orchestra.run.trial197.stdout: Verifying : gperftools-libs-2.9.1-3.el9.x86_64 26/37 2026-02-26T03:00:06.776 INFO:teuthology.orchestra.run.trial197.stdout: Verifying : libarrow-9.0.0-15.el9.x86_64 27/37 2026-02-26T03:00:06.776 INFO:teuthology.orchestra.run.trial197.stdout: Verifying : libarrow-doc-9.0.0-15.el9.noarch 28/37 2026-02-26T03:00:06.776 INFO:teuthology.orchestra.run.trial197.stdout: Verifying : liboath-2.6.12-1.el9.x86_64 29/37 2026-02-26T03:00:06.777 INFO:teuthology.orchestra.run.trial197.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 30/37 2026-02-26T03:00:06.777 INFO:teuthology.orchestra.run.trial197.stdout: Verifying : parquet-libs-9.0.0-15.el9.x86_64 31/37 2026-02-26T03:00:06.777 INFO:teuthology.orchestra.run.trial197.stdout: Verifying : re2-1:20211101-20.el9.x86_64 32/37 2026-02-26T03:00:06.777 INFO:teuthology.orchestra.run.trial197.stdout: Verifying : thrift-0.15.0-4.el9.x86_64 33/37 2026-02-26T03:00:06.777 INFO:teuthology.orchestra.run.trial197.stdout: Verifying : librados2-2:19.2.3-687.g8d724284.el9.x86_64 34/37 2026-02-26T03:00:06.777 INFO:teuthology.orchestra.run.trial197.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 35/37 2026-02-26T03:00:06.777 INFO:teuthology.orchestra.run.trial197.stdout: Verifying : librbd1-2:19.2.3-687.g8d724284.el9.x86_64 36/37 2026-02-26T03:00:06.837 INFO:teuthology.orchestra.run.trial197.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 37/37 2026-02-26T03:00:06.837 INFO:teuthology.orchestra.run.trial197.stdout: 2026-02-26T03:00:06.838 INFO:teuthology.orchestra.run.trial197.stdout:Upgraded: 2026-02-26T03:00:06.838 INFO:teuthology.orchestra.run.trial197.stdout: librados2-2:19.2.3-687.g8d724284.el9.x86_64 2026-02-26T03:00:06.838 INFO:teuthology.orchestra.run.trial197.stdout: librbd1-2:19.2.3-687.g8d724284.el9.x86_64 2026-02-26T03:00:06.838 INFO:teuthology.orchestra.run.trial197.stdout:Installed: 2026-02-26T03:00:06.838 INFO:teuthology.orchestra.run.trial197.stdout: boost-program-options-1.75.0-13.el9.x86_64 2026-02-26T03:00:06.838 INFO:teuthology.orchestra.run.trial197.stdout: ceph-base-2:19.2.3-687.g8d724284.el9.x86_64 2026-02-26T03:00:06.838 INFO:teuthology.orchestra.run.trial197.stdout: ceph-common-2:19.2.3-687.g8d724284.el9.x86_64 2026-02-26T03:00:06.838 INFO:teuthology.orchestra.run.trial197.stdout: ceph-radosgw-2:19.2.3-687.g8d724284.el9.x86_64 2026-02-26T03:00:06.838 INFO:teuthology.orchestra.run.trial197.stdout: ceph-selinux-2:19.2.3-687.g8d724284.el9.x86_64 2026-02-26T03:00:06.838 INFO:teuthology.orchestra.run.trial197.stdout: gperftools-libs-2.9.1-3.el9.x86_64 2026-02-26T03:00:06.839 INFO:teuthology.orchestra.run.trial197.stdout: libarrow-9.0.0-15.el9.x86_64 2026-02-26T03:00:06.839 INFO:teuthology.orchestra.run.trial197.stdout: libarrow-doc-9.0.0-15.el9.noarch 2026-02-26T03:00:06.839 INFO:teuthology.orchestra.run.trial197.stdout: libcephfs2-2:19.2.3-687.g8d724284.el9.x86_64 2026-02-26T03:00:06.839 INFO:teuthology.orchestra.run.trial197.stdout: libnbd-1.20.3-4.el9.x86_64 2026-02-26T03:00:06.839 INFO:teuthology.orchestra.run.trial197.stdout: liboath-2.6.12-1.el9.x86_64 2026-02-26T03:00:06.839 INFO:teuthology.orchestra.run.trial197.stdout: libpmemobj-1.12.1-1.el9.x86_64 2026-02-26T03:00:06.839 INFO:teuthology.orchestra.run.trial197.stdout: librabbitmq-0.11.0-7.el9.x86_64 2026-02-26T03:00:06.839 INFO:teuthology.orchestra.run.trial197.stdout: libradosstriper1-2:19.2.3-687.g8d724284.el9.x86_64 2026-02-26T03:00:06.839 INFO:teuthology.orchestra.run.trial197.stdout: librdkafka-1.6.1-102.el9.x86_64 2026-02-26T03:00:06.839 INFO:teuthology.orchestra.run.trial197.stdout: librgw2-2:19.2.3-687.g8d724284.el9.x86_64 2026-02-26T03:00:06.839 INFO:teuthology.orchestra.run.trial197.stdout: libunwind-1.6.2-1.el9.x86_64 2026-02-26T03:00:06.839 INFO:teuthology.orchestra.run.trial197.stdout: lttng-ust-2.12.0-6.el9.x86_64 2026-02-26T03:00:06.840 INFO:teuthology.orchestra.run.trial197.stdout: mailcap-2.1.49-5.el9.noarch 2026-02-26T03:00:06.840 INFO:teuthology.orchestra.run.trial197.stdout: nvme-cli-2.16-1.el9.x86_64 2026-02-26T03:00:06.840 INFO:teuthology.orchestra.run.trial197.stdout: parquet-libs-9.0.0-15.el9.x86_64 2026-02-26T03:00:06.840 INFO:teuthology.orchestra.run.trial197.stdout: pciutils-3.7.0-7.el9.x86_64 2026-02-26T03:00:06.840 INFO:teuthology.orchestra.run.trial197.stdout: python3-ceph-argparse-2:19.2.3-687.g8d724284.el9.x86_64 2026-02-26T03:00:06.840 INFO:teuthology.orchestra.run.trial197.stdout: python3-ceph-common-2:19.2.3-687.g8d724284.el9.x86_64 2026-02-26T03:00:06.840 INFO:teuthology.orchestra.run.trial197.stdout: python3-cephfs-2:19.2.3-687.g8d724284.el9.x86_64 2026-02-26T03:00:06.840 INFO:teuthology.orchestra.run.trial197.stdout: python3-rados-2:19.2.3-687.g8d724284.el9.x86_64 2026-02-26T03:00:06.840 INFO:teuthology.orchestra.run.trial197.stdout: python3-rbd-2:19.2.3-687.g8d724284.el9.x86_64 2026-02-26T03:00:06.840 INFO:teuthology.orchestra.run.trial197.stdout: python3-rgw-2:19.2.3-687.g8d724284.el9.x86_64 2026-02-26T03:00:06.840 INFO:teuthology.orchestra.run.trial197.stdout: qatlib-25.08.0-2.el9.x86_64 2026-02-26T03:00:06.840 INFO:teuthology.orchestra.run.trial197.stdout: qatlib-service-25.08.0-2.el9.x86_64 2026-02-26T03:00:06.841 INFO:teuthology.orchestra.run.trial197.stdout: qatzip-libs-1.3.1-1.el9.x86_64 2026-02-26T03:00:06.841 INFO:teuthology.orchestra.run.trial197.stdout: re2-1:20211101-20.el9.x86_64 2026-02-26T03:00:06.841 INFO:teuthology.orchestra.run.trial197.stdout: thrift-0.15.0-4.el9.x86_64 2026-02-26T03:00:06.841 INFO:teuthology.orchestra.run.trial197.stdout: 2026-02-26T03:00:06.841 INFO:teuthology.orchestra.run.trial197.stdout:Complete! 2026-02-26T03:00:06.881 DEBUG:teuthology.orchestra.run.trial197:> sudo yum -y install ceph-test 2026-02-26T03:00:07.200 INFO:teuthology.orchestra.run.trial197.stdout:Last metadata expiration check: 0:00:14 ago on Thu 26 Feb 2026 02:59:53 AM UTC. 2026-02-26T03:00:07.284 INFO:teuthology.orchestra.run.trial197.stdout:Dependencies resolved. 2026-02-26T03:00:07.285 INFO:teuthology.orchestra.run.trial197.stdout:================================================================================ 2026-02-26T03:00:07.285 INFO:teuthology.orchestra.run.trial197.stdout: Package Arch Version Repository Size 2026-02-26T03:00:07.285 INFO:teuthology.orchestra.run.trial197.stdout:================================================================================ 2026-02-26T03:00:07.285 INFO:teuthology.orchestra.run.trial197.stdout:Installing: 2026-02-26T03:00:07.285 INFO:teuthology.orchestra.run.trial197.stdout: ceph-test x86_64 2:19.2.3-687.g8d724284.el9 ceph 51 M 2026-02-26T03:00:07.285 INFO:teuthology.orchestra.run.trial197.stdout:Installing dependencies: 2026-02-26T03:00:07.285 INFO:teuthology.orchestra.run.trial197.stdout: libcephsqlite x86_64 2:19.2.3-687.g8d724284.el9 ceph 163 k 2026-02-26T03:00:07.285 INFO:teuthology.orchestra.run.trial197.stdout: libxslt x86_64 1.1.34-12.el9 appstream 233 k 2026-02-26T03:00:07.286 INFO:teuthology.orchestra.run.trial197.stdout: socat x86_64 1.7.4.1-8.el9 appstream 303 k 2026-02-26T03:00:07.286 INFO:teuthology.orchestra.run.trial197.stdout: xmlstarlet x86_64 1.6.1-20.el9 appstream 64 k 2026-02-26T03:00:07.286 INFO:teuthology.orchestra.run.trial197.stdout: 2026-02-26T03:00:07.286 INFO:teuthology.orchestra.run.trial197.stdout:Transaction Summary 2026-02-26T03:00:07.286 INFO:teuthology.orchestra.run.trial197.stdout:================================================================================ 2026-02-26T03:00:07.286 INFO:teuthology.orchestra.run.trial197.stdout:Install 5 Packages 2026-02-26T03:00:07.286 INFO:teuthology.orchestra.run.trial197.stdout: 2026-02-26T03:00:07.286 INFO:teuthology.orchestra.run.trial197.stdout:Total download size: 51 M 2026-02-26T03:00:07.286 INFO:teuthology.orchestra.run.trial197.stdout:Installed size: 222 M 2026-02-26T03:00:07.286 INFO:teuthology.orchestra.run.trial197.stdout:Downloading Packages: 2026-02-26T03:00:07.674 INFO:teuthology.orchestra.run.trial197.stdout:(1/5): libcephsqlite-19.2.3-687.g8d724284.el9.x 1.9 MB/s | 163 kB 00:00 2026-02-26T03:00:07.680 INFO:teuthology.orchestra.run.trial197.stdout:(2/5): libxslt-1.1.34-12.el9.x86_64.rpm 2.5 MB/s | 233 kB 00:00 2026-02-26T03:00:07.702 INFO:teuthology.orchestra.run.trial197.stdout:(3/5): xmlstarlet-1.6.1-20.el9.x86_64.rpm 2.8 MB/s | 64 kB 00:00 2026-02-26T03:00:07.780 INFO:teuthology.orchestra.run.trial197.stdout:(4/5): socat-1.7.4.1-8.el9.x86_64.rpm 2.8 MB/s | 303 kB 00:00 2026-02-26T03:00:08.627 INFO:teuthology.orchestra.run.trial197.stdout:(5/5): ceph-test-19.2.3-687.g8d724284.el9.x86_6 49 MB/s | 51 MB 00:01 2026-02-26T03:00:08.628 INFO:teuthology.orchestra.run.trial197.stdout:-------------------------------------------------------------------------------- 2026-02-26T03:00:08.628 INFO:teuthology.orchestra.run.trial197.stdout:Total 38 MB/s | 51 MB 00:01 2026-02-26T03:00:08.655 INFO:teuthology.orchestra.run.trial197.stdout:Running transaction check 2026-02-26T03:00:08.664 INFO:teuthology.orchestra.run.trial197.stdout:Transaction check succeeded. 2026-02-26T03:00:08.665 INFO:teuthology.orchestra.run.trial197.stdout:Running transaction test 2026-02-26T03:00:08.853 INFO:teuthology.orchestra.run.trial197.stdout:Transaction test succeeded. 2026-02-26T03:00:08.854 INFO:teuthology.orchestra.run.trial197.stdout:Running transaction 2026-02-26T03:00:09.072 INFO:teuthology.orchestra.run.trial197.stdout: Preparing : 1/1 2026-02-26T03:00:09.077 INFO:teuthology.orchestra.run.trial197.stdout: Installing : socat-1.7.4.1-8.el9.x86_64 1/5 2026-02-26T03:00:09.081 INFO:teuthology.orchestra.run.trial197.stdout: Installing : libxslt-1.1.34-12.el9.x86_64 2/5 2026-02-26T03:00:09.089 INFO:teuthology.orchestra.run.trial197.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 3/5 2026-02-26T03:00:09.090 INFO:teuthology.orchestra.run.trial197.stdout: Installing : libcephsqlite-2:19.2.3-687.g8d724284.el9.x86_64 4/5 2026-02-26T03:00:11.396 INFO:teuthology.orchestra.run.trial197.stdout: Running scriptlet: libcephsqlite-2:19.2.3-687.g8d724284.el9.x86_64 4/5 2026-02-26T03:00:11.415 INFO:teuthology.orchestra.run.trial197.stdout: Installing : ceph-test-2:19.2.3-687.g8d724284.el9.x86_64 5/5 2026-02-26T03:00:11.645 INFO:teuthology.orchestra.run.trial197.stdout: Running scriptlet: ceph-test-2:19.2.3-687.g8d724284.el9.x86_64 5/5 2026-02-26T03:00:11.646 INFO:teuthology.orchestra.run.trial197.stdout: Verifying : ceph-test-2:19.2.3-687.g8d724284.el9.x86_64 1/5 2026-02-26T03:00:11.646 INFO:teuthology.orchestra.run.trial197.stdout: Verifying : libcephsqlite-2:19.2.3-687.g8d724284.el9.x86_64 2/5 2026-02-26T03:00:11.646 INFO:teuthology.orchestra.run.trial197.stdout: Verifying : libxslt-1.1.34-12.el9.x86_64 3/5 2026-02-26T03:00:11.646 INFO:teuthology.orchestra.run.trial197.stdout: Verifying : socat-1.7.4.1-8.el9.x86_64 4/5 2026-02-26T03:00:11.706 INFO:teuthology.orchestra.run.trial197.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 5/5 2026-02-26T03:00:11.706 INFO:teuthology.orchestra.run.trial197.stdout: 2026-02-26T03:00:11.706 INFO:teuthology.orchestra.run.trial197.stdout:Installed: 2026-02-26T03:00:11.706 INFO:teuthology.orchestra.run.trial197.stdout: ceph-test-2:19.2.3-687.g8d724284.el9.x86_64 2026-02-26T03:00:11.706 INFO:teuthology.orchestra.run.trial197.stdout: libcephsqlite-2:19.2.3-687.g8d724284.el9.x86_64 2026-02-26T03:00:11.706 INFO:teuthology.orchestra.run.trial197.stdout: libxslt-1.1.34-12.el9.x86_64 2026-02-26T03:00:11.706 INFO:teuthology.orchestra.run.trial197.stdout: socat-1.7.4.1-8.el9.x86_64 2026-02-26T03:00:11.706 INFO:teuthology.orchestra.run.trial197.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2026-02-26T03:00:11.706 INFO:teuthology.orchestra.run.trial197.stdout: 2026-02-26T03:00:11.706 INFO:teuthology.orchestra.run.trial197.stdout:Complete! 2026-02-26T03:00:11.738 DEBUG:teuthology.orchestra.run.trial197:> sudo yum -y install ceph 2026-02-26T03:00:12.048 INFO:teuthology.orchestra.run.trial197.stdout:Last metadata expiration check: 0:00:19 ago on Thu 26 Feb 2026 02:59:53 AM UTC. 2026-02-26T03:00:12.134 INFO:teuthology.orchestra.run.trial197.stdout:Dependencies resolved. 2026-02-26T03:00:12.137 INFO:teuthology.orchestra.run.trial197.stdout:====================================================================================== 2026-02-26T03:00:12.137 INFO:teuthology.orchestra.run.trial197.stdout: Package Arch Version Repository Size 2026-02-26T03:00:12.137 INFO:teuthology.orchestra.run.trial197.stdout:====================================================================================== 2026-02-26T03:00:12.137 INFO:teuthology.orchestra.run.trial197.stdout:Installing: 2026-02-26T03:00:12.137 INFO:teuthology.orchestra.run.trial197.stdout: ceph x86_64 2:19.2.3-687.g8d724284.el9 ceph 6.5 k 2026-02-26T03:00:12.137 INFO:teuthology.orchestra.run.trial197.stdout:Installing dependencies: 2026-02-26T03:00:12.138 INFO:teuthology.orchestra.run.trial197.stdout: abseil-cpp x86_64 20211102.0-4.el9 epel 551 k 2026-02-26T03:00:12.138 INFO:teuthology.orchestra.run.trial197.stdout: ceph-grafana-dashboards noarch 2:19.2.3-687.g8d724284.el9 ceph-noarch 31 k 2026-02-26T03:00:12.138 INFO:teuthology.orchestra.run.trial197.stdout: ceph-mds x86_64 2:19.2.3-687.g8d724284.el9 ceph 2.4 M 2026-02-26T03:00:12.138 INFO:teuthology.orchestra.run.trial197.stdout: ceph-mgr x86_64 2:19.2.3-687.g8d724284.el9 ceph 1.1 M 2026-02-26T03:00:12.138 INFO:teuthology.orchestra.run.trial197.stdout: ceph-mgr-modules-core noarch 2:19.2.3-687.g8d724284.el9 ceph-noarch 253 k 2026-02-26T03:00:12.138 INFO:teuthology.orchestra.run.trial197.stdout: ceph-mon x86_64 2:19.2.3-687.g8d724284.el9 ceph 4.7 M 2026-02-26T03:00:12.138 INFO:teuthology.orchestra.run.trial197.stdout: ceph-osd x86_64 2:19.2.3-687.g8d724284.el9 ceph 17 M 2026-02-26T03:00:12.138 INFO:teuthology.orchestra.run.trial197.stdout: ceph-prometheus-alerts noarch 2:19.2.3-687.g8d724284.el9 ceph-noarch 17 k 2026-02-26T03:00:12.138 INFO:teuthology.orchestra.run.trial197.stdout: cephadm noarch 2:19.2.3-687.g8d724284.el9 ceph-noarch 769 k 2026-02-26T03:00:12.138 INFO:teuthology.orchestra.run.trial197.stdout: cryptsetup x86_64 2.8.1-3.el9 baseos 351 k 2026-02-26T03:00:12.138 INFO:teuthology.orchestra.run.trial197.stdout: flexiblas x86_64 3.0.4-9.el9 appstream 30 k 2026-02-26T03:00:12.138 INFO:teuthology.orchestra.run.trial197.stdout: flexiblas-netlib x86_64 3.0.4-9.el9 appstream 3.0 M 2026-02-26T03:00:12.139 INFO:teuthology.orchestra.run.trial197.stdout: flexiblas-openblas-openmp x86_64 3.0.4-9.el9 appstream 15 k 2026-02-26T03:00:12.139 INFO:teuthology.orchestra.run.trial197.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2026-02-26T03:00:12.139 INFO:teuthology.orchestra.run.trial197.stdout: ledmon-libs x86_64 1.1.0-3.el9 baseos 40 k 2026-02-26T03:00:12.139 INFO:teuthology.orchestra.run.trial197.stdout: libconfig x86_64 1.7.2-9.el9 baseos 72 k 2026-02-26T03:00:12.139 INFO:teuthology.orchestra.run.trial197.stdout: libgfortran x86_64 11.5.0-14.el9 baseos 794 k 2026-02-26T03:00:12.139 INFO:teuthology.orchestra.run.trial197.stdout: libquadmath x86_64 11.5.0-14.el9 baseos 184 k 2026-02-26T03:00:12.139 INFO:teuthology.orchestra.run.trial197.stdout: libstoragemgmt x86_64 1.10.1-1.el9 appstream 246 k 2026-02-26T03:00:12.139 INFO:teuthology.orchestra.run.trial197.stdout: lua x86_64 5.4.4-4.el9 appstream 188 k 2026-02-26T03:00:12.139 INFO:teuthology.orchestra.run.trial197.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2026-02-26T03:00:12.140 INFO:teuthology.orchestra.run.trial197.stdout: luarocks noarch 3.9.2-5.el9 epel 151 k 2026-02-26T03:00:12.140 INFO:teuthology.orchestra.run.trial197.stdout: openblas x86_64 0.3.29-1.el9 appstream 42 k 2026-02-26T03:00:12.140 INFO:teuthology.orchestra.run.trial197.stdout: openblas-openmp x86_64 0.3.29-1.el9 appstream 5.3 M 2026-02-26T03:00:12.140 INFO:teuthology.orchestra.run.trial197.stdout: protobuf x86_64 3.14.0-17.el9 appstream 1.0 M 2026-02-26T03:00:12.140 INFO:teuthology.orchestra.run.trial197.stdout: protobuf-compiler x86_64 3.14.0-17.el9 crb 862 k 2026-02-26T03:00:12.140 INFO:teuthology.orchestra.run.trial197.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2026-02-26T03:00:12.140 INFO:teuthology.orchestra.run.trial197.stdout: python3-autocommand noarch 2.2.2-8.el9 epel 29 k 2026-02-26T03:00:12.140 INFO:teuthology.orchestra.run.trial197.stdout: python3-babel noarch 2.9.1-2.el9 appstream 6.0 M 2026-02-26T03:00:12.140 INFO:teuthology.orchestra.run.trial197.stdout: python3-backports-tarfile noarch 1.2.0-1.el9 epel 60 k 2026-02-26T03:00:12.140 INFO:teuthology.orchestra.run.trial197.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2026-02-26T03:00:12.140 INFO:teuthology.orchestra.run.trial197.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2026-02-26T03:00:12.141 INFO:teuthology.orchestra.run.trial197.stdout: python3-certifi noarch 2023.05.07-4.el9 epel 14 k 2026-02-26T03:00:12.141 INFO:teuthology.orchestra.run.trial197.stdout: python3-cffi x86_64 1.14.5-5.el9 baseos 253 k 2026-02-26T03:00:12.141 INFO:teuthology.orchestra.run.trial197.stdout: python3-cheroot noarch 10.0.1-4.el9 epel 173 k 2026-02-26T03:00:12.141 INFO:teuthology.orchestra.run.trial197.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2026-02-26T03:00:12.141 INFO:teuthology.orchestra.run.trial197.stdout: python3-cryptography x86_64 36.0.1-5.el9 baseos 1.2 M 2026-02-26T03:00:12.141 INFO:teuthology.orchestra.run.trial197.stdout: python3-devel x86_64 3.9.25-3.el9 appstream 244 k 2026-02-26T03:00:12.141 INFO:teuthology.orchestra.run.trial197.stdout: python3-google-auth noarch 1:2.45.0-1.el9 epel 254 k 2026-02-26T03:00:12.141 INFO:teuthology.orchestra.run.trial197.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2026-02-26T03:00:12.141 INFO:teuthology.orchestra.run.trial197.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2026-02-26T03:00:12.141 INFO:teuthology.orchestra.run.trial197.stdout: python3-isodate noarch 0.6.1-3.el9 epel 56 k 2026-02-26T03:00:12.142 INFO:teuthology.orchestra.run.trial197.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2026-02-26T03:00:12.142 INFO:teuthology.orchestra.run.trial197.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2026-02-26T03:00:12.142 INFO:teuthology.orchestra.run.trial197.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2026-02-26T03:00:12.142 INFO:teuthology.orchestra.run.trial197.stdout: python3-jaraco-context noarch 6.0.1-3.el9 epel 20 k 2026-02-26T03:00:12.142 INFO:teuthology.orchestra.run.trial197.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2026-02-26T03:00:12.142 INFO:teuthology.orchestra.run.trial197.stdout: python3-jaraco-text noarch 4.0.0-2.el9 epel 26 k 2026-02-26T03:00:12.142 INFO:teuthology.orchestra.run.trial197.stdout: python3-jinja2 noarch 2.11.3-8.el9 appstream 249 k 2026-02-26T03:00:12.142 INFO:teuthology.orchestra.run.trial197.stdout: python3-kubernetes noarch 1:26.1.0-3.el9 epel 1.0 M 2026-02-26T03:00:12.142 INFO:teuthology.orchestra.run.trial197.stdout: python3-libstoragemgmt x86_64 1.10.1-1.el9 appstream 177 k 2026-02-26T03:00:12.143 INFO:teuthology.orchestra.run.trial197.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2026-02-26T03:00:12.143 INFO:teuthology.orchestra.run.trial197.stdout: python3-lxml x86_64 4.6.5-3.el9 appstream 1.2 M 2026-02-26T03:00:12.143 INFO:teuthology.orchestra.run.trial197.stdout: python3-mako noarch 1.1.4-6.el9 appstream 172 k 2026-02-26T03:00:12.143 INFO:teuthology.orchestra.run.trial197.stdout: python3-markupsafe x86_64 1.1.1-12.el9 appstream 35 k 2026-02-26T03:00:12.143 INFO:teuthology.orchestra.run.trial197.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2026-02-26T03:00:12.143 INFO:teuthology.orchestra.run.trial197.stdout: python3-msgpack x86_64 1.0.3-2.el9 epel 86 k 2026-02-26T03:00:12.143 INFO:teuthology.orchestra.run.trial197.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2026-02-26T03:00:12.143 INFO:teuthology.orchestra.run.trial197.stdout: python3-numpy x86_64 1:1.23.5-2.el9 appstream 6.1 M 2026-02-26T03:00:12.143 INFO:teuthology.orchestra.run.trial197.stdout: python3-numpy-f2py x86_64 1:1.23.5-2.el9 appstream 442 k 2026-02-26T03:00:12.143 INFO:teuthology.orchestra.run.trial197.stdout: python3-packaging noarch 20.9-5.el9 appstream 77 k 2026-02-26T03:00:12.144 INFO:teuthology.orchestra.run.trial197.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2026-02-26T03:00:12.144 INFO:teuthology.orchestra.run.trial197.stdout: python3-ply noarch 3.11-14.el9 baseos 106 k 2026-02-26T03:00:12.144 INFO:teuthology.orchestra.run.trial197.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2026-02-26T03:00:12.144 INFO:teuthology.orchestra.run.trial197.stdout: python3-protobuf noarch 3.14.0-17.el9 appstream 267 k 2026-02-26T03:00:12.144 INFO:teuthology.orchestra.run.trial197.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2026-02-26T03:00:12.144 INFO:teuthology.orchestra.run.trial197.stdout: python3-pyasn1 noarch 0.4.8-7.el9 appstream 157 k 2026-02-26T03:00:12.144 INFO:teuthology.orchestra.run.trial197.stdout: python3-pyasn1-modules noarch 0.4.8-7.el9 appstream 277 k 2026-02-26T03:00:12.144 INFO:teuthology.orchestra.run.trial197.stdout: python3-pycparser noarch 2.20-6.el9 baseos 135 k 2026-02-26T03:00:12.144 INFO:teuthology.orchestra.run.trial197.stdout: python3-pyparsing noarch 2.4.7-9.el9 baseos 150 k 2026-02-26T03:00:12.144 INFO:teuthology.orchestra.run.trial197.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2026-02-26T03:00:12.145 INFO:teuthology.orchestra.run.trial197.stdout: python3-requests noarch 2.25.1-10.el9 baseos 126 k 2026-02-26T03:00:12.145 INFO:teuthology.orchestra.run.trial197.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 appstream 54 k 2026-02-26T03:00:12.145 INFO:teuthology.orchestra.run.trial197.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2026-02-26T03:00:12.145 INFO:teuthology.orchestra.run.trial197.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2026-02-26T03:00:12.145 INFO:teuthology.orchestra.run.trial197.stdout: python3-scipy x86_64 1.9.3-2.el9 appstream 19 M 2026-02-26T03:00:12.145 INFO:teuthology.orchestra.run.trial197.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2026-02-26T03:00:12.145 INFO:teuthology.orchestra.run.trial197.stdout: python3-toml noarch 0.10.2-6.el9 appstream 42 k 2026-02-26T03:00:12.145 INFO:teuthology.orchestra.run.trial197.stdout: python3-typing-extensions noarch 4.15.0-1.el9 epel 86 k 2026-02-26T03:00:12.145 INFO:teuthology.orchestra.run.trial197.stdout: python3-urllib3 noarch 1.26.5-7.el9 baseos 218 k 2026-02-26T03:00:12.146 INFO:teuthology.orchestra.run.trial197.stdout: python3-webob noarch 1.8.8-2.el9 epel 230 k 2026-02-26T03:00:12.146 INFO:teuthology.orchestra.run.trial197.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2026-02-26T03:00:12.146 INFO:teuthology.orchestra.run.trial197.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2026-02-26T03:00:12.146 INFO:teuthology.orchestra.run.trial197.stdout: python3-xmlsec x86_64 1.3.13-1.el9 epel 48 k 2026-02-26T03:00:12.146 INFO:teuthology.orchestra.run.trial197.stdout: python3-xmltodict noarch 0.12.0-15.el9 epel 22 k 2026-02-26T03:00:12.146 INFO:teuthology.orchestra.run.trial197.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2026-02-26T03:00:12.146 INFO:teuthology.orchestra.run.trial197.stdout: unzip x86_64 6.0-59.el9 baseos 182 k 2026-02-26T03:00:12.146 INFO:teuthology.orchestra.run.trial197.stdout: xmlsec1 x86_64 1.2.29-13.el9 appstream 189 k 2026-02-26T03:00:12.147 INFO:teuthology.orchestra.run.trial197.stdout: xmlsec1-openssl x86_64 1.2.29-13.el9 appstream 90 k 2026-02-26T03:00:12.147 INFO:teuthology.orchestra.run.trial197.stdout: zip x86_64 3.0-35.el9 baseos 266 k 2026-02-26T03:00:12.147 INFO:teuthology.orchestra.run.trial197.stdout:Installing weak dependencies: 2026-02-26T03:00:12.147 INFO:teuthology.orchestra.run.trial197.stdout: ceph-mgr-cephadm noarch 2:19.2.3-687.g8d724284.el9 ceph-noarch 150 k 2026-02-26T03:00:12.147 INFO:teuthology.orchestra.run.trial197.stdout: ceph-mgr-dashboard noarch 2:19.2.3-687.g8d724284.el9 ceph-noarch 3.8 M 2026-02-26T03:00:12.147 INFO:teuthology.orchestra.run.trial197.stdout: ceph-mgr-diskprediction-local noarch 2:19.2.3-687.g8d724284.el9 ceph-noarch 7.4 M 2026-02-26T03:00:12.147 INFO:teuthology.orchestra.run.trial197.stdout: ceph-mgr-k8sevents noarch 2:19.2.3-687.g8d724284.el9 ceph-noarch 22 k 2026-02-26T03:00:12.147 INFO:teuthology.orchestra.run.trial197.stdout: ceph-mgr-rook noarch 2:19.2.3-687.g8d724284.el9 ceph-noarch 49 k 2026-02-26T03:00:12.147 INFO:teuthology.orchestra.run.trial197.stdout: ceph-volume noarch 2:19.2.3-687.g8d724284.el9 ceph-noarch 299 k 2026-02-26T03:00:12.147 INFO:teuthology.orchestra.run.trial197.stdout: python3-influxdb noarch 5.3.1-1.el9 epel 139 k 2026-02-26T03:00:12.147 INFO:teuthology.orchestra.run.trial197.stdout: python3-saml noarch 1.16.0-1.el9 epel 125 k 2026-02-26T03:00:12.148 INFO:teuthology.orchestra.run.trial197.stdout: 2026-02-26T03:00:12.148 INFO:teuthology.orchestra.run.trial197.stdout:Transaction Summary 2026-02-26T03:00:12.148 INFO:teuthology.orchestra.run.trial197.stdout:====================================================================================== 2026-02-26T03:00:12.148 INFO:teuthology.orchestra.run.trial197.stdout:Install 99 Packages 2026-02-26T03:00:12.148 INFO:teuthology.orchestra.run.trial197.stdout: 2026-02-26T03:00:12.149 INFO:teuthology.orchestra.run.trial197.stdout:Total download size: 95 M 2026-02-26T03:00:12.149 INFO:teuthology.orchestra.run.trial197.stdout:Installed size: 520 M 2026-02-26T03:00:12.149 INFO:teuthology.orchestra.run.trial197.stdout:Downloading Packages: 2026-02-26T03:00:12.754 INFO:teuthology.orchestra.run.trial197.stdout:(1/99): ceph-19.2.3-687.g8d724284.el9.x86_64.rp 106 kB/s | 6.5 kB 00:00 2026-02-26T03:00:12.839 INFO:teuthology.orchestra.run.trial197.stdout:(2/99): ceph-mds-19.2.3-687.g8d724284.el9.x86_6 17 MB/s | 2.4 MB 00:00 2026-02-26T03:00:12.857 INFO:teuthology.orchestra.run.trial197.stdout:(3/99): ceph-mgr-19.2.3-687.g8d724284.el9.x86_6 6.5 MB/s | 1.1 MB 00:00 2026-02-26T03:00:12.902 INFO:teuthology.orchestra.run.trial197.stdout:(4/99): ceph-grafana-dashboards-19.2.3-687.g8d7 694 kB/s | 31 kB 00:00 2026-02-26T03:00:12.954 INFO:teuthology.orchestra.run.trial197.stdout:(5/99): ceph-mgr-cephadm-19.2.3-687.g8d724284.e 2.8 MB/s | 150 kB 00:00 2026-02-26T03:00:13.216 INFO:teuthology.orchestra.run.trial197.stdout:(6/99): ceph-mgr-dashboard-19.2.3-687.g8d724284 15 MB/s | 3.8 MB 00:00 2026-02-26T03:00:13.261 INFO:teuthology.orchestra.run.trial197.stdout:(7/99): ceph-mon-19.2.3-687.g8d724284.el9.x86_6 9.4 MB/s | 4.7 MB 00:00 2026-02-26T03:00:13.303 INFO:teuthology.orchestra.run.trial197.stdout:(8/99): ceph-mgr-k8sevents-19.2.3-687.g8d724284 516 kB/s | 22 kB 00:00 2026-02-26T03:00:13.336 INFO:teuthology.orchestra.run.trial197.stdout:(9/99): ceph-osd-19.2.3-687.g8d724284.el9.x86_6 34 MB/s | 17 MB 00:00 2026-02-26T03:00:13.337 INFO:teuthology.orchestra.run.trial197.stdout:(10/99): ceph-mgr-modules-core-19.2.3-687.g8d72 7.3 MB/s | 253 kB 00:00 2026-02-26T03:00:13.353 INFO:teuthology.orchestra.run.trial197.stdout:(11/99): ceph-mgr-rook-19.2.3-687.g8d724284.el9 2.8 MB/s | 49 kB 00:00 2026-02-26T03:00:13.354 INFO:teuthology.orchestra.run.trial197.stdout:(12/99): ceph-prometheus-alerts-19.2.3-687.g8d7 1.0 MB/s | 17 kB 00:00 2026-02-26T03:00:13.366 INFO:teuthology.orchestra.run.trial197.stdout:(13/99): ceph-volume-19.2.3-687.g8d724284.el9.n 23 MB/s | 299 kB 00:00 2026-02-26T03:00:13.397 INFO:teuthology.orchestra.run.trial197.stdout:(14/99): cephadm-19.2.3-687.g8d724284.el9.noarc 18 MB/s | 769 kB 00:00 2026-02-26T03:00:13.550 INFO:teuthology.orchestra.run.trial197.stdout:(15/99): ceph-mgr-diskprediction-local-19.2.3-6 22 MB/s | 7.4 MB 00:00 2026-02-26T03:00:13.992 INFO:teuthology.orchestra.run.trial197.stdout:(16/99): ledmon-libs-1.1.0-3.el9.x86_64.rpm 68 kB/s | 40 kB 00:00 2026-02-26T03:00:14.104 INFO:teuthology.orchestra.run.trial197.stdout:(17/99): libconfig-1.7.2-9.el9.x86_64.rpm 130 kB/s | 72 kB 00:00 2026-02-26T03:00:14.358 INFO:teuthology.orchestra.run.trial197.stdout:(18/99): cryptsetup-2.8.1-3.el9.x86_64.rpm 354 kB/s | 351 kB 00:00 2026-02-26T03:00:14.417 INFO:teuthology.orchestra.run.trial197.stdout:(19/99): libquadmath-11.5.0-14.el9.x86_64.rpm 588 kB/s | 184 kB 00:00 2026-02-26T03:00:14.759 INFO:teuthology.orchestra.run.trial197.stdout:(20/99): python3-cffi-1.14.5-5.el9.x86_64.rpm 632 kB/s | 253 kB 00:00 2026-02-26T03:00:15.002 INFO:teuthology.orchestra.run.trial197.stdout:(21/99): python3-ply-3.11-14.el9.noarch.rpm 439 kB/s | 106 kB 00:00 2026-02-26T03:00:15.081 INFO:teuthology.orchestra.run.trial197.stdout:(22/99): libgfortran-11.5.0-14.el9.x86_64.rpm 730 kB/s | 794 kB 00:01 2026-02-26T03:00:15.241 INFO:teuthology.orchestra.run.trial197.stdout:(23/99): python3-pycparser-2.20-6.el9.noarch.rp 563 kB/s | 135 kB 00:00 2026-02-26T03:00:15.394 INFO:teuthology.orchestra.run.trial197.stdout:(24/99): python3-pyparsing-2.4.7-9.el9.noarch.r 480 kB/s | 150 kB 00:00 2026-02-26T03:00:15.482 INFO:teuthology.orchestra.run.trial197.stdout:(25/99): python3-requests-2.25.1-10.el9.noarch. 526 kB/s | 126 kB 00:00 2026-02-26T03:00:15.787 INFO:teuthology.orchestra.run.trial197.stdout:(26/99): python3-urllib3-1.26.5-7.el9.noarch.rp 555 kB/s | 218 kB 00:00 2026-02-26T03:00:15.798 INFO:teuthology.orchestra.run.trial197.stdout:(27/99): unzip-6.0-59.el9.x86_64.rpm 575 kB/s | 182 kB 00:00 2026-02-26T03:00:15.982 INFO:teuthology.orchestra.run.trial197.stdout:(28/99): python3-cryptography-36.0.1-5.el9.x86_ 815 kB/s | 1.2 MB 00:01 2026-02-26T03:00:16.182 INFO:teuthology.orchestra.run.trial197.stdout:(29/99): zip-3.0-35.el9.x86_64.rpm 673 kB/s | 266 kB 00:00 2026-02-26T03:00:16.369 INFO:teuthology.orchestra.run.trial197.stdout:(30/99): flexiblas-3.0.4-9.el9.x86_64.rpm 52 kB/s | 30 kB 00:00 2026-02-26T03:00:16.689 INFO:teuthology.orchestra.run.trial197.stdout:(31/99): flexiblas-openblas-openmp-3.0.4-9.el9. 29 kB/s | 15 kB 00:00 2026-02-26T03:00:17.071 INFO:teuthology.orchestra.run.trial197.stdout:(32/99): flexiblas-netlib-3.0.4-9.el9.x86_64.rp 2.7 MB/s | 3.0 MB 00:01 2026-02-26T03:00:17.171 INFO:teuthology.orchestra.run.trial197.stdout:(33/99): libstoragemgmt-1.10.1-1.el9.x86_64.rpm 307 kB/s | 246 kB 00:00 2026-02-26T03:00:17.483 INFO:teuthology.orchestra.run.trial197.stdout:(34/99): lua-5.4.4-4.el9.x86_64.rpm 237 kB/s | 188 kB 00:00 2026-02-26T03:00:17.678 INFO:teuthology.orchestra.run.trial197.stdout:(35/99): openblas-0.3.29-1.el9.x86_64.rpm 69 kB/s | 42 kB 00:00 2026-02-26T03:00:18.355 INFO:teuthology.orchestra.run.trial197.stdout:(36/99): openblas-openmp-0.3.29-1.el9.x86_64.rp 4.5 MB/s | 5.3 MB 00:01 2026-02-26T03:00:18.441 INFO:teuthology.orchestra.run.trial197.stdout:(37/99): protobuf-3.14.0-17.el9.x86_64.rpm 1.0 MB/s | 1.0 MB 00:00 2026-02-26T03:00:18.901 INFO:teuthology.orchestra.run.trial197.stdout:(38/99): python3-babel-2.9.1-2.el9.noarch.rpm 4.9 MB/s | 6.0 MB 00:01 2026-02-26T03:00:19.155 INFO:teuthology.orchestra.run.trial197.stdout:(39/99): python3-devel-3.9.25-3.el9.x86_64.rpm 306 kB/s | 244 kB 00:00 2026-02-26T03:00:19.232 INFO:teuthology.orchestra.run.trial197.stdout:(40/99): python3-jinja2-2.11.3-8.el9.noarch.rpm 314 kB/s | 249 kB 00:00 2026-02-26T03:00:19.674 INFO:teuthology.orchestra.run.trial197.stdout:(41/99): python3-libstoragemgmt-1.10.1-1.el9.x8 229 kB/s | 177 kB 00:00 2026-02-26T03:00:20.011 INFO:teuthology.orchestra.run.trial197.stdout:(42/99): python3-mako-1.1.4-6.el9.noarch.rpm 221 kB/s | 172 kB 00:00 2026-02-26T03:00:20.154 INFO:teuthology.orchestra.run.trial197.stdout:(43/99): python3-lxml-4.6.5-3.el9.x86_64.rpm 1.2 MB/s | 1.2 MB 00:00 2026-02-26T03:00:20.270 INFO:teuthology.orchestra.run.trial197.stdout:(44/99): python3-markupsafe-1.1.1-12.el9.x86_64 58 kB/s | 35 kB 00:00 2026-02-26T03:00:20.954 INFO:teuthology.orchestra.run.trial197.stdout:(45/99): python3-packaging-20.9-5.el9.noarch.rp 113 kB/s | 77 kB 00:00 2026-02-26T03:00:21.048 INFO:teuthology.orchestra.run.trial197.stdout:(46/99): python3-numpy-f2py-1.23.5-2.el9.x86_64 495 kB/s | 442 kB 00:00 2026-02-26T03:00:21.239 INFO:teuthology.orchestra.run.trial197.stdout:(47/99): python3-numpy-1.23.5-2.el9.x86_64.rpm 5.0 MB/s | 6.1 MB 00:01 2026-02-26T03:00:21.753 INFO:teuthology.orchestra.run.trial197.stdout:(48/99): python3-protobuf-3.14.0-17.el9.noarch. 335 kB/s | 267 kB 00:00 2026-02-26T03:00:21.816 INFO:teuthology.orchestra.run.trial197.stdout:(49/99): python3-pyasn1-0.4.8-7.el9.noarch.rpm 205 kB/s | 157 kB 00:00 2026-02-26T03:00:22.073 INFO:teuthology.orchestra.run.trial197.stdout:(50/99): python3-pyasn1-modules-0.4.8-7.el9.noa 332 kB/s | 277 kB 00:00 2026-02-26T03:00:22.405 INFO:teuthology.orchestra.run.trial197.stdout:(51/99): python3-requests-oauthlib-1.3.0-12.el9 82 kB/s | 54 kB 00:00 2026-02-26T03:00:22.689 INFO:teuthology.orchestra.run.trial197.stdout:(52/99): python3-toml-0.10.2-6.el9.noarch.rpm 68 kB/s | 42 kB 00:00 2026-02-26T03:00:23.159 INFO:teuthology.orchestra.run.trial197.stdout:(53/99): xmlsec1-1.2.29-13.el9.x86_64.rpm 250 kB/s | 189 kB 00:00 2026-02-26T03:00:23.189 INFO:teuthology.orchestra.run.trial197.stdout:(54/99): lua-devel-5.4.4-4.el9.x86_64.rpm 752 kB/s | 22 kB 00:00 2026-02-26T03:00:23.233 INFO:teuthology.orchestra.run.trial197.stdout:(55/99): protobuf-compiler-3.14.0-17.el9.x86_64 19 MB/s | 862 kB 00:00 2026-02-26T03:00:23.379 INFO:teuthology.orchestra.run.trial197.stdout:(56/99): python3-scipy-1.9.3-2.el9.x86_64.rpm 12 MB/s | 19 MB 00:01 2026-02-26T03:00:23.384 INFO:teuthology.orchestra.run.trial197.stdout:(57/99): xmlsec1-openssl-1.2.29-13.el9.x86_64.r 130 kB/s | 90 kB 00:00 2026-02-26T03:00:23.777 INFO:teuthology.orchestra.run.trial197.stdout:(58/99): grpc-data-1.46.7-10.el9.noarch.rpm 49 kB/s | 19 kB 00:00 2026-02-26T03:00:23.935 INFO:teuthology.orchestra.run.trial197.stdout:(59/99): luarocks-3.9.2-5.el9.noarch.rpm 275 kB/s | 151 kB 00:00 2026-02-26T03:00:24.014 INFO:teuthology.orchestra.run.trial197.stdout:(60/99): python3-autocommand-2.2.2-8.el9.noarch 373 kB/s | 29 kB 00:00 2026-02-26T03:00:24.093 INFO:teuthology.orchestra.run.trial197.stdout:(61/99): python3-backports-tarfile-1.2.0-1.el9. 764 kB/s | 60 kB 00:00 2026-02-26T03:00:24.172 INFO:teuthology.orchestra.run.trial197.stdout:(62/99): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 553 kB/s | 43 kB 00:00 2026-02-26T03:00:24.250 INFO:teuthology.orchestra.run.trial197.stdout:(63/99): python3-cachetools-4.2.4-1.el9.noarch. 408 kB/s | 32 kB 00:00 2026-02-26T03:00:24.251 INFO:teuthology.orchestra.run.trial197.stdout:(64/99): abseil-cpp-20211102.0-4.el9.x86_64.rpm 541 kB/s | 551 kB 00:01 2026-02-26T03:00:24.329 INFO:teuthology.orchestra.run.trial197.stdout:(65/99): python3-certifi-2023.05.07-4.el9.noarc 179 kB/s | 14 kB 00:00 2026-02-26T03:00:24.483 INFO:teuthology.orchestra.run.trial197.stdout:(66/99): python3-asyncssh-2.13.2-5.el9.noarch.r 776 kB/s | 548 kB 00:00 2026-02-26T03:00:24.485 INFO:teuthology.orchestra.run.trial197.stdout:(67/99): python3-cheroot-10.0.1-4.el9.noarch.rp 741 kB/s | 173 kB 00:00 2026-02-26T03:00:24.797 INFO:teuthology.orchestra.run.trial197.stdout:(68/99): python3-cherrypy-18.6.1-2.el9.noarch.r 766 kB/s | 358 kB 00:00 2026-02-26T03:00:24.798 INFO:teuthology.orchestra.run.trial197.stdout:(69/99): python3-google-auth-2.45.0-1.el9.noarc 808 kB/s | 254 kB 00:00 2026-02-26T03:00:24.956 INFO:teuthology.orchestra.run.trial197.stdout:(70/99): python3-influxdb-5.3.1-1.el9.noarch.rp 880 kB/s | 139 kB 00:00 2026-02-26T03:00:25.037 INFO:teuthology.orchestra.run.trial197.stdout:(71/99): python3-grpcio-tools-1.46.7-10.el9.x86 601 kB/s | 144 kB 00:00 2026-02-26T03:00:25.038 INFO:teuthology.orchestra.run.trial197.stdout:(72/99): python3-isodate-0.6.1-3.el9.noarch.rpm 682 kB/s | 56 kB 00:00 2026-02-26T03:00:25.117 INFO:teuthology.orchestra.run.trial197.stdout:(73/99): python3-jaraco-8.2.1-3.el9.noarch.rpm 135 kB/s | 11 kB 00:00 2026-02-26T03:00:25.117 INFO:teuthology.orchestra.run.trial197.stdout:(74/99): python3-jaraco-classes-3.2.1-5.el9.noa 223 kB/s | 18 kB 00:00 2026-02-26T03:00:25.196 INFO:teuthology.orchestra.run.trial197.stdout:(75/99): python3-jaraco-context-6.0.1-3.el9.noa 248 kB/s | 20 kB 00:00 2026-02-26T03:00:25.213 INFO:teuthology.orchestra.run.trial197.stdout:(76/99): python3-jaraco-collections-3.0.0-8.el9 239 kB/s | 23 kB 00:00 2026-02-26T03:00:25.276 INFO:teuthology.orchestra.run.trial197.stdout:(77/99): python3-jaraco-functools-3.5.0-2.el9.n 244 kB/s | 19 kB 00:00 2026-02-26T03:00:25.294 INFO:teuthology.orchestra.run.trial197.stdout:(78/99): python3-jaraco-text-4.0.0-2.el9.noarch 326 kB/s | 26 kB 00:00 2026-02-26T03:00:25.375 INFO:teuthology.orchestra.run.trial197.stdout:(79/99): python3-logutils-0.3.5-21.el9.noarch.r 573 kB/s | 46 kB 00:00 2026-02-26T03:00:25.542 INFO:teuthology.orchestra.run.trial197.stdout:(80/99): python3-more-itertools-8.12.0-2.el9.no 475 kB/s | 79 kB 00:00 2026-02-26T03:00:25.704 INFO:teuthology.orchestra.run.trial197.stdout:(81/99): python3-msgpack-1.0.3-2.el9.x86_64.rpm 532 kB/s | 86 kB 00:00 2026-02-26T03:00:25.783 INFO:teuthology.orchestra.run.trial197.stdout:(82/99): python3-natsort-7.1.1-5.el9.noarch.rpm 732 kB/s | 58 kB 00:00 2026-02-26T03:00:26.095 INFO:teuthology.orchestra.run.trial197.stdout:(83/99): python3-pecan-1.4.2-3.el9.noarch.rpm 873 kB/s | 272 kB 00:00 2026-02-26T03:00:26.174 INFO:teuthology.orchestra.run.trial197.stdout:(84/99): python3-portend-3.1.0-2.el9.noarch.rpm 208 kB/s | 16 kB 00:00 2026-02-26T03:00:26.331 INFO:teuthology.orchestra.run.trial197.stdout:(85/99): python3-pyOpenSSL-21.0.0-1.el9.noarch. 574 kB/s | 90 kB 00:00 2026-02-26T03:00:26.410 INFO:teuthology.orchestra.run.trial197.stdout:(86/99): python3-repoze-lru-0.7-16.el9.noarch.r 389 kB/s | 31 kB 00:00 2026-02-26T03:00:26.461 INFO:teuthology.orchestra.run.trial197.stdout:(87/99): python3-kubernetes-26.1.0-3.el9.noarch 882 kB/s | 1.0 MB 00:01 2026-02-26T03:00:26.541 INFO:teuthology.orchestra.run.trial197.stdout:(88/99): python3-rsa-4.9-2.el9.noarch.rpm 737 kB/s | 59 kB 00:00 2026-02-26T03:00:26.644 INFO:teuthology.orchestra.run.trial197.stdout:(89/99): python3-routes-2.5.1-5.el9.noarch.rpm 803 kB/s | 188 kB 00:00 2026-02-26T03:00:26.699 INFO:teuthology.orchestra.run.trial197.stdout:(90/99): python3-saml-1.16.0-1.el9.noarch.rpm 791 kB/s | 125 kB 00:00 2026-02-26T03:00:26.723 INFO:teuthology.orchestra.run.trial197.stdout:(91/99): python3-tempora-5.0.0-2.el9.noarch.rpm 455 kB/s | 36 kB 00:00 2026-02-26T03:00:26.827 INFO:teuthology.orchestra.run.trial197.stdout:(92/99): python3-grpcio-1.46.7-10.el9.x86_64.rp 891 kB/s | 2.0 MB 00:02 2026-02-26T03:00:26.857 INFO:teuthology.orchestra.run.trial197.stdout:(93/99): python3-typing-extensions-4.15.0-1.el9 546 kB/s | 86 kB 00:00 2026-02-26T03:00:26.985 INFO:teuthology.orchestra.run.trial197.stdout:(94/99): python3-websocket-client-1.2.3-2.el9.n 566 kB/s | 90 kB 00:00 2026-02-26T03:00:27.035 INFO:teuthology.orchestra.run.trial197.stdout:(95/99): python3-webob-1.8.8-2.el9.noarch.rpm 739 kB/s | 230 kB 00:00 2026-02-26T03:00:27.064 INFO:teuthology.orchestra.run.trial197.stdout:(96/99): python3-xmlsec-1.3.13-1.el9.x86_64.rpm 614 kB/s | 48 kB 00:00 2026-02-26T03:00:27.114 INFO:teuthology.orchestra.run.trial197.stdout:(97/99): python3-xmltodict-0.12.0-15.el9.noarch 282 kB/s | 22 kB 00:00 2026-02-26T03:00:27.144 INFO:teuthology.orchestra.run.trial197.stdout:(98/99): python3-zc-lockfile-2.0-10.el9.noarch. 251 kB/s | 20 kB 00:00 2026-02-26T03:00:27.408 INFO:teuthology.orchestra.run.trial197.stdout:(99/99): python3-werkzeug-2.0.3-3.el9.1.noarch. 776 kB/s | 427 kB 00:00 2026-02-26T03:00:27.414 INFO:teuthology.orchestra.run.trial197.stdout:-------------------------------------------------------------------------------- 2026-02-26T03:00:27.414 INFO:teuthology.orchestra.run.trial197.stdout:Total 6.2 MB/s | 95 MB 00:15 2026-02-26T03:00:27.851 INFO:teuthology.orchestra.run.trial197.stdout:Running transaction check 2026-02-26T03:00:27.889 INFO:teuthology.orchestra.run.trial197.stdout:Transaction check succeeded. 2026-02-26T03:00:27.889 INFO:teuthology.orchestra.run.trial197.stdout:Running transaction test 2026-02-26T03:00:28.312 INFO:teuthology.orchestra.run.trial197.stdout:Transaction test succeeded. 2026-02-26T03:00:28.312 INFO:teuthology.orchestra.run.trial197.stdout:Running transaction 2026-02-26T03:00:28.772 INFO:teuthology.orchestra.run.trial197.stdout: Preparing : 1/1 2026-02-26T03:00:28.774 INFO:teuthology.orchestra.run.trial197.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/99 2026-02-26T03:00:28.805 INFO:teuthology.orchestra.run.trial197.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/99 2026-02-26T03:00:28.810 INFO:teuthology.orchestra.run.trial197.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 3/99 2026-02-26T03:00:28.832 INFO:teuthology.orchestra.run.trial197.stdout: Installing : xmlsec1-1.2.29-13.el9.x86_64 4/99 2026-02-26T03:00:28.838 INFO:teuthology.orchestra.run.trial197.stdout: Installing : python3-pyasn1-0.4.8-7.el9.noarch 5/99 2026-02-26T03:00:28.872 INFO:teuthology.orchestra.run.trial197.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 6/99 2026-02-26T03:00:28.882 INFO:teuthology.orchestra.run.trial197.stdout: Installing : python3-lxml-4.6.5-3.el9.x86_64 7/99 2026-02-26T03:00:28.887 INFO:teuthology.orchestra.run.trial197.stdout: Installing : protobuf-3.14.0-17.el9.x86_64 8/99 2026-02-26T03:00:28.889 INFO:teuthology.orchestra.run.trial197.stdout: Installing : lua-5.4.4-4.el9.x86_64 9/99 2026-02-26T03:00:28.894 INFO:teuthology.orchestra.run.trial197.stdout: Installing : flexiblas-3.0.4-9.el9.x86_64 10/99 2026-02-26T03:00:28.917 INFO:teuthology.orchestra.run.trial197.stdout: Installing : unzip-6.0-59.el9.x86_64 11/99 2026-02-26T03:00:28.930 INFO:teuthology.orchestra.run.trial197.stdout: Installing : python3-urllib3-1.26.5-7.el9.noarch 12/99 2026-02-26T03:00:28.933 INFO:teuthology.orchestra.run.trial197.stdout: Installing : python3-requests-2.25.1-10.el9.noarch 13/99 2026-02-26T03:00:28.939 INFO:teuthology.orchestra.run.trial197.stdout: Installing : libquadmath-11.5.0-14.el9.x86_64 14/99 2026-02-26T03:00:28.941 INFO:teuthology.orchestra.run.trial197.stdout: Installing : libgfortran-11.5.0-14.el9.x86_64 15/99 2026-02-26T03:00:28.952 INFO:teuthology.orchestra.run.trial197.stdout: Installing : ledmon-libs-1.1.0-3.el9.x86_64 16/99 2026-02-26T03:00:28.958 INFO:teuthology.orchestra.run.trial197.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 17/99 2026-02-26T03:00:28.982 INFO:teuthology.orchestra.run.trial197.stdout: Installing : zip-3.0-35.el9.x86_64 18/99 2026-02-26T03:00:28.985 INFO:teuthology.orchestra.run.trial197.stdout: Installing : luarocks-3.9.2-5.el9.noarch 19/99 2026-02-26T03:00:28.993 INFO:teuthology.orchestra.run.trial197.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 20/99 2026-02-26T03:00:29.017 INFO:teuthology.orchestra.run.trial197.stdout: Installing : protobuf-compiler-3.14.0-17.el9.x86_64 21/99 2026-02-26T03:00:29.068 INFO:teuthology.orchestra.run.trial197.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 22/99 2026-02-26T03:00:29.081 INFO:teuthology.orchestra.run.trial197.stdout: Installing : python3-pyasn1-modules-0.4.8-7.el9.noarch 23/99 2026-02-26T03:00:29.084 INFO:teuthology.orchestra.run.trial197.stdout: Installing : python3-rsa-4.9-2.el9.noarch 24/99 2026-02-26T03:00:29.088 INFO:teuthology.orchestra.run.trial197.stdout: Installing : xmlsec1-openssl-1.2.29-13.el9.x86_64 25/99 2026-02-26T03:00:29.093 INFO:teuthology.orchestra.run.trial197.stdout: Installing : python3-xmlsec-1.3.13-1.el9.x86_64 26/99 2026-02-26T03:00:29.098 INFO:teuthology.orchestra.run.trial197.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 27/99 2026-02-26T03:00:29.101 INFO:teuthology.orchestra.run.trial197.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 28/99 2026-02-26T03:00:29.116 INFO:teuthology.orchestra.run.trial197.stdout: Installing : python3-xmltodict-0.12.0-15.el9.noarch 29/99 2026-02-26T03:00:29.137 INFO:teuthology.orchestra.run.trial197.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 30/99 2026-02-26T03:00:29.141 INFO:teuthology.orchestra.run.trial197.stdout: Installing : python3-webob-1.8.8-2.el9.noarch 31/99 2026-02-26T03:00:29.146 INFO:teuthology.orchestra.run.trial197.stdout: Installing : python3-typing-extensions-4.15.0-1.el9.noarch 32/99 2026-02-26T03:00:29.158 INFO:teuthology.orchestra.run.trial197.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 33/99 2026-02-26T03:00:29.167 INFO:teuthology.orchestra.run.trial197.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 34/99 2026-02-26T03:00:29.173 INFO:teuthology.orchestra.run.trial197.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 35/99 2026-02-26T03:00:29.193 INFO:teuthology.orchestra.run.trial197.stdout: Installing : python3-msgpack-1.0.3-2.el9.x86_64 36/99 2026-02-26T03:00:29.202 INFO:teuthology.orchestra.run.trial197.stdout: Installing : python3-influxdb-5.3.1-1.el9.noarch 37/99 2026-02-26T03:00:29.256 INFO:teuthology.orchestra.run.trial197.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 38/99 2026-02-26T03:00:29.269 INFO:teuthology.orchestra.run.trial197.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 39/99 2026-02-26T03:00:29.284 INFO:teuthology.orchestra.run.trial197.stdout: Installing : python3-isodate-0.6.1-3.el9.noarch 40/99 2026-02-26T03:00:29.289 INFO:teuthology.orchestra.run.trial197.stdout: Installing : python3-saml-1.16.0-1.el9.noarch 41/99 2026-02-26T03:00:29.297 INFO:teuthology.orchestra.run.trial197.stdout: Installing : python3-certifi-2023.05.07-4.el9.noarch 42/99 2026-02-26T03:00:29.337 INFO:teuthology.orchestra.run.trial197.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 43/99 2026-02-26T03:00:29.663 INFO:teuthology.orchestra.run.trial197.stdout: Installing : python3-google-auth-1:2.45.0-1.el9.noarch 44/99 2026-02-26T03:00:29.673 INFO:teuthology.orchestra.run.trial197.stdout: Installing : python3-kubernetes-1:26.1.0-3.el9.noarch 45/99 2026-02-26T03:00:29.677 INFO:teuthology.orchestra.run.trial197.stdout: Installing : python3-backports-tarfile-1.2.0-1.el9.noarch 46/99 2026-02-26T03:00:29.682 INFO:teuthology.orchestra.run.trial197.stdout: Installing : python3-jaraco-context-6.0.1-3.el9.noarch 47/99 2026-02-26T03:00:29.684 INFO:teuthology.orchestra.run.trial197.stdout: Installing : python3-autocommand-2.2.2-8.el9.noarch 48/99 2026-02-26T03:00:29.712 INFO:teuthology.orchestra.run.trial197.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 49/99 2026-02-26T03:00:29.756 INFO:teuthology.orchestra.run.trial197.stdout: Installing : abseil-cpp-20211102.0-4.el9.x86_64 50/99 2026-02-26T03:00:29.762 INFO:teuthology.orchestra.run.trial197.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 51/99 2026-02-26T03:00:29.766 INFO:teuthology.orchestra.run.trial197.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 52/99 2026-02-26T03:00:29.772 INFO:teuthology.orchestra.run.trial197.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 53/99 2026-02-26T03:00:29.775 INFO:teuthology.orchestra.run.trial197.stdout: Installing : python3-jaraco-text-4.0.0-2.el9.noarch 54/99 2026-02-26T03:00:29.782 INFO:teuthology.orchestra.run.trial197.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 55/99 2026-02-26T03:00:29.786 INFO:teuthology.orchestra.run.trial197.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 56/99 2026-02-26T03:00:29.814 INFO:teuthology.orchestra.run.trial197.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 57/99 2026-02-26T03:00:29.826 INFO:teuthology.orchestra.run.trial197.stdout: Installing : python3-protobuf-3.14.0-17.el9.noarch 58/99 2026-02-26T03:00:29.860 INFO:teuthology.orchestra.run.trial197.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 59/99 2026-02-26T03:00:30.086 INFO:teuthology.orchestra.run.trial197.stdout: Installing : python3-devel-3.9.25-3.el9.x86_64 60/99 2026-02-26T03:00:30.110 INFO:teuthology.orchestra.run.trial197.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 61/99 2026-02-26T03:00:30.113 INFO:teuthology.orchestra.run.trial197.stdout: Installing : python3-jinja2-2.11.3-8.el9.noarch 62/99 2026-02-26T03:00:30.173 INFO:teuthology.orchestra.run.trial197.stdout: Installing : openblas-0.3.29-1.el9.x86_64 63/99 2026-02-26T03:00:30.175 INFO:teuthology.orchestra.run.trial197.stdout: Installing : openblas-openmp-0.3.29-1.el9.x86_64 64/99 2026-02-26T03:00:30.196 INFO:teuthology.orchestra.run.trial197.stdout: Installing : flexiblas-openblas-openmp-3.0.4-9.el9.x86_64 65/99 2026-02-26T03:00:30.524 INFO:teuthology.orchestra.run.trial197.stdout: Installing : flexiblas-netlib-3.0.4-9.el9.x86_64 66/99 2026-02-26T03:00:30.598 INFO:teuthology.orchestra.run.trial197.stdout: Installing : python3-numpy-1:1.23.5-2.el9.x86_64 67/99 2026-02-26T03:00:31.272 INFO:teuthology.orchestra.run.trial197.stdout: Installing : python3-numpy-f2py-1:1.23.5-2.el9.x86_64 68/99 2026-02-26T03:00:31.289 INFO:teuthology.orchestra.run.trial197.stdout: Installing : python3-scipy-1.9.3-2.el9.x86_64 69/99 2026-02-26T03:00:31.299 INFO:teuthology.orchestra.run.trial197.stdout: Installing : python3-pyparsing-2.4.7-9.el9.noarch 70/99 2026-02-26T03:00:31.306 INFO:teuthology.orchestra.run.trial197.stdout: Installing : python3-packaging-20.9-5.el9.noarch 71/99 2026-02-26T03:00:31.319 INFO:teuthology.orchestra.run.trial197.stdout: Installing : python3-ply-3.11-14.el9.noarch 72/99 2026-02-26T03:00:31.335 INFO:teuthology.orchestra.run.trial197.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 73/99 2026-02-26T03:00:31.410 INFO:teuthology.orchestra.run.trial197.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 74/99 2026-02-26T03:00:31.419 INFO:teuthology.orchestra.run.trial197.stdout: Installing : python3-cryptography-36.0.1-5.el9.x86_64 75/99 2026-02-26T03:00:31.442 INFO:teuthology.orchestra.run.trial197.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 76/99 2026-02-26T03:00:31.472 INFO:teuthology.orchestra.run.trial197.stdout: Installing : python3-cheroot-10.0.1-4.el9.noarch 77/99 2026-02-26T03:00:31.523 INFO:teuthology.orchestra.run.trial197.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 78/99 2026-02-26T03:00:31.529 INFO:teuthology.orchestra.run.trial197.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 79/99 2026-02-26T03:00:31.533 INFO:teuthology.orchestra.run.trial197.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 80/99 2026-02-26T03:00:31.534 INFO:teuthology.orchestra.run.trial197.stdout: Installing : libconfig-1.7.2-9.el9.x86_64 81/99 2026-02-26T03:00:31.552 INFO:teuthology.orchestra.run.trial197.stdout: Running scriptlet: libstoragemgmt-1.10.1-1.el9.x86_64 82/99 2026-02-26T03:00:31.552 INFO:teuthology.orchestra.run.trial197.stdout:Creating group 'libstoragemgmt' with GID 991. 2026-02-26T03:00:31.552 INFO:teuthology.orchestra.run.trial197.stdout:Creating user 'libstoragemgmt' (daemon account for libstoragemgmt) with UID 991 and GID 991. 2026-02-26T03:00:31.552 INFO:teuthology.orchestra.run.trial197.stdout: 2026-02-26T03:00:31.560 INFO:teuthology.orchestra.run.trial197.stdout: Installing : libstoragemgmt-1.10.1-1.el9.x86_64 82/99 2026-02-26T03:00:31.586 INFO:teuthology.orchestra.run.trial197.stdout: Running scriptlet: libstoragemgmt-1.10.1-1.el9.x86_64 82/99 2026-02-26T03:00:31.586 INFO:teuthology.orchestra.run.trial197.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/libstoragemgmt.service → /usr/lib/systemd/system/libstoragemgmt.service. 2026-02-26T03:00:31.586 INFO:teuthology.orchestra.run.trial197.stdout: 2026-02-26T03:00:31.622 INFO:teuthology.orchestra.run.trial197.stdout: Installing : python3-libstoragemgmt-1.10.1-1.el9.x86_64 83/99 2026-02-26T03:00:31.686 INFO:teuthology.orchestra.run.trial197.stdout: Installing : cryptsetup-2.8.1-3.el9.x86_64 84/99 2026-02-26T03:00:31.687 INFO:teuthology.orchestra.run.trial197.stdout: Installing : ceph-volume-2:19.2.3-687.g8d724284.el9.noarch 85/99 2026-02-26T03:00:31.702 INFO:teuthology.orchestra.run.trial197.stdout: Running scriptlet: ceph-volume-2:19.2.3-687.g8d724284.el9.noarch 85/99 2026-02-26T03:00:31.702 INFO:teuthology.orchestra.run.trial197.stdout:Glob pattern passed to enable, but globs are not supported for this. 2026-02-26T03:00:31.702 INFO:teuthology.orchestra.run.trial197.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2026-02-26T03:00:31.702 INFO:teuthology.orchestra.run.trial197.stdout: 2026-02-26T03:00:32.412 INFO:teuthology.orchestra.run.trial197.stdout: Installing : ceph-osd-2:19.2.3-687.g8d724284.el9.x86_64 86/99 2026-02-26T03:00:32.436 INFO:teuthology.orchestra.run.trial197.stdout: Running scriptlet: ceph-osd-2:19.2.3-687.g8d724284.el9.x86_64 86/99 2026-02-26T03:00:32.436 INFO:teuthology.orchestra.run.trial197.stdout:Glob pattern passed to enable, but globs are not supported for this. 2026-02-26T03:00:32.437 INFO:teuthology.orchestra.run.trial197.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2026-02-26T03:00:32.437 INFO:teuthology.orchestra.run.trial197.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2026-02-26T03:00:32.437 INFO:teuthology.orchestra.run.trial197.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2026-02-26T03:00:32.437 INFO:teuthology.orchestra.run.trial197.stdout: 2026-02-26T03:00:32.490 INFO:teuthology.orchestra.run.trial197.stdout: Running scriptlet: cephadm-2:19.2.3-687.g8d724284.el9.noarch 87/99 2026-02-26T03:00:32.492 INFO:teuthology.orchestra.run.trial197.stdout: Installing : cephadm-2:19.2.3-687.g8d724284.el9.noarch 87/99 2026-02-26T03:00:32.497 INFO:teuthology.orchestra.run.trial197.stdout: Installing : ceph-prometheus-alerts-2:19.2.3-687.g8d724284.el9. 88/99 2026-02-26T03:00:32.517 INFO:teuthology.orchestra.run.trial197.stdout: Installing : ceph-grafana-dashboards-2:19.2.3-687.g8d724284.el9 89/99 2026-02-26T03:00:32.518 INFO:teuthology.orchestra.run.trial197.stdout: Installing : ceph-mgr-cephadm-2:19.2.3-687.g8d724284.el9.noarch 90/99 2026-02-26T03:00:33.008 INFO:teuthology.orchestra.run.trial197.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.2.3-687.g8d724284.el9.noarch 90/99 2026-02-26T03:00:33.011 INFO:teuthology.orchestra.run.trial197.stdout: Installing : ceph-mgr-dashboard-2:19.2.3-687.g8d724284.el9.noar 91/99 2026-02-26T03:00:33.468 INFO:teuthology.orchestra.run.trial197.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.2.3-687.g8d724284.el9.noar 91/99 2026-02-26T03:00:33.469 INFO:teuthology.orchestra.run.trial197.stdout: Installing : ceph-mgr-diskprediction-local-2:19.2.3-687.g8d7242 92/99 2026-02-26T03:00:33.481 INFO:teuthology.orchestra.run.trial197.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.2.3-687.g8d7242 92/99 2026-02-26T03:00:33.482 INFO:teuthology.orchestra.run.trial197.stdout: Installing : ceph-mgr-k8sevents-2:19.2.3-687.g8d724284.el9.noar 93/99 2026-02-26T03:00:33.538 INFO:teuthology.orchestra.run.trial197.stdout: Running scriptlet: ceph-mgr-k8sevents-2:19.2.3-687.g8d724284.el9.noar 93/99 2026-02-26T03:00:33.587 INFO:teuthology.orchestra.run.trial197.stdout: Installing : ceph-mgr-modules-core-2:19.2.3-687.g8d724284.el9.n 94/99 2026-02-26T03:00:33.588 INFO:teuthology.orchestra.run.trial197.stdout: Installing : ceph-mgr-2:19.2.3-687.g8d724284.el9.x86_64 95/99 2026-02-26T03:00:33.610 INFO:teuthology.orchestra.run.trial197.stdout: Running scriptlet: ceph-mgr-2:19.2.3-687.g8d724284.el9.x86_64 95/99 2026-02-26T03:00:33.610 INFO:teuthology.orchestra.run.trial197.stdout:Glob pattern passed to enable, but globs are not supported for this. 2026-02-26T03:00:33.610 INFO:teuthology.orchestra.run.trial197.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2026-02-26T03:00:33.611 INFO:teuthology.orchestra.run.trial197.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2026-02-26T03:00:33.611 INFO:teuthology.orchestra.run.trial197.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2026-02-26T03:00:33.611 INFO:teuthology.orchestra.run.trial197.stdout: 2026-02-26T03:00:33.622 INFO:teuthology.orchestra.run.trial197.stdout: Installing : ceph-mgr-rook-2:19.2.3-687.g8d724284.el9.noarch 96/99 2026-02-26T03:00:33.838 INFO:teuthology.orchestra.run.trial197.stdout: Running scriptlet: ceph-mgr-rook-2:19.2.3-687.g8d724284.el9.noarch 96/99 2026-02-26T03:00:33.839 INFO:teuthology.orchestra.run.trial197.stdout: Installing : ceph-mon-2:19.2.3-687.g8d724284.el9.x86_64 97/99 2026-02-26T03:00:33.861 INFO:teuthology.orchestra.run.trial197.stdout: Running scriptlet: ceph-mon-2:19.2.3-687.g8d724284.el9.x86_64 97/99 2026-02-26T03:00:33.861 INFO:teuthology.orchestra.run.trial197.stdout:Glob pattern passed to enable, but globs are not supported for this. 2026-02-26T03:00:33.861 INFO:teuthology.orchestra.run.trial197.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2026-02-26T03:00:33.861 INFO:teuthology.orchestra.run.trial197.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2026-02-26T03:00:33.861 INFO:teuthology.orchestra.run.trial197.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2026-02-26T03:00:33.861 INFO:teuthology.orchestra.run.trial197.stdout: 2026-02-26T03:00:33.963 INFO:teuthology.orchestra.run.trial197.stdout: Installing : ceph-mds-2:19.2.3-687.g8d724284.el9.x86_64 98/99 2026-02-26T03:00:33.984 INFO:teuthology.orchestra.run.trial197.stdout: Running scriptlet: ceph-mds-2:19.2.3-687.g8d724284.el9.x86_64 98/99 2026-02-26T03:00:33.984 INFO:teuthology.orchestra.run.trial197.stdout:Glob pattern passed to enable, but globs are not supported for this. 2026-02-26T03:00:33.984 INFO:teuthology.orchestra.run.trial197.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2026-02-26T03:00:33.984 INFO:teuthology.orchestra.run.trial197.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2026-02-26T03:00:33.984 INFO:teuthology.orchestra.run.trial197.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2026-02-26T03:00:33.984 INFO:teuthology.orchestra.run.trial197.stdout: 2026-02-26T03:00:34.003 INFO:teuthology.orchestra.run.trial197.stdout: Installing : ceph-2:19.2.3-687.g8d724284.el9.x86_64 99/99 2026-02-26T03:00:34.666 INFO:teuthology.orchestra.run.trial197.stdout: Running scriptlet: ceph-2:19.2.3-687.g8d724284.el9.x86_64 99/99 2026-02-26T03:00:34.666 INFO:teuthology.orchestra.run.trial197.stdout: Verifying : ceph-2:19.2.3-687.g8d724284.el9.x86_64 1/99 2026-02-26T03:00:34.666 INFO:teuthology.orchestra.run.trial197.stdout: Verifying : ceph-mds-2:19.2.3-687.g8d724284.el9.x86_64 2/99 2026-02-26T03:00:34.666 INFO:teuthology.orchestra.run.trial197.stdout: Verifying : ceph-mgr-2:19.2.3-687.g8d724284.el9.x86_64 3/99 2026-02-26T03:00:34.667 INFO:teuthology.orchestra.run.trial197.stdout: Verifying : ceph-mon-2:19.2.3-687.g8d724284.el9.x86_64 4/99 2026-02-26T03:00:34.667 INFO:teuthology.orchestra.run.trial197.stdout: Verifying : ceph-osd-2:19.2.3-687.g8d724284.el9.x86_64 5/99 2026-02-26T03:00:34.667 INFO:teuthology.orchestra.run.trial197.stdout: Verifying : ceph-grafana-dashboards-2:19.2.3-687.g8d724284.el9 6/99 2026-02-26T03:00:34.667 INFO:teuthology.orchestra.run.trial197.stdout: Verifying : ceph-mgr-cephadm-2:19.2.3-687.g8d724284.el9.noarch 7/99 2026-02-26T03:00:34.667 INFO:teuthology.orchestra.run.trial197.stdout: Verifying : ceph-mgr-dashboard-2:19.2.3-687.g8d724284.el9.noar 8/99 2026-02-26T03:00:34.667 INFO:teuthology.orchestra.run.trial197.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.2.3-687.g8d7242 9/99 2026-02-26T03:00:34.667 INFO:teuthology.orchestra.run.trial197.stdout: Verifying : ceph-mgr-k8sevents-2:19.2.3-687.g8d724284.el9.noar 10/99 2026-02-26T03:00:34.667 INFO:teuthology.orchestra.run.trial197.stdout: Verifying : ceph-mgr-modules-core-2:19.2.3-687.g8d724284.el9.n 11/99 2026-02-26T03:00:34.667 INFO:teuthology.orchestra.run.trial197.stdout: Verifying : ceph-mgr-rook-2:19.2.3-687.g8d724284.el9.noarch 12/99 2026-02-26T03:00:34.667 INFO:teuthology.orchestra.run.trial197.stdout: Verifying : ceph-prometheus-alerts-2:19.2.3-687.g8d724284.el9. 13/99 2026-02-26T03:00:34.667 INFO:teuthology.orchestra.run.trial197.stdout: Verifying : ceph-volume-2:19.2.3-687.g8d724284.el9.noarch 14/99 2026-02-26T03:00:34.668 INFO:teuthology.orchestra.run.trial197.stdout: Verifying : cephadm-2:19.2.3-687.g8d724284.el9.noarch 15/99 2026-02-26T03:00:34.668 INFO:teuthology.orchestra.run.trial197.stdout: Verifying : cryptsetup-2.8.1-3.el9.x86_64 16/99 2026-02-26T03:00:34.668 INFO:teuthology.orchestra.run.trial197.stdout: Verifying : ledmon-libs-1.1.0-3.el9.x86_64 17/99 2026-02-26T03:00:34.668 INFO:teuthology.orchestra.run.trial197.stdout: Verifying : libconfig-1.7.2-9.el9.x86_64 18/99 2026-02-26T03:00:34.668 INFO:teuthology.orchestra.run.trial197.stdout: Verifying : libgfortran-11.5.0-14.el9.x86_64 19/99 2026-02-26T03:00:34.668 INFO:teuthology.orchestra.run.trial197.stdout: Verifying : libquadmath-11.5.0-14.el9.x86_64 20/99 2026-02-26T03:00:34.668 INFO:teuthology.orchestra.run.trial197.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 21/99 2026-02-26T03:00:34.668 INFO:teuthology.orchestra.run.trial197.stdout: Verifying : python3-cryptography-36.0.1-5.el9.x86_64 22/99 2026-02-26T03:00:34.668 INFO:teuthology.orchestra.run.trial197.stdout: Verifying : python3-ply-3.11-14.el9.noarch 23/99 2026-02-26T03:00:34.668 INFO:teuthology.orchestra.run.trial197.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 24/99 2026-02-26T03:00:34.668 INFO:teuthology.orchestra.run.trial197.stdout: Verifying : python3-pyparsing-2.4.7-9.el9.noarch 25/99 2026-02-26T03:00:34.669 INFO:teuthology.orchestra.run.trial197.stdout: Verifying : python3-requests-2.25.1-10.el9.noarch 26/99 2026-02-26T03:00:34.669 INFO:teuthology.orchestra.run.trial197.stdout: Verifying : python3-urllib3-1.26.5-7.el9.noarch 27/99 2026-02-26T03:00:34.669 INFO:teuthology.orchestra.run.trial197.stdout: Verifying : unzip-6.0-59.el9.x86_64 28/99 2026-02-26T03:00:34.669 INFO:teuthology.orchestra.run.trial197.stdout: Verifying : zip-3.0-35.el9.x86_64 29/99 2026-02-26T03:00:34.669 INFO:teuthology.orchestra.run.trial197.stdout: Verifying : flexiblas-3.0.4-9.el9.x86_64 30/99 2026-02-26T03:00:34.669 INFO:teuthology.orchestra.run.trial197.stdout: Verifying : flexiblas-netlib-3.0.4-9.el9.x86_64 31/99 2026-02-26T03:00:34.669 INFO:teuthology.orchestra.run.trial197.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-9.el9.x86_64 32/99 2026-02-26T03:00:34.669 INFO:teuthology.orchestra.run.trial197.stdout: Verifying : libstoragemgmt-1.10.1-1.el9.x86_64 33/99 2026-02-26T03:00:34.669 INFO:teuthology.orchestra.run.trial197.stdout: Verifying : lua-5.4.4-4.el9.x86_64 34/99 2026-02-26T03:00:34.669 INFO:teuthology.orchestra.run.trial197.stdout: Verifying : openblas-0.3.29-1.el9.x86_64 35/99 2026-02-26T03:00:34.670 INFO:teuthology.orchestra.run.trial197.stdout: Verifying : openblas-openmp-0.3.29-1.el9.x86_64 36/99 2026-02-26T03:00:34.670 INFO:teuthology.orchestra.run.trial197.stdout: Verifying : protobuf-3.14.0-17.el9.x86_64 37/99 2026-02-26T03:00:34.670 INFO:teuthology.orchestra.run.trial197.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 38/99 2026-02-26T03:00:34.670 INFO:teuthology.orchestra.run.trial197.stdout: Verifying : python3-devel-3.9.25-3.el9.x86_64 39/99 2026-02-26T03:00:34.670 INFO:teuthology.orchestra.run.trial197.stdout: Verifying : python3-jinja2-2.11.3-8.el9.noarch 40/99 2026-02-26T03:00:34.670 INFO:teuthology.orchestra.run.trial197.stdout: Verifying : python3-libstoragemgmt-1.10.1-1.el9.x86_64 41/99 2026-02-26T03:00:34.670 INFO:teuthology.orchestra.run.trial197.stdout: Verifying : python3-lxml-4.6.5-3.el9.x86_64 42/99 2026-02-26T03:00:34.670 INFO:teuthology.orchestra.run.trial197.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 43/99 2026-02-26T03:00:34.671 INFO:teuthology.orchestra.run.trial197.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 44/99 2026-02-26T03:00:34.671 INFO:teuthology.orchestra.run.trial197.stdout: Verifying : python3-numpy-1:1.23.5-2.el9.x86_64 45/99 2026-02-26T03:00:34.671 INFO:teuthology.orchestra.run.trial197.stdout: Verifying : python3-numpy-f2py-1:1.23.5-2.el9.x86_64 46/99 2026-02-26T03:00:34.671 INFO:teuthology.orchestra.run.trial197.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 47/99 2026-02-26T03:00:34.671 INFO:teuthology.orchestra.run.trial197.stdout: Verifying : python3-protobuf-3.14.0-17.el9.noarch 48/99 2026-02-26T03:00:34.671 INFO:teuthology.orchestra.run.trial197.stdout: Verifying : python3-pyasn1-0.4.8-7.el9.noarch 49/99 2026-02-26T03:00:34.671 INFO:teuthology.orchestra.run.trial197.stdout: Verifying : python3-pyasn1-modules-0.4.8-7.el9.noarch 50/99 2026-02-26T03:00:34.671 INFO:teuthology.orchestra.run.trial197.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 51/99 2026-02-26T03:00:34.672 INFO:teuthology.orchestra.run.trial197.stdout: Verifying : python3-scipy-1.9.3-2.el9.x86_64 52/99 2026-02-26T03:00:34.672 INFO:teuthology.orchestra.run.trial197.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 53/99 2026-02-26T03:00:34.672 INFO:teuthology.orchestra.run.trial197.stdout: Verifying : xmlsec1-1.2.29-13.el9.x86_64 54/99 2026-02-26T03:00:34.672 INFO:teuthology.orchestra.run.trial197.stdout: Verifying : xmlsec1-openssl-1.2.29-13.el9.x86_64 55/99 2026-02-26T03:00:34.672 INFO:teuthology.orchestra.run.trial197.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 56/99 2026-02-26T03:00:34.672 INFO:teuthology.orchestra.run.trial197.stdout: Verifying : protobuf-compiler-3.14.0-17.el9.x86_64 57/99 2026-02-26T03:00:34.672 INFO:teuthology.orchestra.run.trial197.stdout: Verifying : abseil-cpp-20211102.0-4.el9.x86_64 58/99 2026-02-26T03:00:34.672 INFO:teuthology.orchestra.run.trial197.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 59/99 2026-02-26T03:00:34.672 INFO:teuthology.orchestra.run.trial197.stdout: Verifying : luarocks-3.9.2-5.el9.noarch 60/99 2026-02-26T03:00:34.672 INFO:teuthology.orchestra.run.trial197.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 61/99 2026-02-26T03:00:34.673 INFO:teuthology.orchestra.run.trial197.stdout: Verifying : python3-autocommand-2.2.2-8.el9.noarch 62/99 2026-02-26T03:00:34.673 INFO:teuthology.orchestra.run.trial197.stdout: Verifying : python3-backports-tarfile-1.2.0-1.el9.noarch 63/99 2026-02-26T03:00:34.673 INFO:teuthology.orchestra.run.trial197.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 64/99 2026-02-26T03:00:34.673 INFO:teuthology.orchestra.run.trial197.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 65/99 2026-02-26T03:00:34.673 INFO:teuthology.orchestra.run.trial197.stdout: Verifying : python3-certifi-2023.05.07-4.el9.noarch 66/99 2026-02-26T03:00:34.673 INFO:teuthology.orchestra.run.trial197.stdout: Verifying : python3-cheroot-10.0.1-4.el9.noarch 67/99 2026-02-26T03:00:34.673 INFO:teuthology.orchestra.run.trial197.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 68/99 2026-02-26T03:00:34.673 INFO:teuthology.orchestra.run.trial197.stdout: Verifying : python3-google-auth-1:2.45.0-1.el9.noarch 69/99 2026-02-26T03:00:34.673 INFO:teuthology.orchestra.run.trial197.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 70/99 2026-02-26T03:00:34.673 INFO:teuthology.orchestra.run.trial197.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 71/99 2026-02-26T03:00:34.674 INFO:teuthology.orchestra.run.trial197.stdout: Verifying : python3-influxdb-5.3.1-1.el9.noarch 72/99 2026-02-26T03:00:34.674 INFO:teuthology.orchestra.run.trial197.stdout: Verifying : python3-isodate-0.6.1-3.el9.noarch 73/99 2026-02-26T03:00:34.674 INFO:teuthology.orchestra.run.trial197.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 74/99 2026-02-26T03:00:34.674 INFO:teuthology.orchestra.run.trial197.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 75/99 2026-02-26T03:00:34.674 INFO:teuthology.orchestra.run.trial197.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 76/99 2026-02-26T03:00:34.674 INFO:teuthology.orchestra.run.trial197.stdout: Verifying : python3-jaraco-context-6.0.1-3.el9.noarch 77/99 2026-02-26T03:00:34.674 INFO:teuthology.orchestra.run.trial197.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 78/99 2026-02-26T03:00:34.674 INFO:teuthology.orchestra.run.trial197.stdout: Verifying : python3-jaraco-text-4.0.0-2.el9.noarch 79/99 2026-02-26T03:00:34.674 INFO:teuthology.orchestra.run.trial197.stdout: Verifying : python3-kubernetes-1:26.1.0-3.el9.noarch 80/99 2026-02-26T03:00:34.674 INFO:teuthology.orchestra.run.trial197.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 81/99 2026-02-26T03:00:34.674 INFO:teuthology.orchestra.run.trial197.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 82/99 2026-02-26T03:00:34.675 INFO:teuthology.orchestra.run.trial197.stdout: Verifying : python3-msgpack-1.0.3-2.el9.x86_64 83/99 2026-02-26T03:00:34.675 INFO:teuthology.orchestra.run.trial197.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 84/99 2026-02-26T03:00:34.675 INFO:teuthology.orchestra.run.trial197.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 85/99 2026-02-26T03:00:34.675 INFO:teuthology.orchestra.run.trial197.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 86/99 2026-02-26T03:00:34.675 INFO:teuthology.orchestra.run.trial197.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 87/99 2026-02-26T03:00:34.675 INFO:teuthology.orchestra.run.trial197.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 88/99 2026-02-26T03:00:34.675 INFO:teuthology.orchestra.run.trial197.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 89/99 2026-02-26T03:00:34.675 INFO:teuthology.orchestra.run.trial197.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 90/99 2026-02-26T03:00:34.675 INFO:teuthology.orchestra.run.trial197.stdout: Verifying : python3-saml-1.16.0-1.el9.noarch 91/99 2026-02-26T03:00:34.675 INFO:teuthology.orchestra.run.trial197.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 92/99 2026-02-26T03:00:34.675 INFO:teuthology.orchestra.run.trial197.stdout: Verifying : python3-typing-extensions-4.15.0-1.el9.noarch 93/99 2026-02-26T03:00:34.675 INFO:teuthology.orchestra.run.trial197.stdout: Verifying : python3-webob-1.8.8-2.el9.noarch 94/99 2026-02-26T03:00:34.676 INFO:teuthology.orchestra.run.trial197.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 95/99 2026-02-26T03:00:34.676 INFO:teuthology.orchestra.run.trial197.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 96/99 2026-02-26T03:00:34.676 INFO:teuthology.orchestra.run.trial197.stdout: Verifying : python3-xmlsec-1.3.13-1.el9.x86_64 97/99 2026-02-26T03:00:34.676 INFO:teuthology.orchestra.run.trial197.stdout: Verifying : python3-xmltodict-0.12.0-15.el9.noarch 98/99 2026-02-26T03:00:34.731 INFO:teuthology.orchestra.run.trial197.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 99/99 2026-02-26T03:00:34.731 INFO:teuthology.orchestra.run.trial197.stdout: 2026-02-26T03:00:34.731 INFO:teuthology.orchestra.run.trial197.stdout:Installed: 2026-02-26T03:00:34.731 INFO:teuthology.orchestra.run.trial197.stdout: abseil-cpp-20211102.0-4.el9.x86_64 2026-02-26T03:00:34.732 INFO:teuthology.orchestra.run.trial197.stdout: ceph-2:19.2.3-687.g8d724284.el9.x86_64 2026-02-26T03:00:34.732 INFO:teuthology.orchestra.run.trial197.stdout: ceph-grafana-dashboards-2:19.2.3-687.g8d724284.el9.noarch 2026-02-26T03:00:34.732 INFO:teuthology.orchestra.run.trial197.stdout: ceph-mds-2:19.2.3-687.g8d724284.el9.x86_64 2026-02-26T03:00:34.732 INFO:teuthology.orchestra.run.trial197.stdout: ceph-mgr-2:19.2.3-687.g8d724284.el9.x86_64 2026-02-26T03:00:34.732 INFO:teuthology.orchestra.run.trial197.stdout: ceph-mgr-cephadm-2:19.2.3-687.g8d724284.el9.noarch 2026-02-26T03:00:34.732 INFO:teuthology.orchestra.run.trial197.stdout: ceph-mgr-dashboard-2:19.2.3-687.g8d724284.el9.noarch 2026-02-26T03:00:34.732 INFO:teuthology.orchestra.run.trial197.stdout: ceph-mgr-diskprediction-local-2:19.2.3-687.g8d724284.el9.noarch 2026-02-26T03:00:34.732 INFO:teuthology.orchestra.run.trial197.stdout: ceph-mgr-k8sevents-2:19.2.3-687.g8d724284.el9.noarch 2026-02-26T03:00:34.732 INFO:teuthology.orchestra.run.trial197.stdout: ceph-mgr-modules-core-2:19.2.3-687.g8d724284.el9.noarch 2026-02-26T03:00:34.732 INFO:teuthology.orchestra.run.trial197.stdout: ceph-mgr-rook-2:19.2.3-687.g8d724284.el9.noarch 2026-02-26T03:00:34.732 INFO:teuthology.orchestra.run.trial197.stdout: ceph-mon-2:19.2.3-687.g8d724284.el9.x86_64 2026-02-26T03:00:34.733 INFO:teuthology.orchestra.run.trial197.stdout: ceph-osd-2:19.2.3-687.g8d724284.el9.x86_64 2026-02-26T03:00:34.733 INFO:teuthology.orchestra.run.trial197.stdout: ceph-prometheus-alerts-2:19.2.3-687.g8d724284.el9.noarch 2026-02-26T03:00:34.733 INFO:teuthology.orchestra.run.trial197.stdout: ceph-volume-2:19.2.3-687.g8d724284.el9.noarch 2026-02-26T03:00:34.733 INFO:teuthology.orchestra.run.trial197.stdout: cephadm-2:19.2.3-687.g8d724284.el9.noarch 2026-02-26T03:00:34.733 INFO:teuthology.orchestra.run.trial197.stdout: cryptsetup-2.8.1-3.el9.x86_64 2026-02-26T03:00:34.733 INFO:teuthology.orchestra.run.trial197.stdout: flexiblas-3.0.4-9.el9.x86_64 2026-02-26T03:00:34.733 INFO:teuthology.orchestra.run.trial197.stdout: flexiblas-netlib-3.0.4-9.el9.x86_64 2026-02-26T03:00:34.733 INFO:teuthology.orchestra.run.trial197.stdout: flexiblas-openblas-openmp-3.0.4-9.el9.x86_64 2026-02-26T03:00:34.733 INFO:teuthology.orchestra.run.trial197.stdout: grpc-data-1.46.7-10.el9.noarch 2026-02-26T03:00:34.733 INFO:teuthology.orchestra.run.trial197.stdout: ledmon-libs-1.1.0-3.el9.x86_64 2026-02-26T03:00:34.733 INFO:teuthology.orchestra.run.trial197.stdout: libconfig-1.7.2-9.el9.x86_64 2026-02-26T03:00:34.733 INFO:teuthology.orchestra.run.trial197.stdout: libgfortran-11.5.0-14.el9.x86_64 2026-02-26T03:00:34.734 INFO:teuthology.orchestra.run.trial197.stdout: libquadmath-11.5.0-14.el9.x86_64 2026-02-26T03:00:34.734 INFO:teuthology.orchestra.run.trial197.stdout: libstoragemgmt-1.10.1-1.el9.x86_64 2026-02-26T03:00:34.734 INFO:teuthology.orchestra.run.trial197.stdout: lua-5.4.4-4.el9.x86_64 2026-02-26T03:00:34.734 INFO:teuthology.orchestra.run.trial197.stdout: lua-devel-5.4.4-4.el9.x86_64 2026-02-26T03:00:34.734 INFO:teuthology.orchestra.run.trial197.stdout: luarocks-3.9.2-5.el9.noarch 2026-02-26T03:00:34.734 INFO:teuthology.orchestra.run.trial197.stdout: openblas-0.3.29-1.el9.x86_64 2026-02-26T03:00:34.734 INFO:teuthology.orchestra.run.trial197.stdout: openblas-openmp-0.3.29-1.el9.x86_64 2026-02-26T03:00:34.734 INFO:teuthology.orchestra.run.trial197.stdout: protobuf-3.14.0-17.el9.x86_64 2026-02-26T03:00:34.734 INFO:teuthology.orchestra.run.trial197.stdout: protobuf-compiler-3.14.0-17.el9.x86_64 2026-02-26T03:00:34.734 INFO:teuthology.orchestra.run.trial197.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2026-02-26T03:00:34.734 INFO:teuthology.orchestra.run.trial197.stdout: python3-autocommand-2.2.2-8.el9.noarch 2026-02-26T03:00:34.735 INFO:teuthology.orchestra.run.trial197.stdout: python3-babel-2.9.1-2.el9.noarch 2026-02-26T03:00:34.735 INFO:teuthology.orchestra.run.trial197.stdout: python3-backports-tarfile-1.2.0-1.el9.noarch 2026-02-26T03:00:34.735 INFO:teuthology.orchestra.run.trial197.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2026-02-26T03:00:34.735 INFO:teuthology.orchestra.run.trial197.stdout: python3-cachetools-4.2.4-1.el9.noarch 2026-02-26T03:00:34.735 INFO:teuthology.orchestra.run.trial197.stdout: python3-certifi-2023.05.07-4.el9.noarch 2026-02-26T03:00:34.735 INFO:teuthology.orchestra.run.trial197.stdout: python3-cffi-1.14.5-5.el9.x86_64 2026-02-26T03:00:34.735 INFO:teuthology.orchestra.run.trial197.stdout: python3-cheroot-10.0.1-4.el9.noarch 2026-02-26T03:00:34.735 INFO:teuthology.orchestra.run.trial197.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2026-02-26T03:00:34.735 INFO:teuthology.orchestra.run.trial197.stdout: python3-cryptography-36.0.1-5.el9.x86_64 2026-02-26T03:00:34.735 INFO:teuthology.orchestra.run.trial197.stdout: python3-devel-3.9.25-3.el9.x86_64 2026-02-26T03:00:34.735 INFO:teuthology.orchestra.run.trial197.stdout: python3-google-auth-1:2.45.0-1.el9.noarch 2026-02-26T03:00:34.735 INFO:teuthology.orchestra.run.trial197.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2026-02-26T03:00:34.736 INFO:teuthology.orchestra.run.trial197.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2026-02-26T03:00:34.736 INFO:teuthology.orchestra.run.trial197.stdout: python3-influxdb-5.3.1-1.el9.noarch 2026-02-26T03:00:34.736 INFO:teuthology.orchestra.run.trial197.stdout: python3-isodate-0.6.1-3.el9.noarch 2026-02-26T03:00:34.736 INFO:teuthology.orchestra.run.trial197.stdout: python3-jaraco-8.2.1-3.el9.noarch 2026-02-26T03:00:34.736 INFO:teuthology.orchestra.run.trial197.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2026-02-26T03:00:34.736 INFO:teuthology.orchestra.run.trial197.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2026-02-26T03:00:34.736 INFO:teuthology.orchestra.run.trial197.stdout: python3-jaraco-context-6.0.1-3.el9.noarch 2026-02-26T03:00:34.736 INFO:teuthology.orchestra.run.trial197.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2026-02-26T03:00:34.736 INFO:teuthology.orchestra.run.trial197.stdout: python3-jaraco-text-4.0.0-2.el9.noarch 2026-02-26T03:00:34.736 INFO:teuthology.orchestra.run.trial197.stdout: python3-jinja2-2.11.3-8.el9.noarch 2026-02-26T03:00:34.736 INFO:teuthology.orchestra.run.trial197.stdout: python3-kubernetes-1:26.1.0-3.el9.noarch 2026-02-26T03:00:34.736 INFO:teuthology.orchestra.run.trial197.stdout: python3-libstoragemgmt-1.10.1-1.el9.x86_64 2026-02-26T03:00:34.737 INFO:teuthology.orchestra.run.trial197.stdout: python3-logutils-0.3.5-21.el9.noarch 2026-02-26T03:00:34.737 INFO:teuthology.orchestra.run.trial197.stdout: python3-lxml-4.6.5-3.el9.x86_64 2026-02-26T03:00:34.737 INFO:teuthology.orchestra.run.trial197.stdout: python3-mako-1.1.4-6.el9.noarch 2026-02-26T03:00:34.737 INFO:teuthology.orchestra.run.trial197.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2026-02-26T03:00:34.737 INFO:teuthology.orchestra.run.trial197.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2026-02-26T03:00:34.737 INFO:teuthology.orchestra.run.trial197.stdout: python3-msgpack-1.0.3-2.el9.x86_64 2026-02-26T03:00:34.737 INFO:teuthology.orchestra.run.trial197.stdout: python3-natsort-7.1.1-5.el9.noarch 2026-02-26T03:00:34.737 INFO:teuthology.orchestra.run.trial197.stdout: python3-numpy-1:1.23.5-2.el9.x86_64 2026-02-26T03:00:34.737 INFO:teuthology.orchestra.run.trial197.stdout: python3-numpy-f2py-1:1.23.5-2.el9.x86_64 2026-02-26T03:00:34.737 INFO:teuthology.orchestra.run.trial197.stdout: python3-packaging-20.9-5.el9.noarch 2026-02-26T03:00:34.737 INFO:teuthology.orchestra.run.trial197.stdout: python3-pecan-1.4.2-3.el9.noarch 2026-02-26T03:00:34.737 INFO:teuthology.orchestra.run.trial197.stdout: python3-ply-3.11-14.el9.noarch 2026-02-26T03:00:34.738 INFO:teuthology.orchestra.run.trial197.stdout: python3-portend-3.1.0-2.el9.noarch 2026-02-26T03:00:34.738 INFO:teuthology.orchestra.run.trial197.stdout: python3-protobuf-3.14.0-17.el9.noarch 2026-02-26T03:00:34.738 INFO:teuthology.orchestra.run.trial197.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2026-02-26T03:00:34.738 INFO:teuthology.orchestra.run.trial197.stdout: python3-pyasn1-0.4.8-7.el9.noarch 2026-02-26T03:00:34.738 INFO:teuthology.orchestra.run.trial197.stdout: python3-pyasn1-modules-0.4.8-7.el9.noarch 2026-02-26T03:00:34.738 INFO:teuthology.orchestra.run.trial197.stdout: python3-pycparser-2.20-6.el9.noarch 2026-02-26T03:00:34.738 INFO:teuthology.orchestra.run.trial197.stdout: python3-pyparsing-2.4.7-9.el9.noarch 2026-02-26T03:00:34.738 INFO:teuthology.orchestra.run.trial197.stdout: python3-repoze-lru-0.7-16.el9.noarch 2026-02-26T03:00:34.738 INFO:teuthology.orchestra.run.trial197.stdout: python3-requests-2.25.1-10.el9.noarch 2026-02-26T03:00:34.738 INFO:teuthology.orchestra.run.trial197.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2026-02-26T03:00:34.738 INFO:teuthology.orchestra.run.trial197.stdout: python3-routes-2.5.1-5.el9.noarch 2026-02-26T03:00:34.739 INFO:teuthology.orchestra.run.trial197.stdout: python3-rsa-4.9-2.el9.noarch 2026-02-26T03:00:34.739 INFO:teuthology.orchestra.run.trial197.stdout: python3-saml-1.16.0-1.el9.noarch 2026-02-26T03:00:34.739 INFO:teuthology.orchestra.run.trial197.stdout: python3-scipy-1.9.3-2.el9.x86_64 2026-02-26T03:00:34.739 INFO:teuthology.orchestra.run.trial197.stdout: python3-tempora-5.0.0-2.el9.noarch 2026-02-26T03:00:34.739 INFO:teuthology.orchestra.run.trial197.stdout: python3-toml-0.10.2-6.el9.noarch 2026-02-26T03:00:34.739 INFO:teuthology.orchestra.run.trial197.stdout: python3-typing-extensions-4.15.0-1.el9.noarch 2026-02-26T03:00:34.739 INFO:teuthology.orchestra.run.trial197.stdout: python3-urllib3-1.26.5-7.el9.noarch 2026-02-26T03:00:34.739 INFO:teuthology.orchestra.run.trial197.stdout: python3-webob-1.8.8-2.el9.noarch 2026-02-26T03:00:34.739 INFO:teuthology.orchestra.run.trial197.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2026-02-26T03:00:34.739 INFO:teuthology.orchestra.run.trial197.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2026-02-26T03:00:34.739 INFO:teuthology.orchestra.run.trial197.stdout: python3-xmlsec-1.3.13-1.el9.x86_64 2026-02-26T03:00:34.739 INFO:teuthology.orchestra.run.trial197.stdout: python3-xmltodict-0.12.0-15.el9.noarch 2026-02-26T03:00:34.740 INFO:teuthology.orchestra.run.trial197.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2026-02-26T03:00:34.740 INFO:teuthology.orchestra.run.trial197.stdout: unzip-6.0-59.el9.x86_64 2026-02-26T03:00:34.740 INFO:teuthology.orchestra.run.trial197.stdout: xmlsec1-1.2.29-13.el9.x86_64 2026-02-26T03:00:34.740 INFO:teuthology.orchestra.run.trial197.stdout: xmlsec1-openssl-1.2.29-13.el9.x86_64 2026-02-26T03:00:34.740 INFO:teuthology.orchestra.run.trial197.stdout: zip-3.0-35.el9.x86_64 2026-02-26T03:00:34.740 INFO:teuthology.orchestra.run.trial197.stdout: 2026-02-26T03:00:34.740 INFO:teuthology.orchestra.run.trial197.stdout:Complete! 2026-02-26T03:00:34.785 DEBUG:teuthology.orchestra.run.trial197:> sudo yum -y install ceph-base 2026-02-26T03:00:35.098 INFO:teuthology.orchestra.run.trial197.stdout:Last metadata expiration check: 0:00:42 ago on Thu 26 Feb 2026 02:59:53 AM UTC. 2026-02-26T03:00:35.163 INFO:teuthology.orchestra.run.trial197.stdout:Package ceph-base-2:19.2.3-687.g8d724284.el9.x86_64 is already installed. 2026-02-26T03:00:35.182 INFO:teuthology.orchestra.run.trial197.stdout:Dependencies resolved. 2026-02-26T03:00:35.182 INFO:teuthology.orchestra.run.trial197.stdout:Nothing to do. 2026-02-26T03:00:35.182 INFO:teuthology.orchestra.run.trial197.stdout:Complete! 2026-02-26T03:00:35.204 DEBUG:teuthology.orchestra.run.trial197:> sudo yum -y install cephadm 2026-02-26T03:00:35.514 INFO:teuthology.orchestra.run.trial197.stdout:Last metadata expiration check: 0:00:42 ago on Thu 26 Feb 2026 02:59:53 AM UTC. 2026-02-26T03:00:35.579 INFO:teuthology.orchestra.run.trial197.stdout:Package cephadm-2:19.2.3-687.g8d724284.el9.noarch is already installed. 2026-02-26T03:00:35.597 INFO:teuthology.orchestra.run.trial197.stdout:Dependencies resolved. 2026-02-26T03:00:35.597 INFO:teuthology.orchestra.run.trial197.stdout:Nothing to do. 2026-02-26T03:00:35.597 INFO:teuthology.orchestra.run.trial197.stdout:Complete! 2026-02-26T03:00:35.618 DEBUG:teuthology.orchestra.run.trial197:> sudo yum -y install ceph-immutable-object-cache 2026-02-26T03:00:35.935 INFO:teuthology.orchestra.run.trial197.stdout:Last metadata expiration check: 0:00:42 ago on Thu 26 Feb 2026 02:59:53 AM UTC. 2026-02-26T03:00:36.019 INFO:teuthology.orchestra.run.trial197.stdout:Dependencies resolved. 2026-02-26T03:00:36.019 INFO:teuthology.orchestra.run.trial197.stdout:================================================================================ 2026-02-26T03:00:36.019 INFO:teuthology.orchestra.run.trial197.stdout: Package Arch Version Repo Size 2026-02-26T03:00:36.020 INFO:teuthology.orchestra.run.trial197.stdout:================================================================================ 2026-02-26T03:00:36.020 INFO:teuthology.orchestra.run.trial197.stdout:Installing: 2026-02-26T03:00:36.020 INFO:teuthology.orchestra.run.trial197.stdout: ceph-immutable-object-cache x86_64 2:19.2.3-687.g8d724284.el9 ceph 145 k 2026-02-26T03:00:36.020 INFO:teuthology.orchestra.run.trial197.stdout: 2026-02-26T03:00:36.020 INFO:teuthology.orchestra.run.trial197.stdout:Transaction Summary 2026-02-26T03:00:36.020 INFO:teuthology.orchestra.run.trial197.stdout:================================================================================ 2026-02-26T03:00:36.020 INFO:teuthology.orchestra.run.trial197.stdout:Install 1 Package 2026-02-26T03:00:36.020 INFO:teuthology.orchestra.run.trial197.stdout: 2026-02-26T03:00:36.020 INFO:teuthology.orchestra.run.trial197.stdout:Total download size: 145 k 2026-02-26T03:00:36.020 INFO:teuthology.orchestra.run.trial197.stdout:Installed size: 431 k 2026-02-26T03:00:36.020 INFO:teuthology.orchestra.run.trial197.stdout:Downloading Packages: 2026-02-26T03:00:36.323 INFO:teuthology.orchestra.run.trial197.stdout:ceph-immutable-object-cache-19.2.3-687.g8d72428 480 kB/s | 145 kB 00:00 2026-02-26T03:00:36.323 INFO:teuthology.orchestra.run.trial197.stdout:-------------------------------------------------------------------------------- 2026-02-26T03:00:36.323 INFO:teuthology.orchestra.run.trial197.stdout:Total 479 kB/s | 145 kB 00:00 2026-02-26T03:00:36.323 INFO:teuthology.orchestra.run.trial197.stdout:Running transaction check 2026-02-26T03:00:36.327 INFO:teuthology.orchestra.run.trial197.stdout:Transaction check succeeded. 2026-02-26T03:00:36.327 INFO:teuthology.orchestra.run.trial197.stdout:Running transaction test 2026-02-26T03:00:36.360 INFO:teuthology.orchestra.run.trial197.stdout:Transaction test succeeded. 2026-02-26T03:00:36.360 INFO:teuthology.orchestra.run.trial197.stdout:Running transaction 2026-02-26T03:00:36.416 INFO:teuthology.orchestra.run.trial197.stdout: Preparing : 1/1 2026-02-26T03:00:36.417 INFO:teuthology.orchestra.run.trial197.stdout: Installing : ceph-immutable-object-cache-2:19.2.3-687.g8d724284.e 1/1 2026-02-26T03:00:36.439 INFO:teuthology.orchestra.run.trial197.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.2.3-687.g8d724284.e 1/1 2026-02-26T03:00:36.439 INFO:teuthology.orchestra.run.trial197.stdout:Glob pattern passed to enable, but globs are not supported for this. 2026-02-26T03:00:36.439 INFO:teuthology.orchestra.run.trial197.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2026-02-26T03:00:36.439 INFO:teuthology.orchestra.run.trial197.stdout: 2026-02-26T03:00:36.746 INFO:teuthology.orchestra.run.trial197.stdout: Verifying : ceph-immutable-object-cache-2:19.2.3-687.g8d724284.e 1/1 2026-02-26T03:00:36.746 INFO:teuthology.orchestra.run.trial197.stdout: 2026-02-26T03:00:36.746 INFO:teuthology.orchestra.run.trial197.stdout:Installed: 2026-02-26T03:00:36.747 INFO:teuthology.orchestra.run.trial197.stdout: ceph-immutable-object-cache-2:19.2.3-687.g8d724284.el9.x86_64 2026-02-26T03:00:36.747 INFO:teuthology.orchestra.run.trial197.stdout: 2026-02-26T03:00:36.747 INFO:teuthology.orchestra.run.trial197.stdout:Complete! 2026-02-26T03:00:36.775 DEBUG:teuthology.orchestra.run.trial197:> sudo yum -y install ceph-mgr 2026-02-26T03:00:37.099 INFO:teuthology.orchestra.run.trial197.stdout:Last metadata expiration check: 0:00:44 ago on Thu 26 Feb 2026 02:59:53 AM UTC. 2026-02-26T03:00:37.164 INFO:teuthology.orchestra.run.trial197.stdout:Package ceph-mgr-2:19.2.3-687.g8d724284.el9.x86_64 is already installed. 2026-02-26T03:00:37.182 INFO:teuthology.orchestra.run.trial197.stdout:Dependencies resolved. 2026-02-26T03:00:37.183 INFO:teuthology.orchestra.run.trial197.stdout:Nothing to do. 2026-02-26T03:00:37.183 INFO:teuthology.orchestra.run.trial197.stdout:Complete! 2026-02-26T03:00:37.205 DEBUG:teuthology.orchestra.run.trial197:> sudo yum -y install ceph-mgr-dashboard 2026-02-26T03:00:37.523 INFO:teuthology.orchestra.run.trial197.stdout:Last metadata expiration check: 0:00:44 ago on Thu 26 Feb 2026 02:59:53 AM UTC. 2026-02-26T03:00:37.589 INFO:teuthology.orchestra.run.trial197.stdout:Package ceph-mgr-dashboard-2:19.2.3-687.g8d724284.el9.noarch is already installed. 2026-02-26T03:00:37.607 INFO:teuthology.orchestra.run.trial197.stdout:Dependencies resolved. 2026-02-26T03:00:37.608 INFO:teuthology.orchestra.run.trial197.stdout:Nothing to do. 2026-02-26T03:00:37.608 INFO:teuthology.orchestra.run.trial197.stdout:Complete! 2026-02-26T03:00:37.630 DEBUG:teuthology.orchestra.run.trial197:> sudo yum -y install ceph-mgr-diskprediction-local 2026-02-26T03:00:37.948 INFO:teuthology.orchestra.run.trial197.stdout:Last metadata expiration check: 0:00:44 ago on Thu 26 Feb 2026 02:59:53 AM UTC. 2026-02-26T03:00:38.013 INFO:teuthology.orchestra.run.trial197.stdout:Package ceph-mgr-diskprediction-local-2:19.2.3-687.g8d724284.el9.noarch is already installed. 2026-02-26T03:00:38.031 INFO:teuthology.orchestra.run.trial197.stdout:Dependencies resolved. 2026-02-26T03:00:38.032 INFO:teuthology.orchestra.run.trial197.stdout:Nothing to do. 2026-02-26T03:00:38.032 INFO:teuthology.orchestra.run.trial197.stdout:Complete! 2026-02-26T03:00:38.052 DEBUG:teuthology.orchestra.run.trial197:> sudo yum -y install ceph-mgr-rook 2026-02-26T03:00:38.369 INFO:teuthology.orchestra.run.trial197.stdout:Last metadata expiration check: 0:00:45 ago on Thu 26 Feb 2026 02:59:53 AM UTC. 2026-02-26T03:00:38.434 INFO:teuthology.orchestra.run.trial197.stdout:Package ceph-mgr-rook-2:19.2.3-687.g8d724284.el9.noarch is already installed. 2026-02-26T03:00:38.453 INFO:teuthology.orchestra.run.trial197.stdout:Dependencies resolved. 2026-02-26T03:00:38.453 INFO:teuthology.orchestra.run.trial197.stdout:Nothing to do. 2026-02-26T03:00:38.453 INFO:teuthology.orchestra.run.trial197.stdout:Complete! 2026-02-26T03:00:38.474 DEBUG:teuthology.orchestra.run.trial197:> sudo yum -y install ceph-mgr-cephadm 2026-02-26T03:00:38.787 INFO:teuthology.orchestra.run.trial197.stdout:Last metadata expiration check: 0:00:45 ago on Thu 26 Feb 2026 02:59:53 AM UTC. 2026-02-26T03:00:38.852 INFO:teuthology.orchestra.run.trial197.stdout:Package ceph-mgr-cephadm-2:19.2.3-687.g8d724284.el9.noarch is already installed. 2026-02-26T03:00:38.871 INFO:teuthology.orchestra.run.trial197.stdout:Dependencies resolved. 2026-02-26T03:00:38.872 INFO:teuthology.orchestra.run.trial197.stdout:Nothing to do. 2026-02-26T03:00:38.872 INFO:teuthology.orchestra.run.trial197.stdout:Complete! 2026-02-26T03:00:38.893 DEBUG:teuthology.orchestra.run.trial197:> sudo yum -y install ceph-fuse 2026-02-26T03:00:39.209 INFO:teuthology.orchestra.run.trial197.stdout:Last metadata expiration check: 0:00:46 ago on Thu 26 Feb 2026 02:59:53 AM UTC. 2026-02-26T03:00:39.295 INFO:teuthology.orchestra.run.trial197.stdout:Dependencies resolved. 2026-02-26T03:00:39.295 INFO:teuthology.orchestra.run.trial197.stdout:================================================================================ 2026-02-26T03:00:39.295 INFO:teuthology.orchestra.run.trial197.stdout: Package Architecture Version Repository Size 2026-02-26T03:00:39.295 INFO:teuthology.orchestra.run.trial197.stdout:================================================================================ 2026-02-26T03:00:39.295 INFO:teuthology.orchestra.run.trial197.stdout:Installing: 2026-02-26T03:00:39.295 INFO:teuthology.orchestra.run.trial197.stdout: ceph-fuse x86_64 2:19.2.3-687.g8d724284.el9 ceph 1.2 M 2026-02-26T03:00:39.295 INFO:teuthology.orchestra.run.trial197.stdout: 2026-02-26T03:00:39.295 INFO:teuthology.orchestra.run.trial197.stdout:Transaction Summary 2026-02-26T03:00:39.295 INFO:teuthology.orchestra.run.trial197.stdout:================================================================================ 2026-02-26T03:00:39.295 INFO:teuthology.orchestra.run.trial197.stdout:Install 1 Package 2026-02-26T03:00:39.295 INFO:teuthology.orchestra.run.trial197.stdout: 2026-02-26T03:00:39.295 INFO:teuthology.orchestra.run.trial197.stdout:Total download size: 1.2 M 2026-02-26T03:00:39.295 INFO:teuthology.orchestra.run.trial197.stdout:Installed size: 3.6 M 2026-02-26T03:00:39.295 INFO:teuthology.orchestra.run.trial197.stdout:Downloading Packages: 2026-02-26T03:00:39.416 INFO:teuthology.orchestra.run.trial197.stdout:ceph-fuse-19.2.3-687.g8d724284.el9.x86_64.rpm 9.6 MB/s | 1.2 MB 00:00 2026-02-26T03:00:39.416 INFO:teuthology.orchestra.run.trial197.stdout:-------------------------------------------------------------------------------- 2026-02-26T03:00:39.416 INFO:teuthology.orchestra.run.trial197.stdout:Total 9.5 MB/s | 1.2 MB 00:00 2026-02-26T03:00:39.416 INFO:teuthology.orchestra.run.trial197.stdout:Running transaction check 2026-02-26T03:00:39.420 INFO:teuthology.orchestra.run.trial197.stdout:Transaction check succeeded. 2026-02-26T03:00:39.420 INFO:teuthology.orchestra.run.trial197.stdout:Running transaction test 2026-02-26T03:00:39.454 INFO:teuthology.orchestra.run.trial197.stdout:Transaction test succeeded. 2026-02-26T03:00:39.454 INFO:teuthology.orchestra.run.trial197.stdout:Running transaction 2026-02-26T03:00:39.556 INFO:teuthology.orchestra.run.trial197.stdout: Preparing : 1/1 2026-02-26T03:00:39.574 INFO:teuthology.orchestra.run.trial197.stdout: Installing : ceph-fuse-2:19.2.3-687.g8d724284.el9.x86_64 1/1 2026-02-26T03:00:39.806 INFO:teuthology.orchestra.run.trial197.stdout: Running scriptlet: ceph-fuse-2:19.2.3-687.g8d724284.el9.x86_64 1/1 2026-02-26T03:00:39.869 INFO:teuthology.orchestra.run.trial197.stdout: Verifying : ceph-fuse-2:19.2.3-687.g8d724284.el9.x86_64 1/1 2026-02-26T03:00:39.869 INFO:teuthology.orchestra.run.trial197.stdout: 2026-02-26T03:00:39.869 INFO:teuthology.orchestra.run.trial197.stdout:Installed: 2026-02-26T03:00:39.869 INFO:teuthology.orchestra.run.trial197.stdout: ceph-fuse-2:19.2.3-687.g8d724284.el9.x86_64 2026-02-26T03:00:39.869 INFO:teuthology.orchestra.run.trial197.stdout: 2026-02-26T03:00:39.869 INFO:teuthology.orchestra.run.trial197.stdout:Complete! 2026-02-26T03:00:39.898 DEBUG:teuthology.orchestra.run.trial197:> sudo yum -y install ceph-volume 2026-02-26T03:00:40.211 INFO:teuthology.orchestra.run.trial197.stdout:Last metadata expiration check: 0:00:47 ago on Thu 26 Feb 2026 02:59:53 AM UTC. 2026-02-26T03:00:40.275 INFO:teuthology.orchestra.run.trial197.stdout:Package ceph-volume-2:19.2.3-687.g8d724284.el9.noarch is already installed. 2026-02-26T03:00:40.294 INFO:teuthology.orchestra.run.trial197.stdout:Dependencies resolved. 2026-02-26T03:00:40.294 INFO:teuthology.orchestra.run.trial197.stdout:Nothing to do. 2026-02-26T03:00:40.295 INFO:teuthology.orchestra.run.trial197.stdout:Complete! 2026-02-26T03:00:40.315 DEBUG:teuthology.orchestra.run.trial197:> sudo yum -y install librados-devel 2026-02-26T03:00:40.630 INFO:teuthology.orchestra.run.trial197.stdout:Last metadata expiration check: 0:00:47 ago on Thu 26 Feb 2026 02:59:53 AM UTC. 2026-02-26T03:00:40.715 INFO:teuthology.orchestra.run.trial197.stdout:Dependencies resolved. 2026-02-26T03:00:40.715 INFO:teuthology.orchestra.run.trial197.stdout:================================================================================ 2026-02-26T03:00:40.715 INFO:teuthology.orchestra.run.trial197.stdout: Package Arch Version Repo Size 2026-02-26T03:00:40.715 INFO:teuthology.orchestra.run.trial197.stdout:================================================================================ 2026-02-26T03:00:40.715 INFO:teuthology.orchestra.run.trial197.stdout:Installing: 2026-02-26T03:00:40.715 INFO:teuthology.orchestra.run.trial197.stdout: librados-devel x86_64 2:19.2.3-687.g8d724284.el9 ceph 126 k 2026-02-26T03:00:40.715 INFO:teuthology.orchestra.run.trial197.stdout: 2026-02-26T03:00:40.715 INFO:teuthology.orchestra.run.trial197.stdout:Transaction Summary 2026-02-26T03:00:40.716 INFO:teuthology.orchestra.run.trial197.stdout:================================================================================ 2026-02-26T03:00:40.716 INFO:teuthology.orchestra.run.trial197.stdout:Install 1 Package 2026-02-26T03:00:40.716 INFO:teuthology.orchestra.run.trial197.stdout: 2026-02-26T03:00:40.716 INFO:teuthology.orchestra.run.trial197.stdout:Total download size: 126 k 2026-02-26T03:00:40.716 INFO:teuthology.orchestra.run.trial197.stdout:Installed size: 456 k 2026-02-26T03:00:40.716 INFO:teuthology.orchestra.run.trial197.stdout:Downloading Packages: 2026-02-26T03:00:40.802 INFO:teuthology.orchestra.run.trial197.stdout:librados-devel-19.2.3-687.g8d724284.el9.x86_64. 1.4 MB/s | 126 kB 00:00 2026-02-26T03:00:40.802 INFO:teuthology.orchestra.run.trial197.stdout:-------------------------------------------------------------------------------- 2026-02-26T03:00:40.802 INFO:teuthology.orchestra.run.trial197.stdout:Total 1.4 MB/s | 126 kB 00:00 2026-02-26T03:00:40.803 INFO:teuthology.orchestra.run.trial197.stdout:Running transaction check 2026-02-26T03:00:40.805 INFO:teuthology.orchestra.run.trial197.stdout:Transaction check succeeded. 2026-02-26T03:00:40.805 INFO:teuthology.orchestra.run.trial197.stdout:Running transaction test 2026-02-26T03:00:40.835 INFO:teuthology.orchestra.run.trial197.stdout:Transaction test succeeded. 2026-02-26T03:00:40.836 INFO:teuthology.orchestra.run.trial197.stdout:Running transaction 2026-02-26T03:00:40.891 INFO:teuthology.orchestra.run.trial197.stdout: Preparing : 1/1 2026-02-26T03:00:40.909 INFO:teuthology.orchestra.run.trial197.stdout: Installing : librados-devel-2:19.2.3-687.g8d724284.el9.x86_64 1/1 2026-02-26T03:00:40.996 INFO:teuthology.orchestra.run.trial197.stdout: Running scriptlet: librados-devel-2:19.2.3-687.g8d724284.el9.x86_64 1/1 2026-02-26T03:00:41.059 INFO:teuthology.orchestra.run.trial197.stdout: Verifying : librados-devel-2:19.2.3-687.g8d724284.el9.x86_64 1/1 2026-02-26T03:00:41.060 INFO:teuthology.orchestra.run.trial197.stdout: 2026-02-26T03:00:41.060 INFO:teuthology.orchestra.run.trial197.stdout:Installed: 2026-02-26T03:00:41.060 INFO:teuthology.orchestra.run.trial197.stdout: librados-devel-2:19.2.3-687.g8d724284.el9.x86_64 2026-02-26T03:00:41.060 INFO:teuthology.orchestra.run.trial197.stdout: 2026-02-26T03:00:41.060 INFO:teuthology.orchestra.run.trial197.stdout:Complete! 2026-02-26T03:00:41.088 DEBUG:teuthology.orchestra.run.trial197:> sudo yum -y install libcephfs2 2026-02-26T03:00:41.407 INFO:teuthology.orchestra.run.trial197.stdout:Last metadata expiration check: 0:00:48 ago on Thu 26 Feb 2026 02:59:53 AM UTC. 2026-02-26T03:00:41.472 INFO:teuthology.orchestra.run.trial197.stdout:Package libcephfs2-2:19.2.3-687.g8d724284.el9.x86_64 is already installed. 2026-02-26T03:00:41.491 INFO:teuthology.orchestra.run.trial197.stdout:Dependencies resolved. 2026-02-26T03:00:41.492 INFO:teuthology.orchestra.run.trial197.stdout:Nothing to do. 2026-02-26T03:00:41.492 INFO:teuthology.orchestra.run.trial197.stdout:Complete! 2026-02-26T03:00:41.513 DEBUG:teuthology.orchestra.run.trial197:> sudo yum -y install libcephfs-devel 2026-02-26T03:00:41.825 INFO:teuthology.orchestra.run.trial197.stdout:Last metadata expiration check: 0:00:48 ago on Thu 26 Feb 2026 02:59:53 AM UTC. 2026-02-26T03:00:41.909 INFO:teuthology.orchestra.run.trial197.stdout:Dependencies resolved. 2026-02-26T03:00:41.910 INFO:teuthology.orchestra.run.trial197.stdout:================================================================================ 2026-02-26T03:00:41.910 INFO:teuthology.orchestra.run.trial197.stdout: Package Arch Version Repo Size 2026-02-26T03:00:41.910 INFO:teuthology.orchestra.run.trial197.stdout:================================================================================ 2026-02-26T03:00:41.910 INFO:teuthology.orchestra.run.trial197.stdout:Installing: 2026-02-26T03:00:41.910 INFO:teuthology.orchestra.run.trial197.stdout: libcephfs-devel x86_64 2:19.2.3-687.g8d724284.el9 ceph 34 k 2026-02-26T03:00:41.910 INFO:teuthology.orchestra.run.trial197.stdout: 2026-02-26T03:00:41.910 INFO:teuthology.orchestra.run.trial197.stdout:Transaction Summary 2026-02-26T03:00:41.910 INFO:teuthology.orchestra.run.trial197.stdout:================================================================================ 2026-02-26T03:00:41.910 INFO:teuthology.orchestra.run.trial197.stdout:Install 1 Package 2026-02-26T03:00:41.910 INFO:teuthology.orchestra.run.trial197.stdout: 2026-02-26T03:00:41.910 INFO:teuthology.orchestra.run.trial197.stdout:Total download size: 34 k 2026-02-26T03:00:41.910 INFO:teuthology.orchestra.run.trial197.stdout:Installed size: 153 k 2026-02-26T03:00:41.910 INFO:teuthology.orchestra.run.trial197.stdout:Downloading Packages: 2026-02-26T03:00:41.982 INFO:teuthology.orchestra.run.trial197.stdout:libcephfs-devel-19.2.3-687.g8d724284.el9.x86_64 468 kB/s | 34 kB 00:00 2026-02-26T03:00:41.982 INFO:teuthology.orchestra.run.trial197.stdout:-------------------------------------------------------------------------------- 2026-02-26T03:00:41.982 INFO:teuthology.orchestra.run.trial197.stdout:Total 461 kB/s | 34 kB 00:00 2026-02-26T03:00:41.983 INFO:teuthology.orchestra.run.trial197.stdout:Running transaction check 2026-02-26T03:00:41.985 INFO:teuthology.orchestra.run.trial197.stdout:Transaction check succeeded. 2026-02-26T03:00:41.985 INFO:teuthology.orchestra.run.trial197.stdout:Running transaction test 2026-02-26T03:00:41.987 INFO:teuthology.orchestra.run.trial197.stdout:Transaction test succeeded. 2026-02-26T03:00:41.987 INFO:teuthology.orchestra.run.trial197.stdout:Running transaction 2026-02-26T03:00:42.008 INFO:teuthology.orchestra.run.trial197.stdout: Preparing : 1/1 2026-02-26T03:00:42.025 INFO:teuthology.orchestra.run.trial197.stdout: Installing : libcephfs-devel-2:19.2.3-687.g8d724284.el9.x86_64 1/1 2026-02-26T03:00:42.085 INFO:teuthology.orchestra.run.trial197.stdout: Running scriptlet: libcephfs-devel-2:19.2.3-687.g8d724284.el9.x86_64 1/1 2026-02-26T03:00:42.147 INFO:teuthology.orchestra.run.trial197.stdout: Verifying : libcephfs-devel-2:19.2.3-687.g8d724284.el9.x86_64 1/1 2026-02-26T03:00:42.147 INFO:teuthology.orchestra.run.trial197.stdout: 2026-02-26T03:00:42.147 INFO:teuthology.orchestra.run.trial197.stdout:Installed: 2026-02-26T03:00:42.148 INFO:teuthology.orchestra.run.trial197.stdout: libcephfs-devel-2:19.2.3-687.g8d724284.el9.x86_64 2026-02-26T03:00:42.148 INFO:teuthology.orchestra.run.trial197.stdout: 2026-02-26T03:00:42.148 INFO:teuthology.orchestra.run.trial197.stdout:Complete! 2026-02-26T03:00:42.175 DEBUG:teuthology.orchestra.run.trial197:> sudo yum -y install librados2 2026-02-26T03:00:42.490 INFO:teuthology.orchestra.run.trial197.stdout:Last metadata expiration check: 0:00:49 ago on Thu 26 Feb 2026 02:59:53 AM UTC. 2026-02-26T03:00:42.555 INFO:teuthology.orchestra.run.trial197.stdout:Package librados2-2:19.2.3-687.g8d724284.el9.x86_64 is already installed. 2026-02-26T03:00:42.573 INFO:teuthology.orchestra.run.trial197.stdout:Dependencies resolved. 2026-02-26T03:00:42.574 INFO:teuthology.orchestra.run.trial197.stdout:Nothing to do. 2026-02-26T03:00:42.574 INFO:teuthology.orchestra.run.trial197.stdout:Complete! 2026-02-26T03:00:42.595 DEBUG:teuthology.orchestra.run.trial197:> sudo yum -y install librbd1 2026-02-26T03:00:42.909 INFO:teuthology.orchestra.run.trial197.stdout:Last metadata expiration check: 0:00:49 ago on Thu 26 Feb 2026 02:59:53 AM UTC. 2026-02-26T03:00:42.973 INFO:teuthology.orchestra.run.trial197.stdout:Package librbd1-2:19.2.3-687.g8d724284.el9.x86_64 is already installed. 2026-02-26T03:00:42.992 INFO:teuthology.orchestra.run.trial197.stdout:Dependencies resolved. 2026-02-26T03:00:42.992 INFO:teuthology.orchestra.run.trial197.stdout:Nothing to do. 2026-02-26T03:00:42.993 INFO:teuthology.orchestra.run.trial197.stdout:Complete! 2026-02-26T03:00:43.014 DEBUG:teuthology.orchestra.run.trial197:> sudo yum -y install python3-rados 2026-02-26T03:00:43.332 INFO:teuthology.orchestra.run.trial197.stdout:Last metadata expiration check: 0:00:50 ago on Thu 26 Feb 2026 02:59:53 AM UTC. 2026-02-26T03:00:43.397 INFO:teuthology.orchestra.run.trial197.stdout:Package python3-rados-2:19.2.3-687.g8d724284.el9.x86_64 is already installed. 2026-02-26T03:00:43.416 INFO:teuthology.orchestra.run.trial197.stdout:Dependencies resolved. 2026-02-26T03:00:43.417 INFO:teuthology.orchestra.run.trial197.stdout:Nothing to do. 2026-02-26T03:00:43.417 INFO:teuthology.orchestra.run.trial197.stdout:Complete! 2026-02-26T03:00:43.438 DEBUG:teuthology.orchestra.run.trial197:> sudo yum -y install python3-rgw 2026-02-26T03:00:43.751 INFO:teuthology.orchestra.run.trial197.stdout:Last metadata expiration check: 0:00:50 ago on Thu 26 Feb 2026 02:59:53 AM UTC. 2026-02-26T03:00:43.816 INFO:teuthology.orchestra.run.trial197.stdout:Package python3-rgw-2:19.2.3-687.g8d724284.el9.x86_64 is already installed. 2026-02-26T03:00:43.835 INFO:teuthology.orchestra.run.trial197.stdout:Dependencies resolved. 2026-02-26T03:00:43.835 INFO:teuthology.orchestra.run.trial197.stdout:Nothing to do. 2026-02-26T03:00:43.835 INFO:teuthology.orchestra.run.trial197.stdout:Complete! 2026-02-26T03:00:43.857 DEBUG:teuthology.orchestra.run.trial197:> sudo yum -y install python3-cephfs 2026-02-26T03:00:44.169 INFO:teuthology.orchestra.run.trial197.stdout:Last metadata expiration check: 0:00:51 ago on Thu 26 Feb 2026 02:59:53 AM UTC. 2026-02-26T03:00:44.235 INFO:teuthology.orchestra.run.trial197.stdout:Package python3-cephfs-2:19.2.3-687.g8d724284.el9.x86_64 is already installed. 2026-02-26T03:00:44.253 INFO:teuthology.orchestra.run.trial197.stdout:Dependencies resolved. 2026-02-26T03:00:44.254 INFO:teuthology.orchestra.run.trial197.stdout:Nothing to do. 2026-02-26T03:00:44.254 INFO:teuthology.orchestra.run.trial197.stdout:Complete! 2026-02-26T03:00:44.277 DEBUG:teuthology.orchestra.run.trial197:> sudo yum -y install python3-rbd 2026-02-26T03:00:44.592 INFO:teuthology.orchestra.run.trial197.stdout:Last metadata expiration check: 0:00:51 ago on Thu 26 Feb 2026 02:59:53 AM UTC. 2026-02-26T03:00:44.657 INFO:teuthology.orchestra.run.trial197.stdout:Package python3-rbd-2:19.2.3-687.g8d724284.el9.x86_64 is already installed. 2026-02-26T03:00:44.675 INFO:teuthology.orchestra.run.trial197.stdout:Dependencies resolved. 2026-02-26T03:00:44.676 INFO:teuthology.orchestra.run.trial197.stdout:Nothing to do. 2026-02-26T03:00:44.677 INFO:teuthology.orchestra.run.trial197.stdout:Complete! 2026-02-26T03:00:44.698 DEBUG:teuthology.orchestra.run.trial197:> sudo yum -y install rbd-fuse 2026-02-26T03:00:45.011 INFO:teuthology.orchestra.run.trial197.stdout:Last metadata expiration check: 0:00:52 ago on Thu 26 Feb 2026 02:59:53 AM UTC. 2026-02-26T03:00:45.096 INFO:teuthology.orchestra.run.trial197.stdout:Dependencies resolved. 2026-02-26T03:00:45.096 INFO:teuthology.orchestra.run.trial197.stdout:================================================================================ 2026-02-26T03:00:45.096 INFO:teuthology.orchestra.run.trial197.stdout: Package Architecture Version Repository Size 2026-02-26T03:00:45.097 INFO:teuthology.orchestra.run.trial197.stdout:================================================================================ 2026-02-26T03:00:45.097 INFO:teuthology.orchestra.run.trial197.stdout:Installing: 2026-02-26T03:00:45.097 INFO:teuthology.orchestra.run.trial197.stdout: rbd-fuse x86_64 2:19.2.3-687.g8d724284.el9 ceph 85 k 2026-02-26T03:00:45.097 INFO:teuthology.orchestra.run.trial197.stdout: 2026-02-26T03:00:45.097 INFO:teuthology.orchestra.run.trial197.stdout:Transaction Summary 2026-02-26T03:00:45.097 INFO:teuthology.orchestra.run.trial197.stdout:================================================================================ 2026-02-26T03:00:45.097 INFO:teuthology.orchestra.run.trial197.stdout:Install 1 Package 2026-02-26T03:00:45.097 INFO:teuthology.orchestra.run.trial197.stdout: 2026-02-26T03:00:45.097 INFO:teuthology.orchestra.run.trial197.stdout:Total download size: 85 k 2026-02-26T03:00:45.097 INFO:teuthology.orchestra.run.trial197.stdout:Installed size: 227 k 2026-02-26T03:00:45.098 INFO:teuthology.orchestra.run.trial197.stdout:Downloading Packages: 2026-02-26T03:00:45.404 INFO:teuthology.orchestra.run.trial197.stdout:rbd-fuse-19.2.3-687.g8d724284.el9.x86_64.rpm 277 kB/s | 85 kB 00:00 2026-02-26T03:00:45.404 INFO:teuthology.orchestra.run.trial197.stdout:-------------------------------------------------------------------------------- 2026-02-26T03:00:45.405 INFO:teuthology.orchestra.run.trial197.stdout:Total 276 kB/s | 85 kB 00:00 2026-02-26T03:00:45.405 INFO:teuthology.orchestra.run.trial197.stdout:Running transaction check 2026-02-26T03:00:45.408 INFO:teuthology.orchestra.run.trial197.stdout:Transaction check succeeded. 2026-02-26T03:00:45.408 INFO:teuthology.orchestra.run.trial197.stdout:Running transaction test 2026-02-26T03:00:45.439 INFO:teuthology.orchestra.run.trial197.stdout:Transaction test succeeded. 2026-02-26T03:00:45.439 INFO:teuthology.orchestra.run.trial197.stdout:Running transaction 2026-02-26T03:00:45.491 INFO:teuthology.orchestra.run.trial197.stdout: Preparing : 1/1 2026-02-26T03:00:45.509 INFO:teuthology.orchestra.run.trial197.stdout: Installing : rbd-fuse-2:19.2.3-687.g8d724284.el9.x86_64 1/1 2026-02-26T03:00:45.596 INFO:teuthology.orchestra.run.trial197.stdout: Running scriptlet: rbd-fuse-2:19.2.3-687.g8d724284.el9.x86_64 1/1 2026-02-26T03:00:45.660 INFO:teuthology.orchestra.run.trial197.stdout: Verifying : rbd-fuse-2:19.2.3-687.g8d724284.el9.x86_64 1/1 2026-02-26T03:00:45.660 INFO:teuthology.orchestra.run.trial197.stdout: 2026-02-26T03:00:45.660 INFO:teuthology.orchestra.run.trial197.stdout:Installed: 2026-02-26T03:00:45.660 INFO:teuthology.orchestra.run.trial197.stdout: rbd-fuse-2:19.2.3-687.g8d724284.el9.x86_64 2026-02-26T03:00:45.660 INFO:teuthology.orchestra.run.trial197.stdout: 2026-02-26T03:00:45.660 INFO:teuthology.orchestra.run.trial197.stdout:Complete! 2026-02-26T03:00:45.690 DEBUG:teuthology.orchestra.run.trial197:> sudo yum -y install rbd-mirror 2026-02-26T03:00:46.001 INFO:teuthology.orchestra.run.trial197.stdout:Last metadata expiration check: 0:00:52 ago on Thu 26 Feb 2026 02:59:53 AM UTC. 2026-02-26T03:00:46.086 INFO:teuthology.orchestra.run.trial197.stdout:Dependencies resolved. 2026-02-26T03:00:46.086 INFO:teuthology.orchestra.run.trial197.stdout:================================================================================ 2026-02-26T03:00:46.086 INFO:teuthology.orchestra.run.trial197.stdout: Package Arch Version Repo Size 2026-02-26T03:00:46.086 INFO:teuthology.orchestra.run.trial197.stdout:================================================================================ 2026-02-26T03:00:46.087 INFO:teuthology.orchestra.run.trial197.stdout:Installing: 2026-02-26T03:00:46.087 INFO:teuthology.orchestra.run.trial197.stdout: rbd-mirror x86_64 2:19.2.3-687.g8d724284.el9 ceph 3.1 M 2026-02-26T03:00:46.087 INFO:teuthology.orchestra.run.trial197.stdout: 2026-02-26T03:00:46.087 INFO:teuthology.orchestra.run.trial197.stdout:Transaction Summary 2026-02-26T03:00:46.087 INFO:teuthology.orchestra.run.trial197.stdout:================================================================================ 2026-02-26T03:00:46.087 INFO:teuthology.orchestra.run.trial197.stdout:Install 1 Package 2026-02-26T03:00:46.087 INFO:teuthology.orchestra.run.trial197.stdout: 2026-02-26T03:00:46.087 INFO:teuthology.orchestra.run.trial197.stdout:Total download size: 3.1 M 2026-02-26T03:00:46.087 INFO:teuthology.orchestra.run.trial197.stdout:Installed size: 13 M 2026-02-26T03:00:46.087 INFO:teuthology.orchestra.run.trial197.stdout:Downloading Packages: 2026-02-26T03:00:46.698 INFO:teuthology.orchestra.run.trial197.stdout:rbd-mirror-19.2.3-687.g8d724284.el9.x86_64.rpm 5.1 MB/s | 3.1 MB 00:00 2026-02-26T03:00:46.698 INFO:teuthology.orchestra.run.trial197.stdout:-------------------------------------------------------------------------------- 2026-02-26T03:00:46.698 INFO:teuthology.orchestra.run.trial197.stdout:Total 5.1 MB/s | 3.1 MB 00:00 2026-02-26T03:00:46.698 INFO:teuthology.orchestra.run.trial197.stdout:Running transaction check 2026-02-26T03:00:46.702 INFO:teuthology.orchestra.run.trial197.stdout:Transaction check succeeded. 2026-02-26T03:00:46.702 INFO:teuthology.orchestra.run.trial197.stdout:Running transaction test 2026-02-26T03:00:46.742 INFO:teuthology.orchestra.run.trial197.stdout:Transaction test succeeded. 2026-02-26T03:00:46.742 INFO:teuthology.orchestra.run.trial197.stdout:Running transaction 2026-02-26T03:00:46.935 INFO:teuthology.orchestra.run.trial197.stdout: Preparing : 1/1 2026-02-26T03:00:46.936 INFO:teuthology.orchestra.run.trial197.stdout: Installing : rbd-mirror-2:19.2.3-687.g8d724284.el9.x86_64 1/1 2026-02-26T03:00:46.959 INFO:teuthology.orchestra.run.trial197.stdout: Running scriptlet: rbd-mirror-2:19.2.3-687.g8d724284.el9.x86_64 1/1 2026-02-26T03:00:46.959 INFO:teuthology.orchestra.run.trial197.stdout:Glob pattern passed to enable, but globs are not supported for this. 2026-02-26T03:00:46.959 INFO:teuthology.orchestra.run.trial197.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2026-02-26T03:00:46.959 INFO:teuthology.orchestra.run.trial197.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2026-02-26T03:00:46.959 INFO:teuthology.orchestra.run.trial197.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2026-02-26T03:00:46.959 INFO:teuthology.orchestra.run.trial197.stdout: 2026-02-26T03:00:47.260 INFO:teuthology.orchestra.run.trial197.stdout: Verifying : rbd-mirror-2:19.2.3-687.g8d724284.el9.x86_64 1/1 2026-02-26T03:00:47.260 INFO:teuthology.orchestra.run.trial197.stdout: 2026-02-26T03:00:47.260 INFO:teuthology.orchestra.run.trial197.stdout:Installed: 2026-02-26T03:00:47.260 INFO:teuthology.orchestra.run.trial197.stdout: rbd-mirror-2:19.2.3-687.g8d724284.el9.x86_64 2026-02-26T03:00:47.260 INFO:teuthology.orchestra.run.trial197.stdout: 2026-02-26T03:00:47.260 INFO:teuthology.orchestra.run.trial197.stdout:Complete! 2026-02-26T03:00:47.289 DEBUG:teuthology.orchestra.run.trial197:> sudo yum -y install rbd-nbd 2026-02-26T03:00:47.607 INFO:teuthology.orchestra.run.trial197.stdout:Last metadata expiration check: 0:00:54 ago on Thu 26 Feb 2026 02:59:53 AM UTC. 2026-02-26T03:00:47.691 INFO:teuthology.orchestra.run.trial197.stdout:Dependencies resolved. 2026-02-26T03:00:47.691 INFO:teuthology.orchestra.run.trial197.stdout:================================================================================ 2026-02-26T03:00:47.691 INFO:teuthology.orchestra.run.trial197.stdout: Package Architecture Version Repository Size 2026-02-26T03:00:47.691 INFO:teuthology.orchestra.run.trial197.stdout:================================================================================ 2026-02-26T03:00:47.692 INFO:teuthology.orchestra.run.trial197.stdout:Installing: 2026-02-26T03:00:47.692 INFO:teuthology.orchestra.run.trial197.stdout: rbd-nbd x86_64 2:19.2.3-687.g8d724284.el9 ceph 170 k 2026-02-26T03:00:47.692 INFO:teuthology.orchestra.run.trial197.stdout: 2026-02-26T03:00:47.692 INFO:teuthology.orchestra.run.trial197.stdout:Transaction Summary 2026-02-26T03:00:47.692 INFO:teuthology.orchestra.run.trial197.stdout:================================================================================ 2026-02-26T03:00:47.692 INFO:teuthology.orchestra.run.trial197.stdout:Install 1 Package 2026-02-26T03:00:47.692 INFO:teuthology.orchestra.run.trial197.stdout: 2026-02-26T03:00:47.692 INFO:teuthology.orchestra.run.trial197.stdout:Total download size: 170 k 2026-02-26T03:00:47.692 INFO:teuthology.orchestra.run.trial197.stdout:Installed size: 490 k 2026-02-26T03:00:47.692 INFO:teuthology.orchestra.run.trial197.stdout:Downloading Packages: 2026-02-26T03:00:48.052 INFO:teuthology.orchestra.run.trial197.stdout:rbd-nbd-19.2.3-687.g8d724284.el9.x86_64.rpm 473 kB/s | 170 kB 00:00 2026-02-26T03:00:48.053 INFO:teuthology.orchestra.run.trial197.stdout:-------------------------------------------------------------------------------- 2026-02-26T03:00:48.053 INFO:teuthology.orchestra.run.trial197.stdout:Total 472 kB/s | 170 kB 00:00 2026-02-26T03:00:48.053 INFO:teuthology.orchestra.run.trial197.stdout:Running transaction check 2026-02-26T03:00:48.057 INFO:teuthology.orchestra.run.trial197.stdout:Transaction check succeeded. 2026-02-26T03:00:48.057 INFO:teuthology.orchestra.run.trial197.stdout:Running transaction test 2026-02-26T03:00:48.088 INFO:teuthology.orchestra.run.trial197.stdout:Transaction test succeeded. 2026-02-26T03:00:48.089 INFO:teuthology.orchestra.run.trial197.stdout:Running transaction 2026-02-26T03:00:48.144 INFO:teuthology.orchestra.run.trial197.stdout: Preparing : 1/1 2026-02-26T03:00:48.162 INFO:teuthology.orchestra.run.trial197.stdout: Installing : rbd-nbd-2:19.2.3-687.g8d724284.el9.x86_64 1/1 2026-02-26T03:00:48.247 INFO:teuthology.orchestra.run.trial197.stdout: Running scriptlet: rbd-nbd-2:19.2.3-687.g8d724284.el9.x86_64 1/1 2026-02-26T03:00:48.310 INFO:teuthology.orchestra.run.trial197.stdout: Verifying : rbd-nbd-2:19.2.3-687.g8d724284.el9.x86_64 1/1 2026-02-26T03:00:48.310 INFO:teuthology.orchestra.run.trial197.stdout: 2026-02-26T03:00:48.310 INFO:teuthology.orchestra.run.trial197.stdout:Installed: 2026-02-26T03:00:48.310 INFO:teuthology.orchestra.run.trial197.stdout: rbd-nbd-2:19.2.3-687.g8d724284.el9.x86_64 2026-02-26T03:00:48.310 INFO:teuthology.orchestra.run.trial197.stdout: 2026-02-26T03:00:48.310 INFO:teuthology.orchestra.run.trial197.stdout:Complete! 2026-02-26T03:00:48.339 DEBUG:teuthology.parallel:result is None 2026-02-26T03:00:48.340 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=8d724284d6d767af9ab8bf4bc6bd4e45f8502cb5 2026-02-26T03:00:48.467 DEBUG:teuthology.orchestra.run.trial197:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2026-02-26T03:00:48.489 INFO:teuthology.orchestra.run.trial197.stdout:19.2.3-687.g8d724284.el9 2026-02-26T03:00:48.489 INFO:teuthology.packaging:The installed version of ceph is 19.2.3-687.g8d724284.el9 2026-02-26T03:00:48.489 INFO:teuthology.task.install:The correct ceph version 19.2.3-687.g8d724284 is installed. 2026-02-26T03:00:48.491 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2026-02-26T03:00:48.491 DEBUG:teuthology.orchestra.run.trial197:> set -ex 2026-02-26T03:00:48.491 DEBUG:teuthology.orchestra.run.trial197:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2026-02-26T03:00:48.555 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2026-02-26T03:00:48.555 DEBUG:teuthology.orchestra.run.trial197:> set -ex 2026-02-26T03:00:48.555 DEBUG:teuthology.orchestra.run.trial197:> sudo dd of=/usr/bin/daemon-helper 2026-02-26T03:00:48.618 DEBUG:teuthology.orchestra.run.trial197:> sudo chmod a=rx -- /usr/bin/daemon-helper 2026-02-26T03:00:48.681 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2026-02-26T03:00:48.681 DEBUG:teuthology.orchestra.run.trial197:> set -ex 2026-02-26T03:00:48.681 DEBUG:teuthology.orchestra.run.trial197:> sudo dd of=/usr/bin/adjust-ulimits 2026-02-26T03:00:48.744 DEBUG:teuthology.orchestra.run.trial197:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2026-02-26T03:00:48.809 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2026-02-26T03:00:48.809 DEBUG:teuthology.orchestra.run.trial197:> set -ex 2026-02-26T03:00:48.809 DEBUG:teuthology.orchestra.run.trial197:> sudo dd of=/usr/bin/stdin-killer 2026-02-26T03:00:48.872 DEBUG:teuthology.orchestra.run.trial197:> sudo chmod a=rx -- /usr/bin/stdin-killer 2026-02-26T03:00:48.937 INFO:teuthology.run_tasks:Running task ceph... 2026-02-26T03:00:49.033 INFO:tasks.ceph:Making ceph log dir writeable by non-root... 2026-02-26T03:00:49.033 DEBUG:teuthology.orchestra.run.trial197:> sudo chmod 777 /var/log/ceph 2026-02-26T03:00:49.058 INFO:tasks.ceph:Disabling ceph logrotate... 2026-02-26T03:00:49.058 DEBUG:teuthology.orchestra.run.trial197:> sudo rm -f -- /etc/logrotate.d/ceph 2026-02-26T03:00:49.122 INFO:tasks.ceph:Creating extra log directories... 2026-02-26T03:00:49.122 DEBUG:teuthology.orchestra.run.trial197:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2026-02-26T03:00:49.193 INFO:tasks.ceph:Creating ceph cluster ceph... 2026-02-26T03:00:49.193 INFO:tasks.ceph:config {'conf': {'global': {'mon client directed command retry': 5, 'mon election default strategy': 1, 'ms inject socket failures': 5000, 'ms type': 'async'}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon scrub interval': 300}, 'osd': {'bluestore block size': 96636764160, 'bluestore compression algorithm': 'snappy', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'bluestore zero block detection': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd mclock override recovery settings': True, 'osd mclock profile': 'high_recovery_ops', 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random'}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'overall HEALTH_', '\\(OSDMAP_FLAGS\\)', '\\(OSD_', '\\(PG_', '\\(OBJECT_', '\\(POOL_APP_NOT_ENABLED\\)', '\\(OSD_SLOW_PING_TIME', '\\(MON_DOWN\\)'], 'cpu_profile': set(), 'cluster': 'ceph', 'mon_bind_msgr2': True, 'mon_bind_addrvec': True} 2026-02-26T03:00:49.193 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/mnt/teuthology/yuriw-2026-02-26_01:51:16-rados-wip-yuri9-testing-2026-02-25-1600-squid-distro-default-trial/71117', 'branch': 'wip-yuri9-testing-2026-02-25-1600-squid', 'description': 'rados/singleton/{all/divergent_priors2 mon_election/classic msgr-failures/few msgr/async objectstore/bluestore-comp-snappy rados supported-random-distro$/{centos_latest}}', 'email': 'yweinste@redhat.com', 'first_in_suite': False, 'flavor': 'default', 'job_id': '71117', 'kernel': {'branch': 'distro', 'kdb': True, 'sha1': 'distro'}, 'last_in_suite': False, 'machine_type': 'trial', 'name': 'yuriw-2026-02-26_01:51:16-rados-wip-yuri9-testing-2026-02-25-1600-squid-distro-default-trial', 'no_nested_subset': False, 'openstack': [{'volumes': {'count': 3, 'size': 10}}], 'os_type': 'centos', 'os_version': '9.stream', 'overrides': {'admin_socket': {'branch': 'wip-yuri9-testing-2026-02-25-1600-squid'}, 'ceph': {'conf': {'global': {'mon client directed command retry': 5, 'mon election default strategy': 1, 'ms inject socket failures': 5000, 'ms type': 'async'}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon scrub interval': 300}, 'osd': {'bluestore block size': 96636764160, 'bluestore compression algorithm': 'snappy', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'bluestore zero block detection': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd mclock override recovery settings': True, 'osd mclock profile': 'high_recovery_ops', 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random'}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'overall HEALTH_', '\\(OSDMAP_FLAGS\\)', '\\(OSD_', '\\(PG_', '\\(OBJECT_', '\\(POOL_APP_NOT_ENABLED\\)', '\\(OSD_SLOW_PING_TIME', '\\(MON_DOWN\\)'], 'sha1': '8d724284d6d767af9ab8bf4bc6bd4e45f8502cb5'}, 'ceph-deploy': {'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {}}}, 'install': {'ceph': {'flavor': 'default', 'sha1': '8d724284d6d767af9ab8bf4bc6bd4e45f8502cb5'}}, 'selinux': {'allowlist': ['scontext=system_u:system_r:getty_t:s0']}, 'thrashosds': {'bdev_inject_crash': 2, 'bdev_inject_crash_probability': 0.5}, 'workunit': {'branch': 'wip-yuri9-testing-2026-02-25-1600-squid', 'sha1': '8d724284d6d767af9ab8bf4bc6bd4e45f8502cb5'}}, 'owner': 'scheduled_yuriw@soko04.front.sepia.ceph.com', 'priority': 69, 'repo': 'https://github.com/ceph/ceph-ci.git', 'roles': [['mon.a', 'mgr.x', 'osd.0', 'osd.1', 'osd.2', 'client.0']], 'seed': 1525, 'sha1': '8d724284d6d767af9ab8bf4bc6bd4e45f8502cb5', 'sleep_before_teardown': 0, 'subset': '111/120000', 'suite': 'rados', 'suite_branch': 'wip-yuri9-testing-2026-02-25-1600-squid', 'suite_path': '/home/teuthworker/src/github.com_ceph_ceph-c_8d724284d6d767af9ab8bf4bc6bd4e45f8502cb5/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://github.com/ceph/ceph-ci.git', 'suite_sha1': '8d724284d6d767af9ab8bf4bc6bd4e45f8502cb5', 'targets': {'trial197.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBP7CMkwbyz7vDJuG2HaLTRkZgbc4OUrKmvLXOk/PM1MD4mX2CZNKLwxpycgmJydkkoUmssKmhi6XrZLW06Lwwxs='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'branch': 'distro', 'kdb': True, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': None}, {'ceph': {'pre-mgr-commands': ['sudo ceph config set mgr mgr_pool false --force'], 'conf': {'global': {'mon client directed command retry': 5, 'mon election default strategy': 1, 'ms inject socket failures': 5000, 'ms type': 'async'}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon scrub interval': 300}, 'osd': {'bluestore block size': 96636764160, 'bluestore compression algorithm': 'snappy', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'bluestore zero block detection': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd mclock override recovery settings': True, 'osd mclock profile': 'high_recovery_ops', 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random'}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'overall HEALTH_', '\\(OSDMAP_FLAGS\\)', '\\(OSD_', '\\(PG_', '\\(OBJECT_', '\\(POOL_APP_NOT_ENABLED\\)', '\\(OSD_SLOW_PING_TIME', '\\(MON_DOWN\\)'], 'sha1': '8d724284d6d767af9ab8bf4bc6bd4e45f8502cb5', 'cluster': 'ceph'}}, {'divergent_priors2': None}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': []}, 'teuthology_branch': 'main', 'teuthology_sha1': '6e3d0419ccca0e75a03e08feb708946e198236f4', 'timestamp': '2026-02-26_01:51:16', 'tube': 'trial', 'user': 'yuriw', 'verbose': True, 'worker_log': '/home/teuthworker/mnt/teuthology/worker_logs/dispatcher.trial.904699'} 2026-02-26T03:00:49.194 DEBUG:teuthology.orchestra.run.trial197:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2026-02-26T03:00:49.241 DEBUG:teuthology.orchestra.run.trial197:> sudo install -d -m0777 -- /var/run/ceph 2026-02-26T03:00:49.305 DEBUG:teuthology.orchestra.run.trial197:> set -ex 2026-02-26T03:00:49.305 DEBUG:teuthology.orchestra.run.trial197:> dd if=/scratch_devs of=/dev/stdout 2026-02-26T03:00:49.361 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2026-02-26T03:00:49.361 DEBUG:teuthology.orchestra.run.trial197:> stat /dev/vg_nvme/lv_1 2026-02-26T03:00:49.418 INFO:teuthology.orchestra.run.trial197.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2026-02-26T03:00:49.418 INFO:teuthology.orchestra.run.trial197.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2026-02-26T03:00:49.418 INFO:teuthology.orchestra.run.trial197.stdout:Device: 6h/6d Inode: 1313 Links: 1 2026-02-26T03:00:49.419 INFO:teuthology.orchestra.run.trial197.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2026-02-26T03:00:49.419 INFO:teuthology.orchestra.run.trial197.stdout:Context: system_u:object_r:device_t:s0 2026-02-26T03:00:49.419 INFO:teuthology.orchestra.run.trial197.stdout:Access: 2026-02-26 03:00:47.099771704 +0000 2026-02-26T03:00:49.419 INFO:teuthology.orchestra.run.trial197.stdout:Modify: 2026-02-26 02:59:59.979436904 +0000 2026-02-26T03:00:49.419 INFO:teuthology.orchestra.run.trial197.stdout:Change: 2026-02-26 02:59:59.979436904 +0000 2026-02-26T03:00:49.419 INFO:teuthology.orchestra.run.trial197.stdout: Birth: 2026-02-26 02:59:59.979436904 +0000 2026-02-26T03:00:49.419 DEBUG:teuthology.orchestra.run.trial197:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2026-02-26T03:00:49.481 INFO:teuthology.orchestra.run.trial197.stderr:1+0 records in 2026-02-26T03:00:49.481 INFO:teuthology.orchestra.run.trial197.stderr:1+0 records out 2026-02-26T03:00:49.481 INFO:teuthology.orchestra.run.trial197.stderr:512 bytes copied, 0.000156472 s, 3.3 MB/s 2026-02-26T03:00:49.482 DEBUG:teuthology.orchestra.run.trial197:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2026-02-26T03:00:49.539 DEBUG:teuthology.orchestra.run.trial197:> stat /dev/vg_nvme/lv_2 2026-02-26T03:00:49.596 INFO:teuthology.orchestra.run.trial197.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2026-02-26T03:00:49.596 INFO:teuthology.orchestra.run.trial197.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2026-02-26T03:00:49.596 INFO:teuthology.orchestra.run.trial197.stdout:Device: 6h/6d Inode: 1317 Links: 1 2026-02-26T03:00:49.597 INFO:teuthology.orchestra.run.trial197.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2026-02-26T03:00:49.597 INFO:teuthology.orchestra.run.trial197.stdout:Context: system_u:object_r:device_t:s0 2026-02-26T03:00:49.597 INFO:teuthology.orchestra.run.trial197.stdout:Access: 2026-02-26 03:00:47.100771690 +0000 2026-02-26T03:00:49.597 INFO:teuthology.orchestra.run.trial197.stdout:Modify: 2026-02-26 02:59:59.979436904 +0000 2026-02-26T03:00:49.597 INFO:teuthology.orchestra.run.trial197.stdout:Change: 2026-02-26 02:59:59.979436904 +0000 2026-02-26T03:00:49.597 INFO:teuthology.orchestra.run.trial197.stdout: Birth: 2026-02-26 02:59:59.979436904 +0000 2026-02-26T03:00:49.598 DEBUG:teuthology.orchestra.run.trial197:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2026-02-26T03:00:49.658 INFO:teuthology.orchestra.run.trial197.stderr:1+0 records in 2026-02-26T03:00:49.658 INFO:teuthology.orchestra.run.trial197.stderr:1+0 records out 2026-02-26T03:00:49.658 INFO:teuthology.orchestra.run.trial197.stderr:512 bytes copied, 0.000138628 s, 3.7 MB/s 2026-02-26T03:00:49.659 DEBUG:teuthology.orchestra.run.trial197:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2026-02-26T03:00:49.715 DEBUG:teuthology.orchestra.run.trial197:> stat /dev/vg_nvme/lv_3 2026-02-26T03:00:49.771 INFO:teuthology.orchestra.run.trial197.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2026-02-26T03:00:49.771 INFO:teuthology.orchestra.run.trial197.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2026-02-26T03:00:49.771 INFO:teuthology.orchestra.run.trial197.stdout:Device: 6h/6d Inode: 1298 Links: 1 2026-02-26T03:00:49.771 INFO:teuthology.orchestra.run.trial197.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2026-02-26T03:00:49.772 INFO:teuthology.orchestra.run.trial197.stdout:Context: system_u:object_r:device_t:s0 2026-02-26T03:00:49.772 INFO:teuthology.orchestra.run.trial197.stdout:Access: 2026-02-26 03:00:47.100771690 +0000 2026-02-26T03:00:49.772 INFO:teuthology.orchestra.run.trial197.stdout:Modify: 2026-02-26 02:59:59.978436918 +0000 2026-02-26T03:00:49.772 INFO:teuthology.orchestra.run.trial197.stdout:Change: 2026-02-26 02:59:59.978436918 +0000 2026-02-26T03:00:49.772 INFO:teuthology.orchestra.run.trial197.stdout: Birth: 2026-02-26 02:59:59.978436918 +0000 2026-02-26T03:00:49.772 DEBUG:teuthology.orchestra.run.trial197:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2026-02-26T03:00:49.834 INFO:teuthology.orchestra.run.trial197.stderr:1+0 records in 2026-02-26T03:00:49.835 INFO:teuthology.orchestra.run.trial197.stderr:1+0 records out 2026-02-26T03:00:49.835 INFO:teuthology.orchestra.run.trial197.stderr:512 bytes copied, 0.000152464 s, 3.4 MB/s 2026-02-26T03:00:49.836 DEBUG:teuthology.orchestra.run.trial197:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2026-02-26T03:00:49.892 DEBUG:teuthology.orchestra.run.trial197:> stat /dev/vg_nvme/lv_4 2026-02-26T03:00:49.949 INFO:teuthology.orchestra.run.trial197.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2026-02-26T03:00:49.949 INFO:teuthology.orchestra.run.trial197.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2026-02-26T03:00:49.949 INFO:teuthology.orchestra.run.trial197.stdout:Device: 6h/6d Inode: 1299 Links: 1 2026-02-26T03:00:49.949 INFO:teuthology.orchestra.run.trial197.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2026-02-26T03:00:49.949 INFO:teuthology.orchestra.run.trial197.stdout:Context: system_u:object_r:device_t:s0 2026-02-26T03:00:49.949 INFO:teuthology.orchestra.run.trial197.stdout:Access: 2026-02-26 03:00:47.100771690 +0000 2026-02-26T03:00:49.949 INFO:teuthology.orchestra.run.trial197.stdout:Modify: 2026-02-26 02:59:59.978436918 +0000 2026-02-26T03:00:49.949 INFO:teuthology.orchestra.run.trial197.stdout:Change: 2026-02-26 02:59:59.978436918 +0000 2026-02-26T03:00:49.950 INFO:teuthology.orchestra.run.trial197.stdout: Birth: 2026-02-26 02:59:59.978436918 +0000 2026-02-26T03:00:49.950 DEBUG:teuthology.orchestra.run.trial197:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2026-02-26T03:00:50.012 INFO:teuthology.orchestra.run.trial197.stderr:1+0 records in 2026-02-26T03:00:50.012 INFO:teuthology.orchestra.run.trial197.stderr:1+0 records out 2026-02-26T03:00:50.012 INFO:teuthology.orchestra.run.trial197.stderr:512 bytes copied, 0.000131655 s, 3.9 MB/s 2026-02-26T03:00:50.013 DEBUG:teuthology.orchestra.run.trial197:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2026-02-26T03:00:50.071 INFO:tasks.ceph:osd dev map: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2026-02-26T03:00:50.071 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@trial197.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'}} 2026-02-26T03:00:50.071 INFO:tasks.ceph:Generating config... 2026-02-26T03:00:50.072 INFO:tasks.ceph:[global] mon client directed command retry = 5 2026-02-26T03:00:50.072 INFO:tasks.ceph:[global] mon election default strategy = 1 2026-02-26T03:00:50.072 INFO:tasks.ceph:[global] ms inject socket failures = 5000 2026-02-26T03:00:50.072 INFO:tasks.ceph:[global] ms type = async 2026-02-26T03:00:50.072 INFO:tasks.ceph:[mgr] debug mgr = 20 2026-02-26T03:00:50.072 INFO:tasks.ceph:[mgr] debug ms = 1 2026-02-26T03:00:50.072 INFO:tasks.ceph:[mon] debug mon = 20 2026-02-26T03:00:50.072 INFO:tasks.ceph:[mon] debug ms = 1 2026-02-26T03:00:50.072 INFO:tasks.ceph:[mon] debug paxos = 20 2026-02-26T03:00:50.072 INFO:tasks.ceph:[mon] mon scrub interval = 300 2026-02-26T03:00:50.072 INFO:tasks.ceph:[osd] bluestore block size = 96636764160 2026-02-26T03:00:50.072 INFO:tasks.ceph:[osd] bluestore compression algorithm = snappy 2026-02-26T03:00:50.072 INFO:tasks.ceph:[osd] bluestore compression mode = aggressive 2026-02-26T03:00:50.073 INFO:tasks.ceph:[osd] bluestore fsck on mount = True 2026-02-26T03:00:50.073 INFO:tasks.ceph:[osd] bluestore zero block detection = True 2026-02-26T03:00:50.073 INFO:tasks.ceph:[osd] debug bluefs = 20 2026-02-26T03:00:50.073 INFO:tasks.ceph:[osd] debug bluestore = 20 2026-02-26T03:00:50.073 INFO:tasks.ceph:[osd] debug ms = 1 2026-02-26T03:00:50.073 INFO:tasks.ceph:[osd] debug osd = 20 2026-02-26T03:00:50.073 INFO:tasks.ceph:[osd] debug rocksdb = 10 2026-02-26T03:00:50.073 INFO:tasks.ceph:[osd] mon osd backfillfull_ratio = 0.85 2026-02-26T03:00:50.073 INFO:tasks.ceph:[osd] mon osd full ratio = 0.9 2026-02-26T03:00:50.073 INFO:tasks.ceph:[osd] mon osd nearfull ratio = 0.8 2026-02-26T03:00:50.073 INFO:tasks.ceph:[osd] osd debug verify cached snaps = True 2026-02-26T03:00:50.073 INFO:tasks.ceph:[osd] osd debug verify missing on start = True 2026-02-26T03:00:50.073 INFO:tasks.ceph:[osd] osd failsafe full ratio = 0.95 2026-02-26T03:00:50.073 INFO:tasks.ceph:[osd] osd mclock override recovery settings = True 2026-02-26T03:00:50.073 INFO:tasks.ceph:[osd] osd mclock profile = high_recovery_ops 2026-02-26T03:00:50.073 INFO:tasks.ceph:[osd] osd objectstore = bluestore 2026-02-26T03:00:50.073 INFO:tasks.ceph:[osd] osd op queue = debug_random 2026-02-26T03:00:50.073 INFO:tasks.ceph:[osd] osd op queue cut off = debug_random 2026-02-26T03:00:50.073 INFO:tasks.ceph:Setting up mon.a... 2026-02-26T03:00:50.073 DEBUG:teuthology.orchestra.run.trial197:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/ceph.keyring 2026-02-26T03:00:50.144 INFO:teuthology.orchestra.run.trial197.stdout:creating /etc/ceph/ceph.keyring 2026-02-26T03:00:50.147 DEBUG:teuthology.orchestra.run.trial197:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/ceph.keyring 2026-02-26T03:00:50.224 DEBUG:teuthology.orchestra.run.trial197:> sudo chmod 0644 /etc/ceph/ceph.keyring 2026-02-26T03:00:50.288 DEBUG:tasks.ceph:Ceph mon addresses: [('mon.a', '10.20.193.197')] 2026-02-26T03:00:50.288 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'cephx', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '10.20.193.197', 'mon client directed command retry': 5, 'mon election default strategy': 1, 'ms inject socket failures': 5000, 'ms type': 'async'}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd mclock profile': 'high_recovery_ops', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd debug trim objects': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'bluestore block size': 96636764160, 'bluestore compression algorithm': 'snappy', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'bluestore zero block detection': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd mclock override recovery settings': True, 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random'}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false'}, 'mon': {'debug ms': 1, 'debug mon': 20, 'debug paxos': 20, 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'mon_down_mkfs_grace': '2m', 'mon_warn_on_filestore_osds': 'false', 'mon scrub interval': 300}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok'}, 'mon.a': {}} 2026-02-26T03:00:50.290 DEBUG:teuthology.orchestra.run.trial197:> set -ex 2026-02-26T03:00:50.290 DEBUG:teuthology.orchestra.run.trial197:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2026-02-26T03:00:50.344 DEBUG:teuthology.orchestra.run.trial197:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --add a 10.20.193.197 --print /home/ubuntu/cephtest/ceph.monmap 2026-02-26T03:00:50.414 INFO:teuthology.orchestra.run.trial197.stdout:monmaptool: monmap file /home/ubuntu/cephtest/ceph.monmap 2026-02-26T03:00:50.414 INFO:teuthology.orchestra.run.trial197.stdout:monmaptool: generated fsid 658087b3-2758-487d-ba7c-f8b171a8f312 2026-02-26T03:00:50.414 INFO:teuthology.orchestra.run.trial197.stdout:setting min_mon_release = quincy 2026-02-26T03:00:50.414 INFO:teuthology.orchestra.run.trial197.stdout:epoch 0 2026-02-26T03:00:50.414 INFO:teuthology.orchestra.run.trial197.stdout:fsid 658087b3-2758-487d-ba7c-f8b171a8f312 2026-02-26T03:00:50.415 INFO:teuthology.orchestra.run.trial197.stdout:last_changed 2026-02-26T03:00:50.413237+0000 2026-02-26T03:00:50.415 INFO:teuthology.orchestra.run.trial197.stdout:created 2026-02-26T03:00:50.413237+0000 2026-02-26T03:00:50.415 INFO:teuthology.orchestra.run.trial197.stdout:min_mon_release 17 (quincy) 2026-02-26T03:00:50.415 INFO:teuthology.orchestra.run.trial197.stdout:election_strategy: 1 2026-02-26T03:00:50.415 INFO:teuthology.orchestra.run.trial197.stdout:0: [v2:10.20.193.197:3300/0,v1:10.20.193.197:6789/0] mon.a 2026-02-26T03:00:50.415 INFO:teuthology.orchestra.run.trial197.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/ceph.monmap (1 monitors) 2026-02-26T03:00:50.417 DEBUG:teuthology.orchestra.run.trial197:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2026-02-26T03:00:50.473 INFO:tasks.ceph:Writing /etc/ceph/ceph.conf for FSID 658087b3-2758-487d-ba7c-f8b171a8f312... 2026-02-26T03:00:50.473 DEBUG:teuthology.orchestra.run.trial197:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2026-02-26T03:00:50.554 INFO:teuthology.orchestra.run.trial197.stdout:[global] 2026-02-26T03:00:50.554 INFO:teuthology.orchestra.run.trial197.stdout: chdir = "" 2026-02-26T03:00:50.554 INFO:teuthology.orchestra.run.trial197.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2026-02-26T03:00:50.554 INFO:teuthology.orchestra.run.trial197.stdout: auth supported = cephx 2026-02-26T03:00:50.554 INFO:teuthology.orchestra.run.trial197.stdout: 2026-02-26T03:00:50.554 INFO:teuthology.orchestra.run.trial197.stdout: filestore xattr use omap = true 2026-02-26T03:00:50.554 INFO:teuthology.orchestra.run.trial197.stdout: 2026-02-26T03:00:50.554 INFO:teuthology.orchestra.run.trial197.stdout: mon clock drift allowed = 1.000 2026-02-26T03:00:50.554 INFO:teuthology.orchestra.run.trial197.stdout: 2026-02-26T03:00:50.554 INFO:teuthology.orchestra.run.trial197.stdout: osd crush chooseleaf type = 0 2026-02-26T03:00:50.554 INFO:teuthology.orchestra.run.trial197.stdout: auth debug = true 2026-02-26T03:00:50.554 INFO:teuthology.orchestra.run.trial197.stdout: 2026-02-26T03:00:50.554 INFO:teuthology.orchestra.run.trial197.stdout: ms die on old message = true 2026-02-26T03:00:50.554 INFO:teuthology.orchestra.run.trial197.stdout: ms die on bug = true 2026-02-26T03:00:50.554 INFO:teuthology.orchestra.run.trial197.stdout: 2026-02-26T03:00:50.554 INFO:teuthology.orchestra.run.trial197.stdout: mon max pg per osd = 10000 # >= luminous 2026-02-26T03:00:50.554 INFO:teuthology.orchestra.run.trial197.stdout: mon pg warn max object skew = 0 2026-02-26T03:00:50.554 INFO:teuthology.orchestra.run.trial197.stdout: 2026-02-26T03:00:50.554 INFO:teuthology.orchestra.run.trial197.stdout: # disable pg_autoscaler by default for new pools 2026-02-26T03:00:50.554 INFO:teuthology.orchestra.run.trial197.stdout: osd_pool_default_pg_autoscale_mode = off 2026-02-26T03:00:50.555 INFO:teuthology.orchestra.run.trial197.stdout: 2026-02-26T03:00:50.555 INFO:teuthology.orchestra.run.trial197.stdout: osd pool default size = 2 2026-02-26T03:00:50.555 INFO:teuthology.orchestra.run.trial197.stdout: 2026-02-26T03:00:50.555 INFO:teuthology.orchestra.run.trial197.stdout: mon osd allow primary affinity = true 2026-02-26T03:00:50.555 INFO:teuthology.orchestra.run.trial197.stdout: mon osd allow pg remap = true 2026-02-26T03:00:50.555 INFO:teuthology.orchestra.run.trial197.stdout: mon warn on legacy crush tunables = false 2026-02-26T03:00:50.555 INFO:teuthology.orchestra.run.trial197.stdout: mon warn on crush straw calc version zero = false 2026-02-26T03:00:50.555 INFO:teuthology.orchestra.run.trial197.stdout: mon warn on no sortbitwise = false 2026-02-26T03:00:50.555 INFO:teuthology.orchestra.run.trial197.stdout: mon warn on osd down out interval zero = false 2026-02-26T03:00:50.555 INFO:teuthology.orchestra.run.trial197.stdout: mon warn on too few osds = false 2026-02-26T03:00:50.555 INFO:teuthology.orchestra.run.trial197.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2026-02-26T03:00:50.555 INFO:teuthology.orchestra.run.trial197.stdout: mon_warn_on_pool_no_redundancy = false 2026-02-26T03:00:50.555 INFO:teuthology.orchestra.run.trial197.stdout: mon_allow_pool_size_one = true 2026-02-26T03:00:50.555 INFO:teuthology.orchestra.run.trial197.stdout: 2026-02-26T03:00:50.555 INFO:teuthology.orchestra.run.trial197.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2026-02-26T03:00:50.555 INFO:teuthology.orchestra.run.trial197.stdout: 2026-02-26T03:00:50.555 INFO:teuthology.orchestra.run.trial197.stdout: osd default data pool replay window = 5 2026-02-26T03:00:50.555 INFO:teuthology.orchestra.run.trial197.stdout: 2026-02-26T03:00:50.555 INFO:teuthology.orchestra.run.trial197.stdout: mon allow pool delete = true 2026-02-26T03:00:50.555 INFO:teuthology.orchestra.run.trial197.stdout: 2026-02-26T03:00:50.555 INFO:teuthology.orchestra.run.trial197.stdout: mon cluster log file level = debug 2026-02-26T03:00:50.555 INFO:teuthology.orchestra.run.trial197.stdout: debug asserts on shutdown = true 2026-02-26T03:00:50.555 INFO:teuthology.orchestra.run.trial197.stdout: mon health detail to clog = false 2026-02-26T03:00:50.555 INFO:teuthology.orchestra.run.trial197.stdout: mon host = 10.20.193.197 2026-02-26T03:00:50.555 INFO:teuthology.orchestra.run.trial197.stdout: mon client directed command retry = 5 2026-02-26T03:00:50.555 INFO:teuthology.orchestra.run.trial197.stdout: mon election default strategy = 1 2026-02-26T03:00:50.555 INFO:teuthology.orchestra.run.trial197.stdout: ms inject socket failures = 5000 2026-02-26T03:00:50.555 INFO:teuthology.orchestra.run.trial197.stdout: ms type = async 2026-02-26T03:00:50.555 INFO:teuthology.orchestra.run.trial197.stdout: fsid = 658087b3-2758-487d-ba7c-f8b171a8f312 2026-02-26T03:00:50.555 INFO:teuthology.orchestra.run.trial197.stdout: 2026-02-26T03:00:50.555 INFO:teuthology.orchestra.run.trial197.stdout:[osd] 2026-02-26T03:00:50.555 INFO:teuthology.orchestra.run.trial197.stdout: osd journal size = 100 2026-02-26T03:00:50.555 INFO:teuthology.orchestra.run.trial197.stdout: 2026-02-26T03:00:50.555 INFO:teuthology.orchestra.run.trial197.stdout: osd scrub load threshold = 5.0 2026-02-26T03:00:50.556 INFO:teuthology.orchestra.run.trial197.stdout: osd scrub max interval = 600 2026-02-26T03:00:50.556 INFO:teuthology.orchestra.run.trial197.stdout: osd mclock profile = high_recovery_ops 2026-02-26T03:00:50.556 INFO:teuthology.orchestra.run.trial197.stdout: 2026-02-26T03:00:50.556 INFO:teuthology.orchestra.run.trial197.stdout: osd recover clone overlap = true 2026-02-26T03:00:50.556 INFO:teuthology.orchestra.run.trial197.stdout: osd recovery max chunk = 1048576 2026-02-26T03:00:50.556 INFO:teuthology.orchestra.run.trial197.stdout: 2026-02-26T03:00:50.556 INFO:teuthology.orchestra.run.trial197.stdout: osd debug shutdown = true 2026-02-26T03:00:50.556 INFO:teuthology.orchestra.run.trial197.stdout: osd debug op order = true 2026-02-26T03:00:50.556 INFO:teuthology.orchestra.run.trial197.stdout: osd debug verify stray on activate = true 2026-02-26T03:00:50.556 INFO:teuthology.orchestra.run.trial197.stdout: osd debug trim objects = true 2026-02-26T03:00:50.556 INFO:teuthology.orchestra.run.trial197.stdout: 2026-02-26T03:00:50.556 INFO:teuthology.orchestra.run.trial197.stdout: osd open classes on start = true 2026-02-26T03:00:50.556 INFO:teuthology.orchestra.run.trial197.stdout: osd debug pg log writeout = true 2026-02-26T03:00:50.556 INFO:teuthology.orchestra.run.trial197.stdout: 2026-02-26T03:00:50.556 INFO:teuthology.orchestra.run.trial197.stdout: osd deep scrub update digest min age = 30 2026-02-26T03:00:50.556 INFO:teuthology.orchestra.run.trial197.stdout: 2026-02-26T03:00:50.556 INFO:teuthology.orchestra.run.trial197.stdout: osd map max advance = 10 2026-02-26T03:00:50.556 INFO:teuthology.orchestra.run.trial197.stdout: 2026-02-26T03:00:50.556 INFO:teuthology.orchestra.run.trial197.stdout: journal zero on create = true 2026-02-26T03:00:50.556 INFO:teuthology.orchestra.run.trial197.stdout: 2026-02-26T03:00:50.556 INFO:teuthology.orchestra.run.trial197.stdout: filestore ondisk finisher threads = 3 2026-02-26T03:00:50.556 INFO:teuthology.orchestra.run.trial197.stdout: filestore apply finisher threads = 3 2026-02-26T03:00:50.556 INFO:teuthology.orchestra.run.trial197.stdout: 2026-02-26T03:00:50.556 INFO:teuthology.orchestra.run.trial197.stdout: bdev debug aio = true 2026-02-26T03:00:50.556 INFO:teuthology.orchestra.run.trial197.stdout: osd debug misdirected ops = true 2026-02-26T03:00:50.556 INFO:teuthology.orchestra.run.trial197.stdout: bluestore block size = 96636764160 2026-02-26T03:00:50.556 INFO:teuthology.orchestra.run.trial197.stdout: bluestore compression algorithm = snappy 2026-02-26T03:00:50.556 INFO:teuthology.orchestra.run.trial197.stdout: bluestore compression mode = aggressive 2026-02-26T03:00:50.556 INFO:teuthology.orchestra.run.trial197.stdout: bluestore fsck on mount = True 2026-02-26T03:00:50.556 INFO:teuthology.orchestra.run.trial197.stdout: bluestore zero block detection = True 2026-02-26T03:00:50.556 INFO:teuthology.orchestra.run.trial197.stdout: debug bluefs = 20 2026-02-26T03:00:50.556 INFO:teuthology.orchestra.run.trial197.stdout: debug bluestore = 20 2026-02-26T03:00:50.556 INFO:teuthology.orchestra.run.trial197.stdout: debug ms = 1 2026-02-26T03:00:50.556 INFO:teuthology.orchestra.run.trial197.stdout: debug osd = 20 2026-02-26T03:00:50.556 INFO:teuthology.orchestra.run.trial197.stdout: debug rocksdb = 10 2026-02-26T03:00:50.556 INFO:teuthology.orchestra.run.trial197.stdout: mon osd backfillfull_ratio = 0.85 2026-02-26T03:00:50.556 INFO:teuthology.orchestra.run.trial197.stdout: mon osd full ratio = 0.9 2026-02-26T03:00:50.557 INFO:teuthology.orchestra.run.trial197.stdout: mon osd nearfull ratio = 0.8 2026-02-26T03:00:50.557 INFO:teuthology.orchestra.run.trial197.stdout: osd debug verify cached snaps = True 2026-02-26T03:00:50.557 INFO:teuthology.orchestra.run.trial197.stdout: osd debug verify missing on start = True 2026-02-26T03:00:50.557 INFO:teuthology.orchestra.run.trial197.stdout: osd failsafe full ratio = 0.95 2026-02-26T03:00:50.557 INFO:teuthology.orchestra.run.trial197.stdout: osd mclock override recovery settings = True 2026-02-26T03:00:50.557 INFO:teuthology.orchestra.run.trial197.stdout: osd objectstore = bluestore 2026-02-26T03:00:50.557 INFO:teuthology.orchestra.run.trial197.stdout: osd op queue = debug_random 2026-02-26T03:00:50.557 INFO:teuthology.orchestra.run.trial197.stdout: osd op queue cut off = debug_random 2026-02-26T03:00:50.557 INFO:teuthology.orchestra.run.trial197.stdout: 2026-02-26T03:00:50.557 INFO:teuthology.orchestra.run.trial197.stdout:[mgr] 2026-02-26T03:00:50.557 INFO:teuthology.orchestra.run.trial197.stdout: debug ms = 1 2026-02-26T03:00:50.557 INFO:teuthology.orchestra.run.trial197.stdout: debug mgr = 20 2026-02-26T03:00:50.557 INFO:teuthology.orchestra.run.trial197.stdout: debug mon = 20 2026-02-26T03:00:50.557 INFO:teuthology.orchestra.run.trial197.stdout: debug auth = 20 2026-02-26T03:00:50.557 INFO:teuthology.orchestra.run.trial197.stdout: mon reweight min pgs per osd = 4 2026-02-26T03:00:50.557 INFO:teuthology.orchestra.run.trial197.stdout: mon reweight min bytes per osd = 10 2026-02-26T03:00:50.557 INFO:teuthology.orchestra.run.trial197.stdout: mgr/telemetry/nag = false 2026-02-26T03:00:50.557 INFO:teuthology.orchestra.run.trial197.stdout: 2026-02-26T03:00:50.557 INFO:teuthology.orchestra.run.trial197.stdout:[mon] 2026-02-26T03:00:50.557 INFO:teuthology.orchestra.run.trial197.stdout: debug ms = 1 2026-02-26T03:00:50.557 INFO:teuthology.orchestra.run.trial197.stdout: debug mon = 20 2026-02-26T03:00:50.557 INFO:teuthology.orchestra.run.trial197.stdout: debug paxos = 20 2026-02-26T03:00:50.557 INFO:teuthology.orchestra.run.trial197.stdout: debug auth = 20 2026-02-26T03:00:50.557 INFO:teuthology.orchestra.run.trial197.stdout: mon data avail warn = 5 2026-02-26T03:00:50.557 INFO:teuthology.orchestra.run.trial197.stdout: mon mgr mkfs grace = 240 2026-02-26T03:00:50.557 INFO:teuthology.orchestra.run.trial197.stdout: mon reweight min pgs per osd = 4 2026-02-26T03:00:50.557 INFO:teuthology.orchestra.run.trial197.stdout: mon osd reporter subtree level = osd 2026-02-26T03:00:50.557 INFO:teuthology.orchestra.run.trial197.stdout: mon osd prime pg temp = true 2026-02-26T03:00:50.557 INFO:teuthology.orchestra.run.trial197.stdout: mon reweight min bytes per osd = 10 2026-02-26T03:00:50.557 INFO:teuthology.orchestra.run.trial197.stdout: 2026-02-26T03:00:50.557 INFO:teuthology.orchestra.run.trial197.stdout: # rotate auth tickets quickly to exercise renewal paths 2026-02-26T03:00:50.557 INFO:teuthology.orchestra.run.trial197.stdout: auth mon ticket ttl = 660 # 11m 2026-02-26T03:00:50.557 INFO:teuthology.orchestra.run.trial197.stdout: auth service ticket ttl = 240 # 4m 2026-02-26T03:00:50.557 INFO:teuthology.orchestra.run.trial197.stdout: 2026-02-26T03:00:50.557 INFO:teuthology.orchestra.run.trial197.stdout: # don't complain about insecure global_id in the test suite 2026-02-26T03:00:50.557 INFO:teuthology.orchestra.run.trial197.stdout: mon_warn_on_insecure_global_id_reclaim = false 2026-02-26T03:00:50.558 INFO:teuthology.orchestra.run.trial197.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2026-02-26T03:00:50.558 INFO:teuthology.orchestra.run.trial197.stdout: 2026-02-26T03:00:50.558 INFO:teuthology.orchestra.run.trial197.stdout: # 1m isn't quite enough 2026-02-26T03:00:50.558 INFO:teuthology.orchestra.run.trial197.stdout: mon_down_mkfs_grace = 2m 2026-02-26T03:00:50.558 INFO:teuthology.orchestra.run.trial197.stdout: 2026-02-26T03:00:50.558 INFO:teuthology.orchestra.run.trial197.stdout: mon_warn_on_filestore_osds = false 2026-02-26T03:00:50.558 INFO:teuthology.orchestra.run.trial197.stdout: mon scrub interval = 300 2026-02-26T03:00:50.558 INFO:teuthology.orchestra.run.trial197.stdout: 2026-02-26T03:00:50.558 INFO:teuthology.orchestra.run.trial197.stdout:[client] 2026-02-26T03:00:50.558 INFO:teuthology.orchestra.run.trial197.stdout: rgw cache enabled = true 2026-02-26T03:00:50.558 INFO:teuthology.orchestra.run.trial197.stdout: rgw enable ops log = true 2026-02-26T03:00:50.558 INFO:teuthology.orchestra.run.trial197.stdout: rgw enable usage log = true 2026-02-26T03:00:50.558 INFO:teuthology.orchestra.run.trial197.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2026-02-26T03:00:50.558 INFO:teuthology.orchestra.run.trial197.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2026-02-26T03:00:50.558 INFO:teuthology.orchestra.run.trial197.stdout:[mon.a] 2026-02-26T03:00:50.564 INFO:tasks.ceph:Creating admin key on mon.a... 2026-02-26T03:00:50.564 DEBUG:teuthology.orchestra.run.trial197:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/ceph.keyring 2026-02-26T03:00:50.645 INFO:tasks.ceph:Copying monmap to all nodes... 2026-02-26T03:00:50.645 DEBUG:teuthology.orchestra.run.trial197:> set -ex 2026-02-26T03:00:50.645 DEBUG:teuthology.orchestra.run.trial197:> dd if=/etc/ceph/ceph.keyring of=/dev/stdout 2026-02-26T03:00:50.701 DEBUG:teuthology.orchestra.run.trial197:> set -ex 2026-02-26T03:00:50.701 DEBUG:teuthology.orchestra.run.trial197:> dd if=/home/ubuntu/cephtest/ceph.monmap of=/dev/stdout 2026-02-26T03:00:50.758 INFO:tasks.ceph:Sending monmap to node ubuntu@trial197.front.sepia.ceph.com 2026-02-26T03:00:50.758 DEBUG:teuthology.orchestra.run.trial197:> set -ex 2026-02-26T03:00:50.759 DEBUG:teuthology.orchestra.run.trial197:> sudo dd of=/etc/ceph/ceph.keyring 2026-02-26T03:00:50.759 DEBUG:teuthology.orchestra.run.trial197:> sudo chmod 0644 /etc/ceph/ceph.keyring 2026-02-26T03:00:50.832 DEBUG:teuthology.orchestra.run.trial197:> set -ex 2026-02-26T03:00:50.832 DEBUG:teuthology.orchestra.run.trial197:> dd of=/home/ubuntu/cephtest/ceph.monmap 2026-02-26T03:00:50.887 INFO:tasks.ceph:Setting up mon nodes... 2026-02-26T03:00:50.887 INFO:tasks.ceph:Setting up mgr nodes... 2026-02-26T03:00:50.887 DEBUG:teuthology.orchestra.run.trial197:> sudo mkdir -p /var/lib/ceph/mgr/ceph-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/ceph-x/keyring 2026-02-26T03:00:50.970 INFO:teuthology.orchestra.run.trial197.stdout:creating /var/lib/ceph/mgr/ceph-x/keyring 2026-02-26T03:00:50.972 INFO:tasks.ceph:Setting up mds nodes... 2026-02-26T03:00:50.972 INFO:tasks.ceph_client:Setting up client nodes... 2026-02-26T03:00:50.972 DEBUG:teuthology.orchestra.run.trial197:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/ceph.client.0.keyring && sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2026-02-26T03:00:51.046 INFO:teuthology.orchestra.run.trial197.stdout:creating /etc/ceph/ceph.client.0.keyring 2026-02-26T03:00:51.056 INFO:tasks.ceph:Running mkfs on osd nodes... 2026-02-26T03:00:51.057 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@trial197.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'}} 2026-02-26T03:00:51.057 DEBUG:teuthology.orchestra.run.trial197:> sudo mkdir -p /var/lib/ceph/osd/ceph-0 2026-02-26T03:00:51.121 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2026-02-26T03:00:51.121 INFO:tasks.ceph:role: osd.0 2026-02-26T03:00:51.121 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@trial197.front.sepia.ceph.com 2026-02-26T03:00:51.121 DEBUG:teuthology.orchestra.run.trial197:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2026-02-26T03:00:51.185 INFO:teuthology.orchestra.run.trial197.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=45875200 blks 2026-02-26T03:00:51.186 INFO:teuthology.orchestra.run.trial197.stdout: = sectsz=4096 attr=2, projid32bit=1 2026-02-26T03:00:51.186 INFO:teuthology.orchestra.run.trial197.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2026-02-26T03:00:51.186 INFO:teuthology.orchestra.run.trial197.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2026-02-26T03:00:51.186 INFO:teuthology.orchestra.run.trial197.stdout:data = bsize=4096 blocks=183500800, imaxpct=25 2026-02-26T03:00:51.186 INFO:teuthology.orchestra.run.trial197.stdout: = sunit=0 swidth=0 blks 2026-02-26T03:00:51.186 INFO:teuthology.orchestra.run.trial197.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2026-02-26T03:00:51.186 INFO:teuthology.orchestra.run.trial197.stdout:log =internal log bsize=4096 blocks=89600, version=2 2026-02-26T03:00:51.186 INFO:teuthology.orchestra.run.trial197.stdout: = sectsz=4096 sunit=1 blks, lazy-count=1 2026-02-26T03:00:51.186 INFO:teuthology.orchestra.run.trial197.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2026-02-26T03:00:51.776 INFO:teuthology.orchestra.run.trial197.stdout:Discarding blocks...Done. 2026-02-26T03:00:51.778 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@trial197.front.sepia.ceph.com -o noatime 2026-02-26T03:00:51.778 DEBUG:teuthology.orchestra.run.trial197:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-0 2026-02-26T03:00:51.845 DEBUG:teuthology.orchestra.run.trial197:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-0 2026-02-26T03:00:51.912 DEBUG:teuthology.orchestra.run.trial197:> sudo mkdir -p /var/lib/ceph/osd/ceph-1 2026-02-26T03:00:51.975 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2026-02-26T03:00:51.975 INFO:tasks.ceph:role: osd.1 2026-02-26T03:00:51.975 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@trial197.front.sepia.ceph.com 2026-02-26T03:00:51.975 DEBUG:teuthology.orchestra.run.trial197:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2026-02-26T03:00:52.040 INFO:teuthology.orchestra.run.trial197.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=45875200 blks 2026-02-26T03:00:52.041 INFO:teuthology.orchestra.run.trial197.stdout: = sectsz=4096 attr=2, projid32bit=1 2026-02-26T03:00:52.041 INFO:teuthology.orchestra.run.trial197.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2026-02-26T03:00:52.041 INFO:teuthology.orchestra.run.trial197.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2026-02-26T03:00:52.041 INFO:teuthology.orchestra.run.trial197.stdout:data = bsize=4096 blocks=183500800, imaxpct=25 2026-02-26T03:00:52.041 INFO:teuthology.orchestra.run.trial197.stdout: = sunit=0 swidth=0 blks 2026-02-26T03:00:52.041 INFO:teuthology.orchestra.run.trial197.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2026-02-26T03:00:52.041 INFO:teuthology.orchestra.run.trial197.stdout:log =internal log bsize=4096 blocks=89600, version=2 2026-02-26T03:00:52.041 INFO:teuthology.orchestra.run.trial197.stdout: = sectsz=4096 sunit=1 blks, lazy-count=1 2026-02-26T03:00:52.041 INFO:teuthology.orchestra.run.trial197.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2026-02-26T03:00:52.602 INFO:teuthology.orchestra.run.trial197.stdout:Discarding blocks...Done. 2026-02-26T03:00:52.604 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@trial197.front.sepia.ceph.com -o noatime 2026-02-26T03:00:52.604 DEBUG:teuthology.orchestra.run.trial197:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-1 2026-02-26T03:00:52.674 DEBUG:teuthology.orchestra.run.trial197:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-1 2026-02-26T03:00:52.742 DEBUG:teuthology.orchestra.run.trial197:> sudo mkdir -p /var/lib/ceph/osd/ceph-2 2026-02-26T03:00:52.805 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2026-02-26T03:00:52.805 INFO:tasks.ceph:role: osd.2 2026-02-26T03:00:52.806 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@trial197.front.sepia.ceph.com 2026-02-26T03:00:52.806 DEBUG:teuthology.orchestra.run.trial197:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2026-02-26T03:00:52.870 INFO:teuthology.orchestra.run.trial197.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=45875200 blks 2026-02-26T03:00:52.870 INFO:teuthology.orchestra.run.trial197.stdout: = sectsz=4096 attr=2, projid32bit=1 2026-02-26T03:00:52.871 INFO:teuthology.orchestra.run.trial197.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2026-02-26T03:00:52.871 INFO:teuthology.orchestra.run.trial197.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2026-02-26T03:00:52.871 INFO:teuthology.orchestra.run.trial197.stdout:data = bsize=4096 blocks=183500800, imaxpct=25 2026-02-26T03:00:52.871 INFO:teuthology.orchestra.run.trial197.stdout: = sunit=0 swidth=0 blks 2026-02-26T03:00:52.871 INFO:teuthology.orchestra.run.trial197.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2026-02-26T03:00:52.871 INFO:teuthology.orchestra.run.trial197.stdout:log =internal log bsize=4096 blocks=89600, version=2 2026-02-26T03:00:52.871 INFO:teuthology.orchestra.run.trial197.stdout: = sectsz=4096 sunit=1 blks, lazy-count=1 2026-02-26T03:00:52.871 INFO:teuthology.orchestra.run.trial197.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2026-02-26T03:00:53.458 INFO:teuthology.orchestra.run.trial197.stdout:Discarding blocks...Done. 2026-02-26T03:00:53.460 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@trial197.front.sepia.ceph.com -o noatime 2026-02-26T03:00:53.460 DEBUG:teuthology.orchestra.run.trial197:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-2 2026-02-26T03:00:53.529 DEBUG:teuthology.orchestra.run.trial197:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-2 2026-02-26T03:00:53.597 DEBUG:teuthology.orchestra.run.trial197:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/ceph.monmap 2026-02-26T03:00:53.675 INFO:teuthology.orchestra.run.trial197.stderr:2026-02-26T03:00:53.673+0000 7eff55751780 -1 auth: error reading file: /var/lib/ceph/osd/ceph-0/keyring: can't open /var/lib/ceph/osd/ceph-0/keyring: (2) No such file or directory 2026-02-26T03:00:53.676 INFO:teuthology.orchestra.run.trial197.stderr:2026-02-26T03:00:53.673+0000 7eff55751780 -1 created new key in keyring /var/lib/ceph/osd/ceph-0/keyring 2026-02-26T03:00:53.676 INFO:teuthology.orchestra.run.trial197.stderr:2026-02-26T03:00:53.673+0000 7eff55751780 -1 bdev(0x564550b11800 /var/lib/ceph/osd/ceph-0/block) open stat got: (1) Operation not permitted 2026-02-26T03:00:53.676 INFO:teuthology.orchestra.run.trial197.stderr:2026-02-26T03:00:53.673+0000 7eff55751780 -1 bluestore(/var/lib/ceph/osd/ceph-0) _read_fsid unparsable uuid 2026-02-26T03:00:56.837 DEBUG:teuthology.orchestra.run.trial197:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2026-02-26T03:00:56.903 DEBUG:teuthology.orchestra.run.trial197:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/ceph.monmap 2026-02-26T03:00:56.981 INFO:teuthology.orchestra.run.trial197.stderr:2026-02-26T03:00:56.979+0000 7f7fc3c0c780 -1 auth: error reading file: /var/lib/ceph/osd/ceph-1/keyring: can't open /var/lib/ceph/osd/ceph-1/keyring: (2) No such file or directory 2026-02-26T03:00:56.982 INFO:teuthology.orchestra.run.trial197.stderr:2026-02-26T03:00:56.979+0000 7f7fc3c0c780 -1 created new key in keyring /var/lib/ceph/osd/ceph-1/keyring 2026-02-26T03:00:56.982 INFO:teuthology.orchestra.run.trial197.stderr:2026-02-26T03:00:56.979+0000 7f7fc3c0c780 -1 bdev(0x55f0097e7800 /var/lib/ceph/osd/ceph-1/block) open stat got: (1) Operation not permitted 2026-02-26T03:00:56.982 INFO:teuthology.orchestra.run.trial197.stderr:2026-02-26T03:00:56.979+0000 7f7fc3c0c780 -1 bluestore(/var/lib/ceph/osd/ceph-1) _read_fsid unparsable uuid 2026-02-26T03:00:59.892 DEBUG:teuthology.orchestra.run.trial197:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2026-02-26T03:00:59.958 DEBUG:teuthology.orchestra.run.trial197:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/ceph.monmap 2026-02-26T03:01:00.037 INFO:teuthology.orchestra.run.trial197.stderr:2026-02-26T03:01:00.034+0000 7fb2a060c780 -1 auth: error reading file: /var/lib/ceph/osd/ceph-2/keyring: can't open /var/lib/ceph/osd/ceph-2/keyring: (2) No such file or directory 2026-02-26T03:01:00.037 INFO:teuthology.orchestra.run.trial197.stderr:2026-02-26T03:01:00.035+0000 7fb2a060c780 -1 created new key in keyring /var/lib/ceph/osd/ceph-2/keyring 2026-02-26T03:01:00.037 INFO:teuthology.orchestra.run.trial197.stderr:2026-02-26T03:01:00.035+0000 7fb2a060c780 -1 bdev(0x55f2741cd800 /var/lib/ceph/osd/ceph-2/block) open stat got: (1) Operation not permitted 2026-02-26T03:01:00.037 INFO:teuthology.orchestra.run.trial197.stderr:2026-02-26T03:01:00.035+0000 7fb2a060c780 -1 bluestore(/var/lib/ceph/osd/ceph-2) _read_fsid unparsable uuid 2026-02-26T03:01:02.968 DEBUG:teuthology.orchestra.run.trial197:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2026-02-26T03:01:03.034 INFO:tasks.ceph:Reading keys from all nodes... 2026-02-26T03:01:03.034 DEBUG:teuthology.orchestra.run.trial197:> set -ex 2026-02-26T03:01:03.034 DEBUG:teuthology.orchestra.run.trial197:> sudo dd if=/var/lib/ceph/mgr/ceph-x/keyring of=/dev/stdout 2026-02-26T03:01:03.098 DEBUG:teuthology.orchestra.run.trial197:> set -ex 2026-02-26T03:01:03.099 DEBUG:teuthology.orchestra.run.trial197:> sudo dd if=/var/lib/ceph/osd/ceph-0/keyring of=/dev/stdout 2026-02-26T03:01:03.163 DEBUG:teuthology.orchestra.run.trial197:> set -ex 2026-02-26T03:01:03.163 DEBUG:teuthology.orchestra.run.trial197:> sudo dd if=/var/lib/ceph/osd/ceph-1/keyring of=/dev/stdout 2026-02-26T03:01:03.228 DEBUG:teuthology.orchestra.run.trial197:> set -ex 2026-02-26T03:01:03.228 DEBUG:teuthology.orchestra.run.trial197:> sudo dd if=/var/lib/ceph/osd/ceph-2/keyring of=/dev/stdout 2026-02-26T03:01:03.291 DEBUG:teuthology.orchestra.run.trial197:> set -ex 2026-02-26T03:01:03.291 DEBUG:teuthology.orchestra.run.trial197:> dd if=/etc/ceph/ceph.client.0.keyring of=/dev/stdout 2026-02-26T03:01:03.346 INFO:tasks.ceph:Adding keys to all mons... 2026-02-26T03:01:03.346 DEBUG:teuthology.orchestra.run.trial197:> sudo tee -a /etc/ceph/ceph.keyring 2026-02-26T03:01:03.409 INFO:teuthology.orchestra.run.trial197.stdout:[mgr.x] 2026-02-26T03:01:03.409 INFO:teuthology.orchestra.run.trial197.stdout: key = AQBit59p3MTKORAAHLrczJfxFJMwMloWQx9Kxw== 2026-02-26T03:01:03.409 INFO:teuthology.orchestra.run.trial197.stdout:[osd.0] 2026-02-26T03:01:03.409 INFO:teuthology.orchestra.run.trial197.stdout: key = AQBlt59pDew0KBAAYIJ7NDzmH7TuDyLQnyZWow== 2026-02-26T03:01:03.409 INFO:teuthology.orchestra.run.trial197.stdout:[osd.1] 2026-02-26T03:01:03.410 INFO:teuthology.orchestra.run.trial197.stdout: key = AQBot59pp8dsOhAA4qdXYO2SkQO90Gvd7xJ3FQ== 2026-02-26T03:01:03.410 INFO:teuthology.orchestra.run.trial197.stdout:[osd.2] 2026-02-26T03:01:03.410 INFO:teuthology.orchestra.run.trial197.stdout: key = AQBst59pEcMnAhAAh+8eMirdrCjnPDJaVDBevw== 2026-02-26T03:01:03.410 INFO:teuthology.orchestra.run.trial197.stdout:[client.0] 2026-02-26T03:01:03.410 INFO:teuthology.orchestra.run.trial197.stdout: key = AQBjt59pftO0AhAApjgpyb+dGyJQPiKP2P/2aQ== 2026-02-26T03:01:03.411 DEBUG:teuthology.orchestra.run.trial197:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2026-02-26T03:01:03.490 DEBUG:teuthology.orchestra.run.trial197:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2026-02-26T03:01:03.569 DEBUG:teuthology.orchestra.run.trial197:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2026-02-26T03:01:03.649 DEBUG:teuthology.orchestra.run.trial197:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2026-02-26T03:01:03.727 DEBUG:teuthology.orchestra.run.trial197:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2026-02-26T03:01:03.808 INFO:tasks.ceph:Running mkfs on mon nodes... 2026-02-26T03:01:03.808 DEBUG:teuthology.orchestra.run.trial197:> sudo mkdir -p /var/lib/ceph/mon/ceph-a 2026-02-26T03:01:03.872 DEBUG:teuthology.orchestra.run.trial197:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i a --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2026-02-26T03:01:03.960 DEBUG:teuthology.orchestra.run.trial197:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-a 2026-02-26T03:01:03.985 DEBUG:teuthology.orchestra.run.trial197:> rm -- /home/ubuntu/cephtest/ceph.monmap 2026-02-26T03:01:04.040 INFO:tasks.ceph:Starting mon daemons in cluster ceph... 2026-02-26T03:01:04.041 INFO:tasks.ceph.mon.a:Restarting daemon 2026-02-26T03:01:04.041 DEBUG:teuthology.orchestra.run.trial197:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2026-02-26T03:01:04.083 INFO:tasks.ceph.mon.a:Started 2026-02-26T03:01:04.083 DEBUG:teuthology.orchestra.run.trial197:> sudo ceph config set mgr mgr_pool false --force 2026-02-26T03:01:04.191 INFO:tasks.ceph:Starting mgr daemons in cluster ceph... 2026-02-26T03:01:04.191 INFO:tasks.ceph.mgr.x:Restarting daemon 2026-02-26T03:01:04.192 DEBUG:teuthology.orchestra.run.trial197:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i x 2026-02-26T03:01:04.194 INFO:tasks.ceph.mgr.x:Started 2026-02-26T03:01:04.194 INFO:tasks.ceph:Setting crush tunables to default 2026-02-26T03:01:04.194 DEBUG:teuthology.orchestra.run.trial197:> sudo ceph --cluster ceph osd crush tunables default 2026-02-26T03:01:04.292 INFO:teuthology.orchestra.run.trial197.stderr:adjusted tunables profile to default 2026-02-26T03:01:04.298 INFO:tasks.ceph:check_enable_crimson: False 2026-02-26T03:01:04.298 INFO:tasks.ceph:Starting osd daemons in cluster ceph... 2026-02-26T03:01:04.299 DEBUG:teuthology.orchestra.run.trial197:> set -ex 2026-02-26T03:01:04.299 DEBUG:teuthology.orchestra.run.trial197:> sudo dd if=/var/lib/ceph/osd/ceph-0/fsid of=/dev/stdout 2026-02-26T03:01:04.323 DEBUG:teuthology.orchestra.run.trial197:> set -ex 2026-02-26T03:01:04.323 DEBUG:teuthology.orchestra.run.trial197:> sudo dd if=/var/lib/ceph/osd/ceph-1/fsid of=/dev/stdout 2026-02-26T03:01:04.386 DEBUG:teuthology.orchestra.run.trial197:> set -ex 2026-02-26T03:01:04.386 DEBUG:teuthology.orchestra.run.trial197:> sudo dd if=/var/lib/ceph/osd/ceph-2/fsid of=/dev/stdout 2026-02-26T03:01:04.449 DEBUG:teuthology.orchestra.run.trial197:> sudo ceph --cluster ceph osd new 696e32c7-b2ad-41e8-9c32-f7e12eb617ee 0 2026-02-26T03:01:04.591 INFO:teuthology.orchestra.run.trial197.stdout:0 2026-02-26T03:01:04.599 DEBUG:teuthology.orchestra.run.trial197:> sudo ceph --cluster ceph osd new 606c6d1b-0452-4860-a1dc-ca58763aa4e9 1 2026-02-26T03:01:04.602 INFO:tasks.ceph.mgr.x.trial197.stderr:2026-02-26T03:01:04.600+0000 7f6019a1c140 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2026-02-26T03:01:04.703 INFO:teuthology.orchestra.run.trial197.stdout:1 2026-02-26T03:01:04.711 DEBUG:teuthology.orchestra.run.trial197:> sudo ceph --cluster ceph osd new 4a54c53b-ac96-4483-97dc-bf76ae085e1b 2 2026-02-26T03:01:04.724 INFO:tasks.ceph.mgr.x.trial197.stderr:2026-02-26T03:01:04.722+0000 7f6019a1c140 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2026-02-26T03:01:04.752 INFO:tasks.ceph.mgr.x.trial197.stderr:2026-02-26T03:01:04.750+0000 7f6019a1c140 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2026-02-26T03:01:04.784 INFO:tasks.ceph.mgr.x.trial197.stderr:2026-02-26T03:01:04.783+0000 7f6019a1c140 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2026-02-26T03:01:04.808 INFO:tasks.ceph.mgr.x.trial197.stderr:2026-02-26T03:01:04.807+0000 7f6019a1c140 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2026-02-26T03:01:04.815 INFO:teuthology.orchestra.run.trial197.stdout:2 2026-02-26T03:01:04.824 INFO:tasks.ceph.osd.0:Restarting daemon 2026-02-26T03:01:04.824 DEBUG:teuthology.orchestra.run.trial197:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 0 2026-02-26T03:01:04.835 INFO:tasks.ceph.mgr.x.trial197.stderr:2026-02-26T03:01:04.833+0000 7f6019a1c140 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2026-02-26T03:01:04.837 INFO:tasks.ceph.osd.0:Started 2026-02-26T03:01:04.837 INFO:tasks.ceph.osd.1:Restarting daemon 2026-02-26T03:01:04.837 DEBUG:teuthology.orchestra.run.trial197:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 1 2026-02-26T03:01:04.839 INFO:tasks.ceph.osd.1:Started 2026-02-26T03:01:04.839 INFO:tasks.ceph.osd.2:Restarting daemon 2026-02-26T03:01:04.839 DEBUG:teuthology.orchestra.run.trial197:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 2 2026-02-26T03:01:04.841 INFO:tasks.ceph.osd.2:Started 2026-02-26T03:01:04.841 DEBUG:teuthology.orchestra.run.trial197:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2026-02-26T03:01:04.895 INFO:tasks.ceph.mgr.x.trial197.stderr:2026-02-26T03:01:04.894+0000 7f6019a1c140 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2026-02-26T03:01:04.942 INFO:teuthology.orchestra.run.trial197.stdout: 2026-02-26T03:01:04.942 INFO:teuthology.orchestra.run.trial197.stdout:{"epoch":5,"fsid":"658087b3-2758-487d-ba7c-f8b171a8f312","created":"2026-02-26T03:01:04.144741+0000","modified":"2026-02-26T03:01:04.814150+0000","last_up_change":"0.000000","last_in_change":"2026-02-26T03:01:04.814150+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"696e32c7-b2ad-41e8-9c32-f7e12eb617ee","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"606c6d1b-0452-4860-a1dc-ca58763aa4e9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"4a54c53b-ac96-4483-97dc-bf76ae085e1b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-26T03:01:04.949 INFO:tasks.ceph.ceph_manager.ceph:[] 2026-02-26T03:01:04.949 INFO:tasks.ceph:Waiting for OSDs to come up 2026-02-26T03:01:04.958 INFO:tasks.ceph.mgr.x.trial197.stderr:/usr/lib64/python3.9/site-packages/scipy/__init__.py:73: UserWarning: NumPy was imported from a Python sub-interpreter but NumPy does not properly support sub-interpreters. This will likely work for most users but might cause hard to track down issues or subtle bugs. A common user of the rare sub-interpreter feature is wsgi which also allows single-interpreter mode. 2026-02-26T03:01:04.958 INFO:tasks.ceph.mgr.x.trial197.stderr:Improvements in the case of bugs are welcome, but is not on the NumPy roadmap, and full support may require significant effort to achieve. 2026-02-26T03:01:04.959 INFO:tasks.ceph.mgr.x.trial197.stderr: from numpy import show_config as show_numpy_config 2026-02-26T03:01:04.962 INFO:tasks.ceph.mgr.x.trial197.stderr:2026-02-26T03:01:04.960+0000 7f6019a1c140 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2026-02-26T03:01:04.988 INFO:tasks.ceph.mgr.x.trial197.stderr:2026-02-26T03:01:04.986+0000 7f6019a1c140 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2026-02-26T03:01:05.044 INFO:tasks.ceph.mgr.x.trial197.stderr:2026-02-26T03:01:05.042+0000 7f6019a1c140 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2026-02-26T03:01:05.087 INFO:tasks.ceph.mgr.x.trial197.stderr:2026-02-26T03:01:05.085+0000 7f6019a1c140 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2026-02-26T03:01:05.113 INFO:tasks.ceph.mgr.x.trial197.stderr:2026-02-26T03:01:05.111+0000 7f6019a1c140 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2026-02-26T03:01:05.137 INFO:tasks.ceph.mgr.x.trial197.stderr:2026-02-26T03:01:05.135+0000 7f6019a1c140 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2026-02-26T03:01:05.163 INFO:tasks.ceph.mgr.x.trial197.stderr:2026-02-26T03:01:05.161+0000 7f6019a1c140 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2026-02-26T03:01:05.186 INFO:tasks.ceph.mgr.x.trial197.stderr:2026-02-26T03:01:05.184+0000 7f6019a1c140 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2026-02-26T03:01:05.251 DEBUG:teuthology.orchestra.run.trial197:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2026-02-26T03:01:05.345 INFO:teuthology.misc.health.trial197.stdout: 2026-02-26T03:01:05.345 INFO:teuthology.misc.health.trial197.stdout:{"epoch":5,"fsid":"658087b3-2758-487d-ba7c-f8b171a8f312","created":"2026-02-26T03:01:04.144741+0000","modified":"2026-02-26T03:01:04.814150+0000","last_up_change":"0.000000","last_in_change":"2026-02-26T03:01:04.814150+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"696e32c7-b2ad-41e8-9c32-f7e12eb617ee","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"606c6d1b-0452-4860-a1dc-ca58763aa4e9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"4a54c53b-ac96-4483-97dc-bf76ae085e1b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-26T03:01:05.350 DEBUG:teuthology.misc:0 of 3 OSDs are up 2026-02-26T03:01:05.383 INFO:tasks.ceph.mgr.x.trial197.stderr:2026-02-26T03:01:05.382+0000 7f6019a1c140 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2026-02-26T03:01:05.438 INFO:tasks.ceph.mgr.x.trial197.stderr:2026-02-26T03:01:05.437+0000 7f6019a1c140 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2026-02-26T03:01:05.461 INFO:tasks.ceph.mgr.x.trial197.stderr:2026-02-26T03:01:05.459+0000 7f6019a1c140 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2026-02-26T03:01:05.484 INFO:tasks.ceph.mgr.x.trial197.stderr:2026-02-26T03:01:05.482+0000 7f6019a1c140 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2026-02-26T03:01:05.537 INFO:tasks.ceph.mgr.x.trial197.stderr:2026-02-26T03:01:05.535+0000 7f6019a1c140 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2026-02-26T03:01:05.558 INFO:tasks.ceph.mgr.x.trial197.stderr:2026-02-26T03:01:05.556+0000 7f6019a1c140 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2026-02-26T03:01:05.609 INFO:tasks.ceph.mgr.x.trial197.stderr:2026-02-26T03:01:05.608+0000 7f6019a1c140 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2026-02-26T03:01:06.010 INFO:tasks.ceph.mgr.x.trial197.stderr:2026-02-26T03:01:06.008+0000 7f6019a1c140 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2026-02-26T03:01:06.032 INFO:tasks.ceph.mgr.x.trial197.stderr:2026-02-26T03:01:06.031+0000 7f6019a1c140 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2026-02-26T03:01:06.303 INFO:tasks.ceph.osd.0.trial197.stderr:2026-02-26T03:01:06.301+0000 7fe2caa1a780 -1 Falling back to public interface 2026-02-26T03:01:06.375 INFO:tasks.ceph.mgr.x.trial197.stderr:2026-02-26T03:01:06.374+0000 7f6019a1c140 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2026-02-26T03:01:06.534 INFO:tasks.ceph.osd.2.trial197.stderr:2026-02-26T03:01:06.532+0000 7f1b59d12780 -1 Falling back to public interface 2026-02-26T03:01:06.810 INFO:tasks.ceph.osd.1.trial197.stderr:2026-02-26T03:01:06.808+0000 7fa6c459c780 -1 Falling back to public interface 2026-02-26T03:01:07.380 INFO:tasks.ceph.mgr.x.trial197.stderr:2026-02-26T03:01:07.378+0000 7f5f371a0640 -1 mgr.server handle_report got status from non-daemon mon.a 2026-02-26T03:01:10.270 INFO:tasks.ceph.osd.2.trial197.stderr:2026-02-26T03:01:10.269+0000 7f1b59d12780 -1 osd.2 0 log_to_monitors true 2026-02-26T03:01:10.317 INFO:tasks.ceph.osd.0.trial197.stderr:2026-02-26T03:01:10.316+0000 7fe2caa1a780 -1 osd.0 0 log_to_monitors true 2026-02-26T03:01:10.613 INFO:tasks.ceph.osd.1.trial197.stderr:2026-02-26T03:01:10.611+0000 7fa6c459c780 -1 osd.1 0 log_to_monitors true 2026-02-26T03:01:11.400 INFO:tasks.ceph.osd.0.trial197.stderr:2026-02-26T03:01:11.399+0000 7fe2c63fc640 -1 osd.0 0 waiting for initial osdmap 2026-02-26T03:01:11.400 INFO:tasks.ceph.osd.2.trial197.stderr:2026-02-26T03:01:11.399+0000 7f1b55c9f640 -1 osd.2 0 waiting for initial osdmap 2026-02-26T03:01:11.402 INFO:tasks.ceph.osd.2.trial197.stderr:2026-02-26T03:01:11.401+0000 7f1b51ac9640 -1 osd.2 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2026-02-26T03:01:11.402 INFO:tasks.ceph.osd.0.trial197.stderr:2026-02-26T03:01:11.401+0000 7fe2c2226640 -1 osd.0 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2026-02-26T03:01:11.652 DEBUG:teuthology.orchestra.run.trial197:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2026-02-26T03:01:11.804 INFO:teuthology.misc.health.trial197.stdout: 2026-02-26T03:01:11.804 INFO:teuthology.misc.health.trial197.stdout:{"epoch":7,"fsid":"658087b3-2758-487d-ba7c-f8b171a8f312","created":"2026-02-26T03:01:04.144741+0000","modified":"2026-02-26T03:01:11.382744+0000","last_up_change":"0.000000","last_in_change":"2026-02-26T03:01:04.814150+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"696e32c7-b2ad-41e8-9c32-f7e12eb617ee","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"606c6d1b-0452-4860-a1dc-ca58763aa4e9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"4a54c53b-ac96-4483-97dc-bf76ae085e1b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-26T03:01:11.811 DEBUG:teuthology.misc:0 of 3 OSDs are up 2026-02-26T03:01:12.657 INFO:tasks.ceph.osd.1.trial197.stderr:2026-02-26T03:01:12.655+0000 7fa6c052b640 -1 osd.1 0 waiting for initial osdmap 2026-02-26T03:01:12.666 INFO:tasks.ceph.osd.1.trial197.stderr:2026-02-26T03:01:12.665+0000 7fa6b834d640 -1 osd.1 8 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2026-02-26T03:01:18.113 DEBUG:teuthology.orchestra.run.trial197:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2026-02-26T03:01:18.266 INFO:teuthology.misc.health.trial197.stdout: 2026-02-26T03:01:18.266 INFO:teuthology.misc.health.trial197.stdout:{"epoch":9,"fsid":"658087b3-2758-487d-ba7c-f8b171a8f312","created":"2026-02-26T03:01:04.144741+0000","modified":"2026-02-26T03:01:13.657148+0000","last_up_change":"2026-02-26T03:01:13.657148+0000","last_in_change":"2026-02-26T03:01:04.814150+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"696e32c7-b2ad-41e8-9c32-f7e12eb617ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.197:6800","nonce":176999330},{"type":"v1","addr":"10.20.193.197:6801","nonce":176999330}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.197:6802","nonce":176999330},{"type":"v1","addr":"10.20.193.197:6803","nonce":176999330}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.197:6806","nonce":176999330},{"type":"v1","addr":"10.20.193.197:6807","nonce":176999330}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.197:6804","nonce":176999330},{"type":"v1","addr":"10.20.193.197:6805","nonce":176999330}]},"public_addr":"10.20.193.197:6801/176999330","cluster_addr":"10.20.193.197:6803/176999330","heartbeat_back_addr":"10.20.193.197:6807/176999330","heartbeat_front_addr":"10.20.193.197:6805/176999330","state":["exists","up"]},{"osd":1,"uuid":"606c6d1b-0452-4860-a1dc-ca58763aa4e9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.197:6818","nonce":4254931048},{"type":"v1","addr":"10.20.193.197:6819","nonce":4254931048}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.197:6820","nonce":4254931048},{"type":"v1","addr":"10.20.193.197:6821","nonce":4254931048}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.197:6824","nonce":4254931048},{"type":"v1","addr":"10.20.193.197:6825","nonce":4254931048}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.197:6822","nonce":4254931048},{"type":"v1","addr":"10.20.193.197:6823","nonce":4254931048}]},"public_addr":"10.20.193.197:6819/4254931048","cluster_addr":"10.20.193.197:6821/4254931048","heartbeat_back_addr":"10.20.193.197:6825/4254931048","heartbeat_front_addr":"10.20.193.197:6823/4254931048","state":["exists","up"]},{"osd":2,"uuid":"4a54c53b-ac96-4483-97dc-bf76ae085e1b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.197:6810","nonce":2617788373},{"type":"v1","addr":"10.20.193.197:6811","nonce":2617788373}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.197:6812","nonce":2617788373},{"type":"v1","addr":"10.20.193.197:6813","nonce":2617788373}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.197:6816","nonce":2617788373},{"type":"v1","addr":"10.20.193.197:6817","nonce":2617788373}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.197:6814","nonce":2617788373},{"type":"v1","addr":"10.20.193.197:6815","nonce":2617788373}]},"public_addr":"10.20.193.197:6811/2617788373","cluster_addr":"10.20.193.197:6813/2617788373","heartbeat_back_addr":"10.20.193.197:6817/2617788373","heartbeat_front_addr":"10.20.193.197:6815/2617788373","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-26T03:01:18.273 DEBUG:teuthology.misc:3 of 3 OSDs are up 2026-02-26T03:01:18.273 INFO:tasks.ceph:Creating RBD pool 2026-02-26T03:01:18.273 DEBUG:teuthology.orchestra.run.trial197:> sudo ceph --cluster ceph osd pool create rbd 8 2026-02-26T03:01:18.663 INFO:teuthology.orchestra.run.trial197.stderr:pool 'rbd' created 2026-02-26T03:01:18.674 DEBUG:teuthology.orchestra.run.trial197:> rbd --cluster ceph pool init rbd 2026-02-26T03:01:21.676 INFO:tasks.ceph:Starting mds daemons in cluster ceph... 2026-02-26T03:01:21.677 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2026-02-26T03:01:21.677 INFO:tasks.ceph.ceph_manager.ceph:waiting for mgr available 2026-02-26T03:01:21.678 DEBUG:teuthology.orchestra.run.trial197:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr dump --format=json 2026-02-26T03:01:21.678 INFO:tasks.daemonwatchdog.daemon_watchdog:watchdog starting 2026-02-26T03:01:21.856 INFO:teuthology.orchestra.run.trial197.stdout: 2026-02-26T03:01:21.863 INFO:teuthology.orchestra.run.trial197.stdout:{"epoch":4,"flags":0,"active_gid":4101,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.197:6808","nonce":271184291},{"type":"v1","addr":"10.20.193.197:6809","nonce":271184291}]},"active_addr":"10.20.193.197:6809/271184291","active_change":"2026-02-26T03:01:06.376244+0000","active_mgr_features":4540701547738038271,"available":true,"standbys":[],"modules":["iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"update_pg_upmap_activity":{"name":"update_pg_upmap_activity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Updates pg_upmap activity stats to be used in `balancer status detail`","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:10.4.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/loki:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.7.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.2.5","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.51.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/promtail:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba":{"name":"container_image_samba","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-server:devbuilds-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"quay.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"location of dashboards to include in grafana deployments","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"ssh_keepalive_count_max":{"name":"ssh_keepalive_count_max","type":"int","level":"advanced","flags":0,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"How many times ssh connections can fail liveness checks before the host is marked offline","long_desc":"","tags":[],"see_also":[]},"ssh_keepalive_interval":{"name":"ssh_keepalive_interval","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"How often ssh connections are checked for liveness","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":true,"error_string":"","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_requests":{"name":"max_requests","type":"int","level":"advanced","flags":0,"default_value":"500","min":"","max":"","enum_allowed":[],"desc":"Maximum number of requests to keep in memory. When new request comes in, the oldest request will be removed if the number of requests exceeds the max request number. if un-finished request is removed, error message will be logged in the ceph-mgr log.","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"secondary_zone_period_retry_limit":{"name":"secondary_zone_period_retry_limit","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"RGW module period update retry limit for secondary site","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"squid":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"force_disabled_modules":{},"last_failure_osd_epoch":0,"active_clients":[{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"10.20.193.197:0","nonce":2369983347}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"10.20.193.197:0","nonce":3191281977}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"10.20.193.197:0","nonce":2249736693}]}]} 2026-02-26T03:01:21.864 INFO:tasks.ceph.ceph_manager.ceph:mgr available! 2026-02-26T03:01:21.864 INFO:tasks.ceph.ceph_manager.ceph:waiting for all up 2026-02-26T03:01:21.865 DEBUG:teuthology.orchestra.run.trial197:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2026-02-26T03:01:22.021 INFO:teuthology.orchestra.run.trial197.stdout: 2026-02-26T03:01:22.021 INFO:teuthology.orchestra.run.trial197.stdout:{"epoch":13,"fsid":"658087b3-2758-487d-ba7c-f8b171a8f312","created":"2026-02-26T03:01:04.144741+0000","modified":"2026-02-26T03:01:21.666620+0000","last_up_change":"2026-02-26T03:01:13.657148+0000","last_in_change":"2026-02-26T03:01:04.814150+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":"rbd","create_time":"2026-02-26T03:01:18.430774+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"13","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":2,"snap_epoch":13,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"696e32c7-b2ad-41e8-9c32-f7e12eb617ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":10,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.197:6800","nonce":176999330},{"type":"v1","addr":"10.20.193.197:6801","nonce":176999330}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.197:6802","nonce":176999330},{"type":"v1","addr":"10.20.193.197:6803","nonce":176999330}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.197:6806","nonce":176999330},{"type":"v1","addr":"10.20.193.197:6807","nonce":176999330}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.197:6804","nonce":176999330},{"type":"v1","addr":"10.20.193.197:6805","nonce":176999330}]},"public_addr":"10.20.193.197:6801/176999330","cluster_addr":"10.20.193.197:6803/176999330","heartbeat_back_addr":"10.20.193.197:6807/176999330","heartbeat_front_addr":"10.20.193.197:6805/176999330","state":["exists","up"]},{"osd":1,"uuid":"606c6d1b-0452-4860-a1dc-ca58763aa4e9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":10,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.197:6818","nonce":4254931048},{"type":"v1","addr":"10.20.193.197:6819","nonce":4254931048}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.197:6820","nonce":4254931048},{"type":"v1","addr":"10.20.193.197:6821","nonce":4254931048}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.197:6824","nonce":4254931048},{"type":"v1","addr":"10.20.193.197:6825","nonce":4254931048}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.197:6822","nonce":4254931048},{"type":"v1","addr":"10.20.193.197:6823","nonce":4254931048}]},"public_addr":"10.20.193.197:6819/4254931048","cluster_addr":"10.20.193.197:6821/4254931048","heartbeat_back_addr":"10.20.193.197:6825/4254931048","heartbeat_front_addr":"10.20.193.197:6823/4254931048","state":["exists","up"]},{"osd":2,"uuid":"4a54c53b-ac96-4483-97dc-bf76ae085e1b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":10,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.197:6810","nonce":2617788373},{"type":"v1","addr":"10.20.193.197:6811","nonce":2617788373}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.197:6812","nonce":2617788373},{"type":"v1","addr":"10.20.193.197:6813","nonce":2617788373}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.197:6816","nonce":2617788373},{"type":"v1","addr":"10.20.193.197:6817","nonce":2617788373}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.197:6814","nonce":2617788373},{"type":"v1","addr":"10.20.193.197:6815","nonce":2617788373}]},"public_addr":"10.20.193.197:6811/2617788373","cluster_addr":"10.20.193.197:6813/2617788373","heartbeat_back_addr":"10.20.193.197:6817/2617788373","heartbeat_front_addr":"10.20.193.197:6815/2617788373","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2026-02-26T03:01:11.283523+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2026-02-26T03:01:11.607888+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2026-02-26T03:01:11.277139+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":1,"snaps":[{"begin":2,"length":1}]}],"new_removed_snaps":[{"pool":1,"snaps":[{"begin":2,"length":1}]}],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-26T03:01:22.029 INFO:tasks.ceph.ceph_manager.ceph:all up! 2026-02-26T03:01:22.029 DEBUG:teuthology.orchestra.run.trial197:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2026-02-26T03:01:22.190 INFO:teuthology.orchestra.run.trial197.stdout: 2026-02-26T03:01:22.191 INFO:teuthology.orchestra.run.trial197.stdout:{"epoch":13,"fsid":"658087b3-2758-487d-ba7c-f8b171a8f312","created":"2026-02-26T03:01:04.144741+0000","modified":"2026-02-26T03:01:21.666620+0000","last_up_change":"2026-02-26T03:01:13.657148+0000","last_in_change":"2026-02-26T03:01:04.814150+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":"rbd","create_time":"2026-02-26T03:01:18.430774+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"13","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":2,"snap_epoch":13,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"696e32c7-b2ad-41e8-9c32-f7e12eb617ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":10,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.197:6800","nonce":176999330},{"type":"v1","addr":"10.20.193.197:6801","nonce":176999330}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.197:6802","nonce":176999330},{"type":"v1","addr":"10.20.193.197:6803","nonce":176999330}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.197:6806","nonce":176999330},{"type":"v1","addr":"10.20.193.197:6807","nonce":176999330}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.197:6804","nonce":176999330},{"type":"v1","addr":"10.20.193.197:6805","nonce":176999330}]},"public_addr":"10.20.193.197:6801/176999330","cluster_addr":"10.20.193.197:6803/176999330","heartbeat_back_addr":"10.20.193.197:6807/176999330","heartbeat_front_addr":"10.20.193.197:6805/176999330","state":["exists","up"]},{"osd":1,"uuid":"606c6d1b-0452-4860-a1dc-ca58763aa4e9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":10,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.197:6818","nonce":4254931048},{"type":"v1","addr":"10.20.193.197:6819","nonce":4254931048}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.197:6820","nonce":4254931048},{"type":"v1","addr":"10.20.193.197:6821","nonce":4254931048}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.197:6824","nonce":4254931048},{"type":"v1","addr":"10.20.193.197:6825","nonce":4254931048}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.197:6822","nonce":4254931048},{"type":"v1","addr":"10.20.193.197:6823","nonce":4254931048}]},"public_addr":"10.20.193.197:6819/4254931048","cluster_addr":"10.20.193.197:6821/4254931048","heartbeat_back_addr":"10.20.193.197:6825/4254931048","heartbeat_front_addr":"10.20.193.197:6823/4254931048","state":["exists","up"]},{"osd":2,"uuid":"4a54c53b-ac96-4483-97dc-bf76ae085e1b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":10,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.197:6810","nonce":2617788373},{"type":"v1","addr":"10.20.193.197:6811","nonce":2617788373}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.197:6812","nonce":2617788373},{"type":"v1","addr":"10.20.193.197:6813","nonce":2617788373}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.197:6816","nonce":2617788373},{"type":"v1","addr":"10.20.193.197:6817","nonce":2617788373}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.197:6814","nonce":2617788373},{"type":"v1","addr":"10.20.193.197:6815","nonce":2617788373}]},"public_addr":"10.20.193.197:6811/2617788373","cluster_addr":"10.20.193.197:6813/2617788373","heartbeat_back_addr":"10.20.193.197:6817/2617788373","heartbeat_front_addr":"10.20.193.197:6815/2617788373","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2026-02-26T03:01:11.283523+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2026-02-26T03:01:11.607888+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2026-02-26T03:01:11.277139+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":1,"snaps":[{"begin":2,"length":1}]}],"new_removed_snaps":[{"pool":1,"snaps":[{"begin":2,"length":1}]}],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-26T03:01:22.200 DEBUG:teuthology.orchestra.run.trial197:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2026-02-26T03:01:22.201 DEBUG:teuthology.orchestra.run.trial197:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2026-02-26T03:01:22.202 DEBUG:teuthology.orchestra.run.trial197:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2026-02-26T03:01:22.288 INFO:teuthology.orchestra.run.trial197.stdout:34359738372 2026-02-26T03:01:22.289 DEBUG:teuthology.orchestra.run.trial197:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2026-02-26T03:01:22.290 INFO:teuthology.orchestra.run.trial197.stdout:34359738372 2026-02-26T03:01:22.290 DEBUG:teuthology.orchestra.run.trial197:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2026-02-26T03:01:22.291 INFO:teuthology.orchestra.run.trial197.stdout:38654705668 2026-02-26T03:01:22.292 DEBUG:teuthology.orchestra.run.trial197:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2026-02-26T03:01:22.491 INFO:teuthology.orchestra.run.trial197.stdout:38654705667 2026-02-26T03:01:22.492 INFO:teuthology.orchestra.run.trial197.stdout:34359738371 2026-02-26T03:01:22.494 INFO:teuthology.orchestra.run.trial197.stdout:34359738371 2026-02-26T03:01:22.501 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654705668 got 38654705667 for osd.1 2026-02-26T03:01:22.502 INFO:tasks.ceph.ceph_manager.ceph:need seq 34359738372 got 34359738371 for osd.2 2026-02-26T03:01:22.504 INFO:tasks.ceph.ceph_manager.ceph:need seq 34359738372 got 34359738371 for osd.0 2026-02-26T03:01:23.502 DEBUG:teuthology.orchestra.run.trial197:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2026-02-26T03:01:23.502 DEBUG:teuthology.orchestra.run.trial197:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2026-02-26T03:01:23.504 DEBUG:teuthology.orchestra.run.trial197:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2026-02-26T03:01:23.663 INFO:teuthology.orchestra.run.trial197.stdout:34359738372 2026-02-26T03:01:23.665 INFO:teuthology.orchestra.run.trial197.stdout:38654705668 2026-02-26T03:01:23.667 INFO:teuthology.orchestra.run.trial197.stdout:34359738372 2026-02-26T03:01:23.673 INFO:tasks.ceph.ceph_manager.ceph:need seq 34359738372 got 34359738372 for osd.2 2026-02-26T03:01:23.673 DEBUG:teuthology.parallel:result is None 2026-02-26T03:01:23.675 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654705668 got 38654705668 for osd.1 2026-02-26T03:01:23.676 DEBUG:teuthology.parallel:result is None 2026-02-26T03:01:23.677 INFO:tasks.ceph.ceph_manager.ceph:need seq 34359738372 got 34359738372 for osd.0 2026-02-26T03:01:23.677 DEBUG:teuthology.parallel:result is None 2026-02-26T03:01:23.677 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2026-02-26T03:01:23.678 DEBUG:teuthology.orchestra.run.trial197:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-26T03:01:23.874 INFO:teuthology.orchestra.run.trial197.stdout: 2026-02-26T03:01:23.875 INFO:teuthology.orchestra.run.trial197.stderr:dumped all 2026-02-26T03:01:23.883 INFO:teuthology.orchestra.run.trial197.stdout:{"pg_ready":true,"pg_map":{"version":16,"stamp":"2026-02-26T03:01:22.380066+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":15,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":81240,"kb_used_data":336,"kb_used_omap":4,"kb_used_meta":80827,"kb_avail":283034280,"statfs":{"total":289910292480,"available":289827102720,"internally_reserved":0,"allocated":344064,"data_stored":86978,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4782,"internal_metadata":82767186},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"0.000000"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":19,"reported_epoch":13,"state":"active+clean","last_fresh":"2026-02-26T03:01:21.671597+0000","last_change":"2026-02-26T03:01:21.671776+0000","last_active":"2026-02-26T03:01:21.671597+0000","last_peered":"2026-02-26T03:01:21.671597+0000","last_clean":"2026-02-26T03:01:21.671597+0000","last_became_active":"2026-02-26T03:01:19.668398+0000","last_became_peered":"2026-02-26T03:01:19.668398+0000","last_unstale":"2026-02-26T03:01:21.671597+0000","last_undegraded":"2026-02-26T03:01:21.671597+0000","last_fullsized":"2026-02-26T03:01:21.671597+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-26T03:01:18.659251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-26T03:01:18.659251+0000","last_clean_scrub_stamp":"2026-02-26T03:01:18.659251+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-27T04:08:38.487713+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00060011299999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"1.6","version":"0'0","reported_seq":19,"reported_epoch":13,"state":"active+clean","last_fresh":"2026-02-26T03:01:21.671689+0000","last_change":"2026-02-26T03:01:21.671905+0000","last_active":"2026-02-26T03:01:21.671689+0000","last_peered":"2026-02-26T03:01:21.671689+0000","last_clean":"2026-02-26T03:01:21.671689+0000","last_became_active":"2026-02-26T03:01:19.669316+0000","last_became_peered":"2026-02-26T03:01:19.669316+0000","last_unstale":"2026-02-26T03:01:21.671689+0000","last_undegraded":"2026-02-26T03:01:21.671689+0000","last_fullsized":"2026-02-26T03:01:21.671689+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-26T03:01:18.659251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-26T03:01:18.659251+0000","last_clean_scrub_stamp":"2026-02-26T03:01:18.659251+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-27T13:43:20.762574+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00068042299999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"1.5","version":"0'0","reported_seq":19,"reported_epoch":13,"state":"active+clean","last_fresh":"2026-02-26T03:01:21.669362+0000","last_change":"2026-02-26T03:01:21.669362+0000","last_active":"2026-02-26T03:01:21.669362+0000","last_peered":"2026-02-26T03:01:21.669362+0000","last_clean":"2026-02-26T03:01:21.669362+0000","last_became_active":"2026-02-26T03:01:19.669313+0000","last_became_peered":"2026-02-26T03:01:19.669313+0000","last_unstale":"2026-02-26T03:01:21.669362+0000","last_undegraded":"2026-02-26T03:01:21.669362+0000","last_fullsized":"2026-02-26T03:01:21.669362+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-26T03:01:18.659251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-26T03:01:18.659251+0000","last_clean_scrub_stamp":"2026-02-26T03:01:18.659251+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-27T10:49:28.489885+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00034704899999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"1.4","version":"0'0","reported_seq":19,"reported_epoch":13,"state":"active+clean","last_fresh":"2026-02-26T03:01:21.671544+0000","last_change":"2026-02-26T03:01:21.671647+0000","last_active":"2026-02-26T03:01:21.671544+0000","last_peered":"2026-02-26T03:01:21.671544+0000","last_clean":"2026-02-26T03:01:21.671544+0000","last_became_active":"2026-02-26T03:01:19.669171+0000","last_became_peered":"2026-02-26T03:01:19.669171+0000","last_unstale":"2026-02-26T03:01:21.671544+0000","last_undegraded":"2026-02-26T03:01:21.671544+0000","last_fullsized":"2026-02-26T03:01:21.671544+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-26T03:01:18.659251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-26T03:01:18.659251+0000","last_clean_scrub_stamp":"2026-02-26T03:01:18.659251+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-27T04:13:58.662319+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000482733,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"1.3","version":"11'1","reported_seq":20,"reported_epoch":13,"state":"active+clean","last_fresh":"2026-02-26T03:01:21.671359+0000","last_change":"2026-02-26T03:01:21.671498+0000","last_active":"2026-02-26T03:01:21.671359+0000","last_peered":"2026-02-26T03:01:21.671359+0000","last_clean":"2026-02-26T03:01:21.671359+0000","last_became_active":"2026-02-26T03:01:19.668677+0000","last_became_peered":"2026-02-26T03:01:19.668677+0000","last_unstale":"2026-02-26T03:01:21.671359+0000","last_undegraded":"2026-02-26T03:01:21.671359+0000","last_fullsized":"2026-02-26T03:01:21.671359+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-26T03:01:18.659251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-26T03:01:18.659251+0000","last_clean_scrub_stamp":"2026-02-26T03:01:18.659251+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-27T09:30:44.521018+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00034461499999999999,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"1.2","version":"13'2","reported_seq":21,"reported_epoch":13,"state":"active+clean","last_fresh":"2026-02-26T03:01:21.670688+0000","last_change":"2026-02-26T03:01:21.670052+0000","last_active":"2026-02-26T03:01:21.670688+0000","last_peered":"2026-02-26T03:01:21.670688+0000","last_clean":"2026-02-26T03:01:21.670688+0000","last_became_active":"2026-02-26T03:01:19.668415+0000","last_became_peered":"2026-02-26T03:01:19.668415+0000","last_unstale":"2026-02-26T03:01:21.670688+0000","last_undegraded":"2026-02-26T03:01:21.670688+0000","last_fullsized":"2026-02-26T03:01:21.670688+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-26T03:01:18.659251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-26T03:01:18.659251+0000","last_clean_scrub_stamp":"2026-02-26T03:01:18.659251+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-27T12:03:13.020042+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00054467999999999997,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"1.1","version":"0'0","reported_seq":19,"reported_epoch":13,"state":"active+clean","last_fresh":"2026-02-26T03:01:21.669370+0000","last_change":"2026-02-26T03:01:21.669370+0000","last_active":"2026-02-26T03:01:21.669370+0000","last_peered":"2026-02-26T03:01:21.669370+0000","last_clean":"2026-02-26T03:01:21.669370+0000","last_became_active":"2026-02-26T03:01:19.669112+0000","last_became_peered":"2026-02-26T03:01:19.669112+0000","last_unstale":"2026-02-26T03:01:21.669370+0000","last_undegraded":"2026-02-26T03:01:21.669370+0000","last_fullsized":"2026-02-26T03:01:21.669370+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-26T03:01:18.659251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-26T03:01:18.659251+0000","last_clean_scrub_stamp":"2026-02-26T03:01:18.659251+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-27T14:09:53.100227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00037735600000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"1.0","version":"0'0","reported_seq":19,"reported_epoch":13,"state":"active+clean","last_fresh":"2026-02-26T03:01:21.671410+0000","last_change":"2026-02-26T03:01:21.671520+0000","last_active":"2026-02-26T03:01:21.671410+0000","last_peered":"2026-02-26T03:01:21.671410+0000","last_clean":"2026-02-26T03:01:21.671410+0000","last_became_active":"2026-02-26T03:01:19.669362+0000","last_became_peered":"2026-02-26T03:01:19.669362+0000","last_unstale":"2026-02-26T03:01:21.671410+0000","last_undegraded":"2026-02-26T03:01:21.671410+0000","last_fullsized":"2026-02-26T03:01:21.671410+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-26T03:01:18.659251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-26T03:01:18.659251+0000","last_clean_scrub_stamp":"2026-02-26T03:01:18.659251+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-27T03:50:25.293079+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00037062400000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":1,"up_from":9,"seq":38654705668,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27080,"kb_used_data":116,"kb_used_omap":1,"kb_used_meta":26942,"kb_avail":94344760,"statfs":{"total":96636764160,"available":96609034240,"internally_reserved":0,"allocated":118784,"data_stored":30057,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27589062},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":8,"seq":34359738372,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27072,"kb_used_data":104,"kb_used_omap":1,"kb_used_meta":26942,"kb_avail":94344768,"statfs":{"total":96636764160,"available":96609042432,"internally_reserved":0,"allocated":106496,"data_stored":26864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27589062},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738372,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27088,"kb_used_data":116,"kb_used_omap":1,"kb_used_meta":26942,"kb_avail":94344752,"statfs":{"total":96636764160,"available":96609026048,"internally_reserved":0,"allocated":118784,"data_stored":30057,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27589062},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-26T03:01:23.884 DEBUG:teuthology.orchestra.run.trial197:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-26T03:01:24.041 INFO:teuthology.orchestra.run.trial197.stdout: 2026-02-26T03:01:24.042 INFO:teuthology.orchestra.run.trial197.stderr:dumped all 2026-02-26T03:01:24.050 INFO:teuthology.orchestra.run.trial197.stdout:{"pg_ready":true,"pg_map":{"version":16,"stamp":"2026-02-26T03:01:22.380066+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":15,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":81240,"kb_used_data":336,"kb_used_omap":4,"kb_used_meta":80827,"kb_avail":283034280,"statfs":{"total":289910292480,"available":289827102720,"internally_reserved":0,"allocated":344064,"data_stored":86978,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4782,"internal_metadata":82767186},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"0.000000"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":19,"reported_epoch":13,"state":"active+clean","last_fresh":"2026-02-26T03:01:21.671597+0000","last_change":"2026-02-26T03:01:21.671776+0000","last_active":"2026-02-26T03:01:21.671597+0000","last_peered":"2026-02-26T03:01:21.671597+0000","last_clean":"2026-02-26T03:01:21.671597+0000","last_became_active":"2026-02-26T03:01:19.668398+0000","last_became_peered":"2026-02-26T03:01:19.668398+0000","last_unstale":"2026-02-26T03:01:21.671597+0000","last_undegraded":"2026-02-26T03:01:21.671597+0000","last_fullsized":"2026-02-26T03:01:21.671597+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-26T03:01:18.659251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-26T03:01:18.659251+0000","last_clean_scrub_stamp":"2026-02-26T03:01:18.659251+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-27T04:08:38.487713+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00060011299999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"1.6","version":"0'0","reported_seq":19,"reported_epoch":13,"state":"active+clean","last_fresh":"2026-02-26T03:01:21.671689+0000","last_change":"2026-02-26T03:01:21.671905+0000","last_active":"2026-02-26T03:01:21.671689+0000","last_peered":"2026-02-26T03:01:21.671689+0000","last_clean":"2026-02-26T03:01:21.671689+0000","last_became_active":"2026-02-26T03:01:19.669316+0000","last_became_peered":"2026-02-26T03:01:19.669316+0000","last_unstale":"2026-02-26T03:01:21.671689+0000","last_undegraded":"2026-02-26T03:01:21.671689+0000","last_fullsized":"2026-02-26T03:01:21.671689+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-26T03:01:18.659251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-26T03:01:18.659251+0000","last_clean_scrub_stamp":"2026-02-26T03:01:18.659251+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-27T13:43:20.762574+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00068042299999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"1.5","version":"0'0","reported_seq":19,"reported_epoch":13,"state":"active+clean","last_fresh":"2026-02-26T03:01:21.669362+0000","last_change":"2026-02-26T03:01:21.669362+0000","last_active":"2026-02-26T03:01:21.669362+0000","last_peered":"2026-02-26T03:01:21.669362+0000","last_clean":"2026-02-26T03:01:21.669362+0000","last_became_active":"2026-02-26T03:01:19.669313+0000","last_became_peered":"2026-02-26T03:01:19.669313+0000","last_unstale":"2026-02-26T03:01:21.669362+0000","last_undegraded":"2026-02-26T03:01:21.669362+0000","last_fullsized":"2026-02-26T03:01:21.669362+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-26T03:01:18.659251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-26T03:01:18.659251+0000","last_clean_scrub_stamp":"2026-02-26T03:01:18.659251+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-27T10:49:28.489885+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00034704899999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"1.4","version":"0'0","reported_seq":19,"reported_epoch":13,"state":"active+clean","last_fresh":"2026-02-26T03:01:21.671544+0000","last_change":"2026-02-26T03:01:21.671647+0000","last_active":"2026-02-26T03:01:21.671544+0000","last_peered":"2026-02-26T03:01:21.671544+0000","last_clean":"2026-02-26T03:01:21.671544+0000","last_became_active":"2026-02-26T03:01:19.669171+0000","last_became_peered":"2026-02-26T03:01:19.669171+0000","last_unstale":"2026-02-26T03:01:21.671544+0000","last_undegraded":"2026-02-26T03:01:21.671544+0000","last_fullsized":"2026-02-26T03:01:21.671544+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-26T03:01:18.659251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-26T03:01:18.659251+0000","last_clean_scrub_stamp":"2026-02-26T03:01:18.659251+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-27T04:13:58.662319+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000482733,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"1.3","version":"11'1","reported_seq":20,"reported_epoch":13,"state":"active+clean","last_fresh":"2026-02-26T03:01:21.671359+0000","last_change":"2026-02-26T03:01:21.671498+0000","last_active":"2026-02-26T03:01:21.671359+0000","last_peered":"2026-02-26T03:01:21.671359+0000","last_clean":"2026-02-26T03:01:21.671359+0000","last_became_active":"2026-02-26T03:01:19.668677+0000","last_became_peered":"2026-02-26T03:01:19.668677+0000","last_unstale":"2026-02-26T03:01:21.671359+0000","last_undegraded":"2026-02-26T03:01:21.671359+0000","last_fullsized":"2026-02-26T03:01:21.671359+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-26T03:01:18.659251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-26T03:01:18.659251+0000","last_clean_scrub_stamp":"2026-02-26T03:01:18.659251+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-27T09:30:44.521018+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00034461499999999999,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"1.2","version":"13'2","reported_seq":21,"reported_epoch":13,"state":"active+clean","last_fresh":"2026-02-26T03:01:21.670688+0000","last_change":"2026-02-26T03:01:21.670052+0000","last_active":"2026-02-26T03:01:21.670688+0000","last_peered":"2026-02-26T03:01:21.670688+0000","last_clean":"2026-02-26T03:01:21.670688+0000","last_became_active":"2026-02-26T03:01:19.668415+0000","last_became_peered":"2026-02-26T03:01:19.668415+0000","last_unstale":"2026-02-26T03:01:21.670688+0000","last_undegraded":"2026-02-26T03:01:21.670688+0000","last_fullsized":"2026-02-26T03:01:21.670688+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-26T03:01:18.659251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-26T03:01:18.659251+0000","last_clean_scrub_stamp":"2026-02-26T03:01:18.659251+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-27T12:03:13.020042+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00054467999999999997,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"1.1","version":"0'0","reported_seq":19,"reported_epoch":13,"state":"active+clean","last_fresh":"2026-02-26T03:01:21.669370+0000","last_change":"2026-02-26T03:01:21.669370+0000","last_active":"2026-02-26T03:01:21.669370+0000","last_peered":"2026-02-26T03:01:21.669370+0000","last_clean":"2026-02-26T03:01:21.669370+0000","last_became_active":"2026-02-26T03:01:19.669112+0000","last_became_peered":"2026-02-26T03:01:19.669112+0000","last_unstale":"2026-02-26T03:01:21.669370+0000","last_undegraded":"2026-02-26T03:01:21.669370+0000","last_fullsized":"2026-02-26T03:01:21.669370+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-26T03:01:18.659251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-26T03:01:18.659251+0000","last_clean_scrub_stamp":"2026-02-26T03:01:18.659251+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-27T14:09:53.100227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00037735600000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"1.0","version":"0'0","reported_seq":19,"reported_epoch":13,"state":"active+clean","last_fresh":"2026-02-26T03:01:21.671410+0000","last_change":"2026-02-26T03:01:21.671520+0000","last_active":"2026-02-26T03:01:21.671410+0000","last_peered":"2026-02-26T03:01:21.671410+0000","last_clean":"2026-02-26T03:01:21.671410+0000","last_became_active":"2026-02-26T03:01:19.669362+0000","last_became_peered":"2026-02-26T03:01:19.669362+0000","last_unstale":"2026-02-26T03:01:21.671410+0000","last_undegraded":"2026-02-26T03:01:21.671410+0000","last_fullsized":"2026-02-26T03:01:21.671410+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-26T03:01:18.659251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-26T03:01:18.659251+0000","last_clean_scrub_stamp":"2026-02-26T03:01:18.659251+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-27T03:50:25.293079+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00037062400000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":1,"up_from":9,"seq":38654705668,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27080,"kb_used_data":116,"kb_used_omap":1,"kb_used_meta":26942,"kb_avail":94344760,"statfs":{"total":96636764160,"available":96609034240,"internally_reserved":0,"allocated":118784,"data_stored":30057,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27589062},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":8,"seq":34359738372,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27072,"kb_used_data":104,"kb_used_omap":1,"kb_used_meta":26942,"kb_avail":94344768,"statfs":{"total":96636764160,"available":96609042432,"internally_reserved":0,"allocated":106496,"data_stored":26864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27589062},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738372,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27088,"kb_used_data":116,"kb_used_omap":1,"kb_used_meta":26942,"kb_avail":94344752,"statfs":{"total":96636764160,"available":96609026048,"internally_reserved":0,"allocated":118784,"data_stored":30057,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27589062},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-26T03:01:24.052 INFO:tasks.ceph.ceph_manager.ceph:clean! 2026-02-26T03:01:24.052 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2026-02-26T03:01:24.052 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy 2026-02-26T03:01:24.052 DEBUG:teuthology.orchestra.run.trial197:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2026-02-26T03:01:24.226 INFO:teuthology.orchestra.run.trial197.stdout: 2026-02-26T03:01:24.226 INFO:teuthology.orchestra.run.trial197.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2026-02-26T03:01:24.234 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy done 2026-02-26T03:01:24.235 INFO:teuthology.run_tasks:Running task divergent_priors2... 2026-02-26T03:01:24.254 DEBUG:teuthology.orchestra.run.trial197:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump 2026-02-26T03:01:24.411 INFO:teuthology.orchestra.run.trial197.stdout:epoch 14 2026-02-26T03:01:24.411 INFO:teuthology.orchestra.run.trial197.stdout:fsid 658087b3-2758-487d-ba7c-f8b171a8f312 2026-02-26T03:01:24.411 INFO:teuthology.orchestra.run.trial197.stdout:created 2026-02-26T03:01:04.144741+0000 2026-02-26T03:01:24.412 INFO:teuthology.orchestra.run.trial197.stdout:modified 2026-02-26T03:01:24.142155+0000 2026-02-26T03:01:24.412 INFO:teuthology.orchestra.run.trial197.stdout:flags sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit 2026-02-26T03:01:24.412 INFO:teuthology.orchestra.run.trial197.stdout:crush_version 5 2026-02-26T03:01:24.412 INFO:teuthology.orchestra.run.trial197.stdout:full_ratio 0.95 2026-02-26T03:01:24.412 INFO:teuthology.orchestra.run.trial197.stdout:backfillfull_ratio 0.9 2026-02-26T03:01:24.412 INFO:teuthology.orchestra.run.trial197.stdout:nearfull_ratio 0.85 2026-02-26T03:01:24.412 INFO:teuthology.orchestra.run.trial197.stdout:require_min_compat_client luminous 2026-02-26T03:01:24.412 INFO:teuthology.orchestra.run.trial197.stdout:min_compat_client jewel 2026-02-26T03:01:24.412 INFO:teuthology.orchestra.run.trial197.stdout:require_osd_release squid 2026-02-26T03:01:24.412 INFO:teuthology.orchestra.run.trial197.stdout:stretch_mode_enabled false 2026-02-26T03:01:24.412 INFO:teuthology.orchestra.run.trial197.stdout:pool 1 'rbd' replicated size 2 min_size 1 crush_rule 0 object_hash rjenkins pg_num 8 pgp_num 8 autoscale_mode off last_change 13 flags hashpspool,selfmanaged_snaps stripe_width 0 application rbd read_balance_score 1.88 2026-02-26T03:01:24.413 INFO:teuthology.orchestra.run.trial197.stdout:max_osd 3 2026-02-26T03:01:24.413 INFO:teuthology.orchestra.run.trial197.stdout:osd.0 up in weight 1 up_from 8 up_thru 10 down_at 0 last_clean_interval [0,0) [v2:10.20.193.197:6800/176999330,v1:10.20.193.197:6801/176999330] [v2:10.20.193.197:6802/176999330,v1:10.20.193.197:6803/176999330] exists,up 696e32c7-b2ad-41e8-9c32-f7e12eb617ee 2026-02-26T03:01:24.413 INFO:teuthology.orchestra.run.trial197.stdout:osd.1 up in weight 1 up_from 9 up_thru 10 down_at 0 last_clean_interval [0,0) [v2:10.20.193.197:6818/4254931048,v1:10.20.193.197:6819/4254931048] [v2:10.20.193.197:6820/4254931048,v1:10.20.193.197:6821/4254931048] exists,up 606c6d1b-0452-4860-a1dc-ca58763aa4e9 2026-02-26T03:01:24.413 INFO:teuthology.orchestra.run.trial197.stdout:osd.2 up in weight 1 up_from 8 up_thru 10 down_at 0 last_clean_interval [0,0) [v2:10.20.193.197:6810/2617788373,v1:10.20.193.197:6811/2617788373] [v2:10.20.193.197:6812/2617788373,v1:10.20.193.197:6813/2617788373] exists,up 4a54c53b-ac96-4483-97dc-bf76ae085e1b 2026-02-26T03:01:24.419 INFO:tasks.ceph.ceph_manager.ceph:['osd.0 up in weight 1 up_from 8 up_thru 10 down_at 0 last_clean_interval [0,0) [v2:10.20.193.197:6800/176999330,v1:10.20.193.197:6801/176999330] [v2:10.20.193.197:6802/176999330,v1:10.20.193.197:6803/176999330] exists,up 696e32c7-b2ad-41e8-9c32-f7e12eb617ee', 'osd.1 up in weight 1 up_from 9 up_thru 10 down_at 0 last_clean_interval [0,0) [v2:10.20.193.197:6818/4254931048,v1:10.20.193.197:6819/4254931048] [v2:10.20.193.197:6820/4254931048,v1:10.20.193.197:6821/4254931048] exists,up 606c6d1b-0452-4860-a1dc-ca58763aa4e9', 'osd.2 up in weight 1 up_from 8 up_thru 10 down_at 0 last_clean_interval [0,0) [v2:10.20.193.197:6810/2617788373,v1:10.20.193.197:6811/2617788373] [v2:10.20.193.197:6812/2617788373,v1:10.20.193.197:6813/2617788373] exists,up 4a54c53b-ac96-4483-97dc-bf76ae085e1b'] 2026-02-26T03:01:24.420 DEBUG:teuthology.orchestra.run.trial197:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2026-02-26T03:01:24.420 DEBUG:teuthology.orchestra.run.trial197:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2026-02-26T03:01:24.421 DEBUG:teuthology.orchestra.run.trial197:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2026-02-26T03:01:24.506 INFO:teuthology.orchestra.run.trial197.stdout:38654705670 2026-02-26T03:01:24.506 DEBUG:teuthology.orchestra.run.trial197:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2026-02-26T03:01:24.507 INFO:teuthology.orchestra.run.trial197.stdout:34359738374 2026-02-26T03:01:24.507 DEBUG:teuthology.orchestra.run.trial197:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2026-02-26T03:01:24.507 INFO:teuthology.orchestra.run.trial197.stdout:34359738374 2026-02-26T03:01:24.507 DEBUG:teuthology.orchestra.run.trial197:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2026-02-26T03:01:24.707 INFO:teuthology.orchestra.run.trial197.stdout:34359738372 2026-02-26T03:01:24.711 INFO:teuthology.orchestra.run.trial197.stdout:34359738372 2026-02-26T03:01:24.713 INFO:teuthology.orchestra.run.trial197.stdout:38654705668 2026-02-26T03:01:24.717 INFO:tasks.ceph.ceph_manager.ceph:need seq 34359738374 got 34359738372 for osd.0 2026-02-26T03:01:24.721 INFO:tasks.ceph.ceph_manager.ceph:need seq 34359738374 got 34359738372 for osd.2 2026-02-26T03:01:24.724 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654705670 got 38654705668 for osd.1 2026-02-26T03:01:25.718 DEBUG:teuthology.orchestra.run.trial197:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2026-02-26T03:01:25.722 DEBUG:teuthology.orchestra.run.trial197:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2026-02-26T03:01:25.725 DEBUG:teuthology.orchestra.run.trial197:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2026-02-26T03:01:25.880 INFO:teuthology.orchestra.run.trial197.stdout:34359738373 2026-02-26T03:01:25.890 INFO:tasks.ceph.ceph_manager.ceph:need seq 34359738374 got 34359738373 for osd.0 2026-02-26T03:01:25.922 INFO:teuthology.orchestra.run.trial197.stdout:38654705668 2026-02-26T03:01:25.926 INFO:teuthology.orchestra.run.trial197.stdout:34359738373 2026-02-26T03:01:25.931 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654705670 got 38654705668 for osd.1 2026-02-26T03:01:25.936 INFO:tasks.ceph.ceph_manager.ceph:need seq 34359738374 got 34359738373 for osd.2 2026-02-26T03:01:26.892 DEBUG:teuthology.orchestra.run.trial197:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2026-02-26T03:01:26.932 DEBUG:teuthology.orchestra.run.trial197:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2026-02-26T03:01:26.936 DEBUG:teuthology.orchestra.run.trial197:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2026-02-26T03:01:27.052 INFO:teuthology.orchestra.run.trial197.stdout:34359738373 2026-02-26T03:01:27.063 INFO:tasks.ceph.ceph_manager.ceph:need seq 34359738374 got 34359738373 for osd.0 2026-02-26T03:01:27.095 INFO:teuthology.orchestra.run.trial197.stdout:34359738373 2026-02-26T03:01:27.097 INFO:teuthology.orchestra.run.trial197.stdout:38654705668 2026-02-26T03:01:27.104 INFO:tasks.ceph.ceph_manager.ceph:need seq 34359738374 got 34359738373 for osd.2 2026-02-26T03:01:27.108 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654705670 got 38654705668 for osd.1 2026-02-26T03:01:28.064 DEBUG:teuthology.orchestra.run.trial197:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2026-02-26T03:01:28.104 DEBUG:teuthology.orchestra.run.trial197:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2026-02-26T03:01:28.108 DEBUG:teuthology.orchestra.run.trial197:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2026-02-26T03:01:28.225 INFO:teuthology.orchestra.run.trial197.stdout:34359738374 2026-02-26T03:01:28.235 INFO:tasks.ceph.ceph_manager.ceph:need seq 34359738374 got 34359738374 for osd.0 2026-02-26T03:01:28.235 DEBUG:teuthology.parallel:result is None 2026-02-26T03:01:28.268 INFO:teuthology.orchestra.run.trial197.stdout:38654705670 2026-02-26T03:01:28.273 INFO:teuthology.orchestra.run.trial197.stdout:34359738374 2026-02-26T03:01:28.277 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654705670 got 38654705670 for osd.1 2026-02-26T03:01:28.278 DEBUG:teuthology.parallel:result is None 2026-02-26T03:01:28.284 INFO:tasks.ceph.ceph_manager.ceph:need seq 34359738374 got 34359738374 for osd.2 2026-02-26T03:01:28.284 DEBUG:teuthology.parallel:result is None 2026-02-26T03:01:28.284 DEBUG:teuthology.orchestra.run.trial197:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set noout 2026-02-26T03:01:29.151 INFO:teuthology.orchestra.run.trial197.stderr:noout is set 2026-02-26T03:01:29.162 DEBUG:teuthology.orchestra.run.trial197:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set noin 2026-02-26T03:01:30.155 INFO:teuthology.orchestra.run.trial197.stderr:noin is set 2026-02-26T03:01:30.166 DEBUG:teuthology.orchestra.run.trial197:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set nodown 2026-02-26T03:01:31.157 INFO:teuthology.orchestra.run.trial197.stderr:nodown is set 2026-02-26T03:01:31.169 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2026-02-26T03:01:31.170 DEBUG:teuthology.orchestra.run.trial197:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-26T03:01:31.325 INFO:teuthology.orchestra.run.trial197.stdout: 2026-02-26T03:01:31.325 INFO:teuthology.orchestra.run.trial197.stderr:dumped all 2026-02-26T03:01:31.333 INFO:teuthology.orchestra.run.trial197.stdout:{"pg_ready":true,"pg_map":{"version":24,"stamp":"2026-02-26T03:01:31.157349+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":81260,"kb_used_data":368,"kb_used_omap":4,"kb_used_meta":80827,"kb_avail":283034260,"statfs":{"total":289910292480,"available":289827082240,"internally_reserved":0,"allocated":376832,"data_stored":98576,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4782,"internal_metadata":82767186},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.777091"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":21,"reported_epoch":14,"state":"active+clean","last_fresh":"2026-02-26T03:01:24.715497+0000","last_change":"2026-02-26T03:01:21.671776+0000","last_active":"2026-02-26T03:01:24.715497+0000","last_peered":"2026-02-26T03:01:24.715497+0000","last_clean":"2026-02-26T03:01:24.715497+0000","last_became_active":"2026-02-26T03:01:19.668398+0000","last_became_peered":"2026-02-26T03:01:19.668398+0000","last_unstale":"2026-02-26T03:01:24.715497+0000","last_undegraded":"2026-02-26T03:01:24.715497+0000","last_fullsized":"2026-02-26T03:01:24.715497+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-26T03:01:18.659251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-26T03:01:18.659251+0000","last_clean_scrub_stamp":"2026-02-26T03:01:18.659251+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-27T04:08:38.487713+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00060011299999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":21,"reported_epoch":14,"state":"active+clean","last_fresh":"2026-02-26T03:01:24.715484+0000","last_change":"2026-02-26T03:01:21.671905+0000","last_active":"2026-02-26T03:01:24.715484+0000","last_peered":"2026-02-26T03:01:24.715484+0000","last_clean":"2026-02-26T03:01:24.715484+0000","last_became_active":"2026-02-26T03:01:19.669316+0000","last_became_peered":"2026-02-26T03:01:19.669316+0000","last_unstale":"2026-02-26T03:01:24.715484+0000","last_undegraded":"2026-02-26T03:01:24.715484+0000","last_fullsized":"2026-02-26T03:01:24.715484+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-26T03:01:18.659251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-26T03:01:18.659251+0000","last_clean_scrub_stamp":"2026-02-26T03:01:18.659251+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-27T13:43:20.762574+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00068042299999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":21,"reported_epoch":14,"state":"active+clean","last_fresh":"2026-02-26T03:01:24.720277+0000","last_change":"2026-02-26T03:01:21.669362+0000","last_active":"2026-02-26T03:01:24.720277+0000","last_peered":"2026-02-26T03:01:24.720277+0000","last_clean":"2026-02-26T03:01:24.720277+0000","last_became_active":"2026-02-26T03:01:19.669313+0000","last_became_peered":"2026-02-26T03:01:19.669313+0000","last_unstale":"2026-02-26T03:01:24.720277+0000","last_undegraded":"2026-02-26T03:01:24.720277+0000","last_fullsized":"2026-02-26T03:01:24.720277+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-26T03:01:18.659251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-26T03:01:18.659251+0000","last_clean_scrub_stamp":"2026-02-26T03:01:18.659251+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-27T10:49:28.489885+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00034704899999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":21,"reported_epoch":14,"state":"active+clean","last_fresh":"2026-02-26T03:01:24.715459+0000","last_change":"2026-02-26T03:01:21.671647+0000","last_active":"2026-02-26T03:01:24.715459+0000","last_peered":"2026-02-26T03:01:24.715459+0000","last_clean":"2026-02-26T03:01:24.715459+0000","last_became_active":"2026-02-26T03:01:19.669171+0000","last_became_peered":"2026-02-26T03:01:19.669171+0000","last_unstale":"2026-02-26T03:01:24.715459+0000","last_undegraded":"2026-02-26T03:01:24.715459+0000","last_fullsized":"2026-02-26T03:01:24.715459+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-26T03:01:18.659251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-26T03:01:18.659251+0000","last_clean_scrub_stamp":"2026-02-26T03:01:18.659251+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-27T04:13:58.662319+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000482733,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.3","version":"11'1","reported_seq":22,"reported_epoch":14,"state":"active+clean","last_fresh":"2026-02-26T03:01:24.715474+0000","last_change":"2026-02-26T03:01:21.671498+0000","last_active":"2026-02-26T03:01:24.715474+0000","last_peered":"2026-02-26T03:01:24.715474+0000","last_clean":"2026-02-26T03:01:24.715474+0000","last_became_active":"2026-02-26T03:01:19.668677+0000","last_became_peered":"2026-02-26T03:01:19.668677+0000","last_unstale":"2026-02-26T03:01:24.715474+0000","last_undegraded":"2026-02-26T03:01:24.715474+0000","last_fullsized":"2026-02-26T03:01:24.715474+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-26T03:01:18.659251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-26T03:01:18.659251+0000","last_clean_scrub_stamp":"2026-02-26T03:01:18.659251+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-27T09:30:44.521018+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00034461499999999999,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"13'2","reported_seq":23,"reported_epoch":14,"state":"active+clean","last_fresh":"2026-02-26T03:01:24.143883+0000","last_change":"2026-02-26T03:01:21.670052+0000","last_active":"2026-02-26T03:01:24.143883+0000","last_peered":"2026-02-26T03:01:24.143883+0000","last_clean":"2026-02-26T03:01:24.143883+0000","last_became_active":"2026-02-26T03:01:19.668415+0000","last_became_peered":"2026-02-26T03:01:19.668415+0000","last_unstale":"2026-02-26T03:01:24.143883+0000","last_undegraded":"2026-02-26T03:01:24.143883+0000","last_fullsized":"2026-02-26T03:01:24.143883+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-26T03:01:18.659251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-26T03:01:18.659251+0000","last_clean_scrub_stamp":"2026-02-26T03:01:18.659251+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-27T12:03:13.020042+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00054467999999999997,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":21,"reported_epoch":14,"state":"active+clean","last_fresh":"2026-02-26T03:01:24.720279+0000","last_change":"2026-02-26T03:01:21.669370+0000","last_active":"2026-02-26T03:01:24.720279+0000","last_peered":"2026-02-26T03:01:24.720279+0000","last_clean":"2026-02-26T03:01:24.720279+0000","last_became_active":"2026-02-26T03:01:19.669112+0000","last_became_peered":"2026-02-26T03:01:19.669112+0000","last_unstale":"2026-02-26T03:01:24.720279+0000","last_undegraded":"2026-02-26T03:01:24.720279+0000","last_fullsized":"2026-02-26T03:01:24.720279+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-26T03:01:18.659251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-26T03:01:18.659251+0000","last_clean_scrub_stamp":"2026-02-26T03:01:18.659251+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-27T14:09:53.100227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00037735600000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":21,"reported_epoch":14,"state":"active+clean","last_fresh":"2026-02-26T03:01:24.715437+0000","last_change":"2026-02-26T03:01:21.671520+0000","last_active":"2026-02-26T03:01:24.715437+0000","last_peered":"2026-02-26T03:01:24.715437+0000","last_clean":"2026-02-26T03:01:24.715437+0000","last_became_active":"2026-02-26T03:01:19.669362+0000","last_became_peered":"2026-02-26T03:01:19.669362+0000","last_unstale":"2026-02-26T03:01:24.715437+0000","last_undegraded":"2026-02-26T03:01:24.715437+0000","last_fullsized":"2026-02-26T03:01:24.715437+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-26T03:01:18.659251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-26T03:01:18.659251+0000","last_clean_scrub_stamp":"2026-02-26T03:01:18.659251+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-27T03:50:25.293079+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00037062400000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":1,"up_from":9,"seq":38654705671,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27088,"kb_used_data":124,"kb_used_omap":1,"kb_used_meta":26942,"kb_avail":94344752,"statfs":{"total":96636764160,"available":96609026048,"internally_reserved":0,"allocated":126976,"data_stored":32865,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27589062},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":8,"seq":34359738375,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27084,"kb_used_data":120,"kb_used_omap":1,"kb_used_meta":26942,"kb_avail":94344756,"statfs":{"total":96636764160,"available":96609030144,"internally_reserved":0,"allocated":122880,"data_stored":32846,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27589062},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738375,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27088,"kb_used_data":124,"kb_used_omap":1,"kb_used_meta":26942,"kb_avail":94344752,"statfs":{"total":96636764160,"available":96609026048,"internally_reserved":0,"allocated":126976,"data_stored":32865,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27589062},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-26T03:01:31.334 DEBUG:teuthology.orchestra.run.trial197:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-26T03:01:31.491 INFO:teuthology.orchestra.run.trial197.stdout: 2026-02-26T03:01:31.492 INFO:teuthology.orchestra.run.trial197.stderr:dumped all 2026-02-26T03:01:31.500 INFO:teuthology.orchestra.run.trial197.stdout:{"pg_ready":true,"pg_map":{"version":24,"stamp":"2026-02-26T03:01:31.157349+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":81260,"kb_used_data":368,"kb_used_omap":4,"kb_used_meta":80827,"kb_avail":283034260,"statfs":{"total":289910292480,"available":289827082240,"internally_reserved":0,"allocated":376832,"data_stored":98576,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4782,"internal_metadata":82767186},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.777091"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":21,"reported_epoch":14,"state":"active+clean","last_fresh":"2026-02-26T03:01:24.715497+0000","last_change":"2026-02-26T03:01:21.671776+0000","last_active":"2026-02-26T03:01:24.715497+0000","last_peered":"2026-02-26T03:01:24.715497+0000","last_clean":"2026-02-26T03:01:24.715497+0000","last_became_active":"2026-02-26T03:01:19.668398+0000","last_became_peered":"2026-02-26T03:01:19.668398+0000","last_unstale":"2026-02-26T03:01:24.715497+0000","last_undegraded":"2026-02-26T03:01:24.715497+0000","last_fullsized":"2026-02-26T03:01:24.715497+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-26T03:01:18.659251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-26T03:01:18.659251+0000","last_clean_scrub_stamp":"2026-02-26T03:01:18.659251+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-27T04:08:38.487713+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00060011299999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":21,"reported_epoch":14,"state":"active+clean","last_fresh":"2026-02-26T03:01:24.715484+0000","last_change":"2026-02-26T03:01:21.671905+0000","last_active":"2026-02-26T03:01:24.715484+0000","last_peered":"2026-02-26T03:01:24.715484+0000","last_clean":"2026-02-26T03:01:24.715484+0000","last_became_active":"2026-02-26T03:01:19.669316+0000","last_became_peered":"2026-02-26T03:01:19.669316+0000","last_unstale":"2026-02-26T03:01:24.715484+0000","last_undegraded":"2026-02-26T03:01:24.715484+0000","last_fullsized":"2026-02-26T03:01:24.715484+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-26T03:01:18.659251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-26T03:01:18.659251+0000","last_clean_scrub_stamp":"2026-02-26T03:01:18.659251+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-27T13:43:20.762574+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00068042299999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":21,"reported_epoch":14,"state":"active+clean","last_fresh":"2026-02-26T03:01:24.720277+0000","last_change":"2026-02-26T03:01:21.669362+0000","last_active":"2026-02-26T03:01:24.720277+0000","last_peered":"2026-02-26T03:01:24.720277+0000","last_clean":"2026-02-26T03:01:24.720277+0000","last_became_active":"2026-02-26T03:01:19.669313+0000","last_became_peered":"2026-02-26T03:01:19.669313+0000","last_unstale":"2026-02-26T03:01:24.720277+0000","last_undegraded":"2026-02-26T03:01:24.720277+0000","last_fullsized":"2026-02-26T03:01:24.720277+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-26T03:01:18.659251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-26T03:01:18.659251+0000","last_clean_scrub_stamp":"2026-02-26T03:01:18.659251+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-27T10:49:28.489885+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00034704899999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":21,"reported_epoch":14,"state":"active+clean","last_fresh":"2026-02-26T03:01:24.715459+0000","last_change":"2026-02-26T03:01:21.671647+0000","last_active":"2026-02-26T03:01:24.715459+0000","last_peered":"2026-02-26T03:01:24.715459+0000","last_clean":"2026-02-26T03:01:24.715459+0000","last_became_active":"2026-02-26T03:01:19.669171+0000","last_became_peered":"2026-02-26T03:01:19.669171+0000","last_unstale":"2026-02-26T03:01:24.715459+0000","last_undegraded":"2026-02-26T03:01:24.715459+0000","last_fullsized":"2026-02-26T03:01:24.715459+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-26T03:01:18.659251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-26T03:01:18.659251+0000","last_clean_scrub_stamp":"2026-02-26T03:01:18.659251+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-27T04:13:58.662319+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000482733,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.3","version":"11'1","reported_seq":22,"reported_epoch":14,"state":"active+clean","last_fresh":"2026-02-26T03:01:24.715474+0000","last_change":"2026-02-26T03:01:21.671498+0000","last_active":"2026-02-26T03:01:24.715474+0000","last_peered":"2026-02-26T03:01:24.715474+0000","last_clean":"2026-02-26T03:01:24.715474+0000","last_became_active":"2026-02-26T03:01:19.668677+0000","last_became_peered":"2026-02-26T03:01:19.668677+0000","last_unstale":"2026-02-26T03:01:24.715474+0000","last_undegraded":"2026-02-26T03:01:24.715474+0000","last_fullsized":"2026-02-26T03:01:24.715474+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-26T03:01:18.659251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-26T03:01:18.659251+0000","last_clean_scrub_stamp":"2026-02-26T03:01:18.659251+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-27T09:30:44.521018+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00034461499999999999,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"13'2","reported_seq":23,"reported_epoch":14,"state":"active+clean","last_fresh":"2026-02-26T03:01:24.143883+0000","last_change":"2026-02-26T03:01:21.670052+0000","last_active":"2026-02-26T03:01:24.143883+0000","last_peered":"2026-02-26T03:01:24.143883+0000","last_clean":"2026-02-26T03:01:24.143883+0000","last_became_active":"2026-02-26T03:01:19.668415+0000","last_became_peered":"2026-02-26T03:01:19.668415+0000","last_unstale":"2026-02-26T03:01:24.143883+0000","last_undegraded":"2026-02-26T03:01:24.143883+0000","last_fullsized":"2026-02-26T03:01:24.143883+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-26T03:01:18.659251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-26T03:01:18.659251+0000","last_clean_scrub_stamp":"2026-02-26T03:01:18.659251+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-27T12:03:13.020042+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00054467999999999997,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":21,"reported_epoch":14,"state":"active+clean","last_fresh":"2026-02-26T03:01:24.720279+0000","last_change":"2026-02-26T03:01:21.669370+0000","last_active":"2026-02-26T03:01:24.720279+0000","last_peered":"2026-02-26T03:01:24.720279+0000","last_clean":"2026-02-26T03:01:24.720279+0000","last_became_active":"2026-02-26T03:01:19.669112+0000","last_became_peered":"2026-02-26T03:01:19.669112+0000","last_unstale":"2026-02-26T03:01:24.720279+0000","last_undegraded":"2026-02-26T03:01:24.720279+0000","last_fullsized":"2026-02-26T03:01:24.720279+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-26T03:01:18.659251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-26T03:01:18.659251+0000","last_clean_scrub_stamp":"2026-02-26T03:01:18.659251+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-27T14:09:53.100227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00037735600000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":21,"reported_epoch":14,"state":"active+clean","last_fresh":"2026-02-26T03:01:24.715437+0000","last_change":"2026-02-26T03:01:21.671520+0000","last_active":"2026-02-26T03:01:24.715437+0000","last_peered":"2026-02-26T03:01:24.715437+0000","last_clean":"2026-02-26T03:01:24.715437+0000","last_became_active":"2026-02-26T03:01:19.669362+0000","last_became_peered":"2026-02-26T03:01:19.669362+0000","last_unstale":"2026-02-26T03:01:24.715437+0000","last_undegraded":"2026-02-26T03:01:24.715437+0000","last_fullsized":"2026-02-26T03:01:24.715437+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-26T03:01:18.659251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-26T03:01:18.659251+0000","last_clean_scrub_stamp":"2026-02-26T03:01:18.659251+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-27T03:50:25.293079+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00037062400000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":1,"up_from":9,"seq":38654705671,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27088,"kb_used_data":124,"kb_used_omap":1,"kb_used_meta":26942,"kb_avail":94344752,"statfs":{"total":96636764160,"available":96609026048,"internally_reserved":0,"allocated":126976,"data_stored":32865,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27589062},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":8,"seq":34359738375,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27084,"kb_used_data":120,"kb_used_omap":1,"kb_used_meta":26942,"kb_avail":94344756,"statfs":{"total":96636764160,"available":96609030144,"internally_reserved":0,"allocated":122880,"data_stored":32846,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27589062},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738375,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27088,"kb_used_data":124,"kb_used_omap":1,"kb_used_meta":26942,"kb_avail":94344752,"statfs":{"total":96636764160,"available":96609026048,"internally_reserved":0,"allocated":126976,"data_stored":32865,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27589062},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-26T03:01:31.502 INFO:tasks.ceph.ceph_manager.ceph:clean! 2026-02-26T03:01:31.502 INFO:tasks.divergent_priors2:creating foo 2026-02-26T03:01:31.502 DEBUG:teuthology.orchestra.run.trial197:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create foo 1 2026-02-26T03:01:32.161 INFO:teuthology.orchestra.run.trial197.stderr:pool 'foo' created 2026-02-26T03:01:32.172 DEBUG:teuthology.orchestra.run.trial197:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok config set osd_min_pg_log_entries 10 2026-02-26T03:01:32.242 INFO:teuthology.orchestra.run.trial197.stdout:{ 2026-02-26T03:01:32.242 INFO:teuthology.orchestra.run.trial197.stdout: "success": "osd_min_pg_log_entries = '' (not observed, change may require restart) " 2026-02-26T03:01:32.242 INFO:teuthology.orchestra.run.trial197.stdout:} 2026-02-26T03:01:32.247 DEBUG:teuthology.orchestra.run.trial197:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok config set osd_max_pg_log_entries 10 2026-02-26T03:01:32.316 INFO:teuthology.orchestra.run.trial197.stdout:{ 2026-02-26T03:01:32.316 INFO:teuthology.orchestra.run.trial197.stdout: "success": "osd_max_pg_log_entries = '' (not observed, change may require restart) " 2026-02-26T03:01:32.316 INFO:teuthology.orchestra.run.trial197.stdout:} 2026-02-26T03:01:32.321 DEBUG:teuthology.orchestra.run.trial197:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok config set osd_pg_log_trim_min 5 2026-02-26T03:01:32.390 INFO:teuthology.orchestra.run.trial197.stdout:{ 2026-02-26T03:01:32.390 INFO:teuthology.orchestra.run.trial197.stdout: "success": "osd_pg_log_trim_min = '' (not observed, change may require restart) " 2026-02-26T03:01:32.391 INFO:teuthology.orchestra.run.trial197.stdout:} 2026-02-26T03:01:32.396 DEBUG:teuthology.orchestra.run.trial197:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok config set osd_min_pg_log_entries 10 2026-02-26T03:01:32.466 INFO:teuthology.orchestra.run.trial197.stdout:{ 2026-02-26T03:01:32.466 INFO:teuthology.orchestra.run.trial197.stdout: "success": "osd_min_pg_log_entries = '' (not observed, change may require restart) " 2026-02-26T03:01:32.466 INFO:teuthology.orchestra.run.trial197.stdout:} 2026-02-26T03:01:32.472 DEBUG:teuthology.orchestra.run.trial197:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok config set osd_max_pg_log_entries 10 2026-02-26T03:01:32.541 INFO:teuthology.orchestra.run.trial197.stdout:{ 2026-02-26T03:01:32.542 INFO:teuthology.orchestra.run.trial197.stdout: "success": "osd_max_pg_log_entries = '' (not observed, change may require restart) " 2026-02-26T03:01:32.542 INFO:teuthology.orchestra.run.trial197.stdout:} 2026-02-26T03:01:32.547 DEBUG:teuthology.orchestra.run.trial197:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok config set osd_pg_log_trim_min 5 2026-02-26T03:01:32.616 INFO:teuthology.orchestra.run.trial197.stdout:{ 2026-02-26T03:01:32.616 INFO:teuthology.orchestra.run.trial197.stdout: "success": "osd_pg_log_trim_min = '' (not observed, change may require restart) " 2026-02-26T03:01:32.616 INFO:teuthology.orchestra.run.trial197.stdout:} 2026-02-26T03:01:32.622 DEBUG:teuthology.orchestra.run.trial197:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok config set osd_min_pg_log_entries 10 2026-02-26T03:01:32.691 INFO:teuthology.orchestra.run.trial197.stdout:{ 2026-02-26T03:01:32.691 INFO:teuthology.orchestra.run.trial197.stdout: "success": "osd_min_pg_log_entries = '' (not observed, change may require restart) " 2026-02-26T03:01:32.691 INFO:teuthology.orchestra.run.trial197.stdout:} 2026-02-26T03:01:32.697 DEBUG:teuthology.orchestra.run.trial197:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok config set osd_max_pg_log_entries 10 2026-02-26T03:01:32.767 INFO:teuthology.orchestra.run.trial197.stdout:{ 2026-02-26T03:01:32.767 INFO:teuthology.orchestra.run.trial197.stdout: "success": "osd_max_pg_log_entries = '' (not observed, change may require restart) " 2026-02-26T03:01:32.767 INFO:teuthology.orchestra.run.trial197.stdout:} 2026-02-26T03:01:32.773 DEBUG:teuthology.orchestra.run.trial197:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok config set osd_pg_log_trim_min 5 2026-02-26T03:01:32.841 INFO:teuthology.orchestra.run.trial197.stdout:{ 2026-02-26T03:01:32.841 INFO:teuthology.orchestra.run.trial197.stdout: "success": "osd_pg_log_trim_min = '' (not observed, change may require restart) " 2026-02-26T03:01:32.841 INFO:teuthology.orchestra.run.trial197.stdout:} 2026-02-26T03:01:32.847 DEBUG:teuthology.orchestra.run.trial197:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2026-02-26T03:01:33.006 INFO:teuthology.orchestra.run.trial197.stdout: 2026-02-26T03:01:33.006 INFO:teuthology.orchestra.run.trial197.stdout:{"epoch":18,"fsid":"658087b3-2758-487d-ba7c-f8b171a8f312","created":"2026-02-26T03:01:04.144741+0000","modified":"2026-02-26T03:01:32.157530+0000","last_up_change":"2026-02-26T03:01:13.657148+0000","last_in_change":"2026-02-26T03:01:04.814150+0000","flags":"nodown,noout,noin,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5800384,"flags_set":["nodown","noin","noout","pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":"rbd","create_time":"2026-02-26T03:01:18.430774+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"13","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":2,"snap_epoch":13,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"foo","create_time":"2026-02-26T03:01:31.663753+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{},"read_balance":{"score_type":"Fair distribution","score_acting":2.9900000095367432,"score_stable":2.9900000095367432,"optimal_score":0.67000001668930054,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"696e32c7-b2ad-41e8-9c32-f7e12eb617ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":10,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.197:6800","nonce":176999330},{"type":"v1","addr":"10.20.193.197:6801","nonce":176999330}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.197:6802","nonce":176999330},{"type":"v1","addr":"10.20.193.197:6803","nonce":176999330}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.197:6806","nonce":176999330},{"type":"v1","addr":"10.20.193.197:6807","nonce":176999330}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.197:6804","nonce":176999330},{"type":"v1","addr":"10.20.193.197:6805","nonce":176999330}]},"public_addr":"10.20.193.197:6801/176999330","cluster_addr":"10.20.193.197:6803/176999330","heartbeat_back_addr":"10.20.193.197:6807/176999330","heartbeat_front_addr":"10.20.193.197:6805/176999330","state":["exists","up"]},{"osd":1,"uuid":"606c6d1b-0452-4860-a1dc-ca58763aa4e9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":10,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.197:6818","nonce":4254931048},{"type":"v1","addr":"10.20.193.197:6819","nonce":4254931048}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.197:6820","nonce":4254931048},{"type":"v1","addr":"10.20.193.197:6821","nonce":4254931048}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.197:6824","nonce":4254931048},{"type":"v1","addr":"10.20.193.197:6825","nonce":4254931048}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.197:6822","nonce":4254931048},{"type":"v1","addr":"10.20.193.197:6823","nonce":4254931048}]},"public_addr":"10.20.193.197:6819/4254931048","cluster_addr":"10.20.193.197:6821/4254931048","heartbeat_back_addr":"10.20.193.197:6825/4254931048","heartbeat_front_addr":"10.20.193.197:6823/4254931048","state":["exists","up"]},{"osd":2,"uuid":"4a54c53b-ac96-4483-97dc-bf76ae085e1b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":10,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.197:6810","nonce":2617788373},{"type":"v1","addr":"10.20.193.197:6811","nonce":2617788373}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.197:6812","nonce":2617788373},{"type":"v1","addr":"10.20.193.197:6813","nonce":2617788373}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.197:6816","nonce":2617788373},{"type":"v1","addr":"10.20.193.197:6817","nonce":2617788373}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.197:6814","nonce":2617788373},{"type":"v1","addr":"10.20.193.197:6815","nonce":2617788373}]},"public_addr":"10.20.193.197:6811/2617788373","cluster_addr":"10.20.193.197:6813/2617788373","heartbeat_back_addr":"10.20.193.197:6817/2617788373","heartbeat_front_addr":"10.20.193.197:6815/2617788373","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2026-02-26T03:01:11.283523+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2026-02-26T03:01:11.607888+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2026-02-26T03:01:11.277139+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-26T03:01:33.015 DEBUG:teuthology.orchestra.run.trial197:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg map 2.0 --format=json 2026-02-26T03:01:33.173 INFO:teuthology.orchestra.run.trial197.stdout: 2026-02-26T03:01:33.182 INFO:teuthology.orchestra.run.trial197.stdout:{"epoch":19,"raw_pgid":"2.0","pgid":"2.0","up":[2,1],"acting":[2,1]} 2026-02-26T03:01:33.182 INFO:tasks.divergent_priors2:primary and soon to be divergent is 2 2026-02-26T03:01:33.182 INFO:tasks.divergent_priors2:writing initial objects 2026-02-26T03:01:33.182 INFO:tasks.util.rados:rados -p foo put existing_0 /etc/fstab 2026-02-26T03:01:33.182 DEBUG:teuthology.orchestra.run.trial197:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p foo put existing_0 /etc/fstab 2026-02-26T03:01:33.678 INFO:tasks.util.rados:rados -p foo put existing_1 /etc/fstab 2026-02-26T03:01:33.678 DEBUG:teuthology.orchestra.run.trial197:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p foo put existing_1 /etc/fstab 2026-02-26T03:01:33.716 INFO:tasks.util.rados:rados -p foo put existing_2 /etc/fstab 2026-02-26T03:01:33.716 DEBUG:teuthology.orchestra.run.trial197:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p foo put existing_2 /etc/fstab 2026-02-26T03:01:33.794 INFO:tasks.util.rados:rados -p foo put existing_3 /etc/fstab 2026-02-26T03:01:33.794 DEBUG:teuthology.orchestra.run.trial197:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p foo put existing_3 /etc/fstab 2026-02-26T03:01:33.872 INFO:tasks.util.rados:rados -p foo put existing_4 /etc/fstab 2026-02-26T03:01:33.872 DEBUG:teuthology.orchestra.run.trial197:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p foo put existing_4 /etc/fstab 2026-02-26T03:01:33.950 INFO:tasks.util.rados:rados -p foo put existing_5 /etc/fstab 2026-02-26T03:01:33.951 DEBUG:teuthology.orchestra.run.trial197:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p foo put existing_5 /etc/fstab 2026-02-26T03:01:34.028 INFO:tasks.util.rados:rados -p foo put existing_6 /etc/fstab 2026-02-26T03:01:34.028 DEBUG:teuthology.orchestra.run.trial197:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p foo put existing_6 /etc/fstab 2026-02-26T03:01:34.107 INFO:tasks.util.rados:rados -p foo put existing_7 /etc/fstab 2026-02-26T03:01:34.107 DEBUG:teuthology.orchestra.run.trial197:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p foo put existing_7 /etc/fstab 2026-02-26T03:01:34.186 INFO:tasks.util.rados:rados -p foo put existing_8 /etc/fstab 2026-02-26T03:01:34.186 DEBUG:teuthology.orchestra.run.trial197:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p foo put existing_8 /etc/fstab 2026-02-26T03:01:34.264 INFO:tasks.util.rados:rados -p foo put existing_9 /etc/fstab 2026-02-26T03:01:34.265 DEBUG:teuthology.orchestra.run.trial197:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p foo put existing_9 /etc/fstab 2026-02-26T03:01:34.341 INFO:tasks.util.rados:rados -p foo put existing_10 /etc/fstab 2026-02-26T03:01:34.342 DEBUG:teuthology.orchestra.run.trial197:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p foo put existing_10 /etc/fstab 2026-02-26T03:01:34.419 INFO:tasks.util.rados:rados -p foo put existing_11 /etc/fstab 2026-02-26T03:01:34.419 DEBUG:teuthology.orchestra.run.trial197:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p foo put existing_11 /etc/fstab 2026-02-26T03:01:34.496 INFO:tasks.util.rados:rados -p foo put existing_12 /etc/fstab 2026-02-26T03:01:34.496 DEBUG:teuthology.orchestra.run.trial197:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p foo put existing_12 /etc/fstab 2026-02-26T03:01:34.573 INFO:tasks.util.rados:rados -p foo put existing_13 /etc/fstab 2026-02-26T03:01:34.573 DEBUG:teuthology.orchestra.run.trial197:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p foo put existing_13 /etc/fstab 2026-02-26T03:01:34.651 INFO:tasks.util.rados:rados -p foo put existing_14 /etc/fstab 2026-02-26T03:01:34.651 DEBUG:teuthology.orchestra.run.trial197:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p foo put existing_14 /etc/fstab 2026-02-26T03:01:34.729 INFO:tasks.util.rados:rados -p foo put existing_15 /etc/fstab 2026-02-26T03:01:34.729 DEBUG:teuthology.orchestra.run.trial197:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p foo put existing_15 /etc/fstab 2026-02-26T03:01:34.805 INFO:tasks.util.rados:rados -p foo put existing_16 /etc/fstab 2026-02-26T03:01:34.805 DEBUG:teuthology.orchestra.run.trial197:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p foo put existing_16 /etc/fstab 2026-02-26T03:01:34.882 INFO:tasks.util.rados:rados -p foo put existing_17 /etc/fstab 2026-02-26T03:01:34.882 DEBUG:teuthology.orchestra.run.trial197:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p foo put existing_17 /etc/fstab 2026-02-26T03:01:34.961 INFO:tasks.util.rados:rados -p foo put existing_18 /etc/fstab 2026-02-26T03:01:34.961 DEBUG:teuthology.orchestra.run.trial197:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p foo put existing_18 /etc/fstab 2026-02-26T03:01:35.037 INFO:tasks.util.rados:rados -p foo put existing_19 /etc/fstab 2026-02-26T03:01:35.037 DEBUG:teuthology.orchestra.run.trial197:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p foo put existing_19 /etc/fstab 2026-02-26T03:01:35.115 INFO:tasks.util.rados:rados -p foo put existing_20 /etc/fstab 2026-02-26T03:01:35.115 DEBUG:teuthology.orchestra.run.trial197:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p foo put existing_20 /etc/fstab 2026-02-26T03:01:35.194 INFO:tasks.util.rados:rados -p foo put existing_21 /etc/fstab 2026-02-26T03:01:35.194 DEBUG:teuthology.orchestra.run.trial197:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p foo put existing_21 /etc/fstab 2026-02-26T03:01:35.272 INFO:tasks.util.rados:rados -p foo put existing_22 /etc/fstab 2026-02-26T03:01:35.272 DEBUG:teuthology.orchestra.run.trial197:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p foo put existing_22 /etc/fstab 2026-02-26T03:01:35.348 INFO:tasks.util.rados:rados -p foo put existing_23 /etc/fstab 2026-02-26T03:01:35.348 DEBUG:teuthology.orchestra.run.trial197:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p foo put existing_23 /etc/fstab 2026-02-26T03:01:35.427 INFO:tasks.util.rados:rados -p foo put existing_24 /etc/fstab 2026-02-26T03:01:35.427 DEBUG:teuthology.orchestra.run.trial197:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p foo put existing_24 /etc/fstab 2026-02-26T03:01:35.504 INFO:tasks.util.rados:rados -p foo put existing_25 /etc/fstab 2026-02-26T03:01:35.504 DEBUG:teuthology.orchestra.run.trial197:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p foo put existing_25 /etc/fstab 2026-02-26T03:01:35.580 INFO:tasks.util.rados:rados -p foo put existing_26 /etc/fstab 2026-02-26T03:01:35.581 DEBUG:teuthology.orchestra.run.trial197:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p foo put existing_26 /etc/fstab 2026-02-26T03:01:35.657 INFO:tasks.util.rados:rados -p foo put existing_27 /etc/fstab 2026-02-26T03:01:35.657 DEBUG:teuthology.orchestra.run.trial197:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p foo put existing_27 /etc/fstab 2026-02-26T03:01:35.734 INFO:tasks.util.rados:rados -p foo put existing_28 /etc/fstab 2026-02-26T03:01:35.734 DEBUG:teuthology.orchestra.run.trial197:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p foo put existing_28 /etc/fstab 2026-02-26T03:01:35.811 INFO:tasks.util.rados:rados -p foo put existing_29 /etc/fstab 2026-02-26T03:01:35.811 DEBUG:teuthology.orchestra.run.trial197:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p foo put existing_29 /etc/fstab 2026-02-26T03:01:35.889 INFO:tasks.util.rados:rados -p foo put existing_30 /etc/fstab 2026-02-26T03:01:35.889 DEBUG:teuthology.orchestra.run.trial197:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p foo put existing_30 /etc/fstab 2026-02-26T03:01:35.967 INFO:tasks.util.rados:rados -p foo put existing_31 /etc/fstab 2026-02-26T03:01:35.968 DEBUG:teuthology.orchestra.run.trial197:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p foo put existing_31 /etc/fstab 2026-02-26T03:01:36.045 INFO:tasks.util.rados:rados -p foo put existing_32 /etc/fstab 2026-02-26T03:01:36.046 DEBUG:teuthology.orchestra.run.trial197:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p foo put existing_32 /etc/fstab 2026-02-26T03:01:36.124 INFO:tasks.util.rados:rados -p foo put existing_33 /etc/fstab 2026-02-26T03:01:36.124 DEBUG:teuthology.orchestra.run.trial197:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p foo put existing_33 /etc/fstab 2026-02-26T03:01:36.202 INFO:tasks.util.rados:rados -p foo put existing_34 /etc/fstab 2026-02-26T03:01:36.202 DEBUG:teuthology.orchestra.run.trial197:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p foo put existing_34 /etc/fstab 2026-02-26T03:01:36.279 INFO:tasks.util.rados:rados -p foo put existing_35 /etc/fstab 2026-02-26T03:01:36.280 DEBUG:teuthology.orchestra.run.trial197:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p foo put existing_35 /etc/fstab 2026-02-26T03:01:36.356 INFO:tasks.util.rados:rados -p foo put existing_36 /etc/fstab 2026-02-26T03:01:36.357 DEBUG:teuthology.orchestra.run.trial197:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p foo put existing_36 /etc/fstab 2026-02-26T03:01:36.433 INFO:tasks.util.rados:rados -p foo put existing_37 /etc/fstab 2026-02-26T03:01:36.434 DEBUG:teuthology.orchestra.run.trial197:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p foo put existing_37 /etc/fstab 2026-02-26T03:01:36.510 INFO:tasks.util.rados:rados -p foo put existing_38 /etc/fstab 2026-02-26T03:01:36.511 DEBUG:teuthology.orchestra.run.trial197:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p foo put existing_38 /etc/fstab 2026-02-26T03:01:36.588 INFO:tasks.util.rados:rados -p foo put existing_39 /etc/fstab 2026-02-26T03:01:36.588 DEBUG:teuthology.orchestra.run.trial197:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p foo put existing_39 /etc/fstab 2026-02-26T03:01:36.667 INFO:tasks.util.rados:rados -p foo put existing_40 /etc/fstab 2026-02-26T03:01:36.667 DEBUG:teuthology.orchestra.run.trial197:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p foo put existing_40 /etc/fstab 2026-02-26T03:01:36.745 INFO:tasks.util.rados:rados -p foo put existing_41 /etc/fstab 2026-02-26T03:01:36.746 DEBUG:teuthology.orchestra.run.trial197:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p foo put existing_41 /etc/fstab 2026-02-26T03:01:36.824 INFO:tasks.util.rados:rados -p foo put existing_42 /etc/fstab 2026-02-26T03:01:36.824 DEBUG:teuthology.orchestra.run.trial197:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p foo put existing_42 /etc/fstab 2026-02-26T03:01:36.899 INFO:tasks.util.rados:rados -p foo put existing_43 /etc/fstab 2026-02-26T03:01:36.900 DEBUG:teuthology.orchestra.run.trial197:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p foo put existing_43 /etc/fstab 2026-02-26T03:01:36.978 INFO:tasks.util.rados:rados -p foo put existing_44 /etc/fstab 2026-02-26T03:01:36.979 DEBUG:teuthology.orchestra.run.trial197:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p foo put existing_44 /etc/fstab 2026-02-26T03:01:37.056 INFO:tasks.util.rados:rados -p foo put existing_45 /etc/fstab 2026-02-26T03:01:37.056 DEBUG:teuthology.orchestra.run.trial197:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p foo put existing_45 /etc/fstab 2026-02-26T03:01:37.134 INFO:tasks.util.rados:rados -p foo put existing_46 /etc/fstab 2026-02-26T03:01:37.135 DEBUG:teuthology.orchestra.run.trial197:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p foo put existing_46 /etc/fstab 2026-02-26T03:01:37.213 INFO:tasks.util.rados:rados -p foo put existing_47 /etc/fstab 2026-02-26T03:01:37.213 DEBUG:teuthology.orchestra.run.trial197:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p foo put existing_47 /etc/fstab 2026-02-26T03:01:37.290 INFO:tasks.util.rados:rados -p foo put existing_48 /etc/fstab 2026-02-26T03:01:37.290 DEBUG:teuthology.orchestra.run.trial197:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p foo put existing_48 /etc/fstab 2026-02-26T03:01:37.368 INFO:tasks.util.rados:rados -p foo put existing_49 /etc/fstab 2026-02-26T03:01:37.368 DEBUG:teuthology.orchestra.run.trial197:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p foo put existing_49 /etc/fstab 2026-02-26T03:01:37.445 INFO:tasks.util.rados:rados -p foo put existing_50 /etc/fstab 2026-02-26T03:01:37.446 DEBUG:teuthology.orchestra.run.trial197:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p foo put existing_50 /etc/fstab 2026-02-26T03:01:37.725 INFO:tasks.util.rados:rados -p foo put existing_51 /etc/fstab 2026-02-26T03:01:37.726 DEBUG:teuthology.orchestra.run.trial197:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p foo put existing_51 /etc/fstab 2026-02-26T03:01:37.764 INFO:tasks.util.rados:rados -p foo put existing_52 /etc/fstab 2026-02-26T03:01:37.764 DEBUG:teuthology.orchestra.run.trial197:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p foo put existing_52 /etc/fstab 2026-02-26T03:01:37.841 INFO:tasks.util.rados:rados -p foo put existing_53 /etc/fstab 2026-02-26T03:01:37.841 DEBUG:teuthology.orchestra.run.trial197:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p foo put existing_53 /etc/fstab 2026-02-26T03:01:37.918 INFO:tasks.util.rados:rados -p foo put existing_54 /etc/fstab 2026-02-26T03:01:37.918 DEBUG:teuthology.orchestra.run.trial197:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p foo put existing_54 /etc/fstab 2026-02-26T03:01:37.996 INFO:tasks.util.rados:rados -p foo put existing_55 /etc/fstab 2026-02-26T03:01:37.996 DEBUG:teuthology.orchestra.run.trial197:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p foo put existing_55 /etc/fstab 2026-02-26T03:01:38.074 INFO:tasks.util.rados:rados -p foo put existing_56 /etc/fstab 2026-02-26T03:01:38.074 DEBUG:teuthology.orchestra.run.trial197:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p foo put existing_56 /etc/fstab 2026-02-26T03:01:38.152 INFO:tasks.util.rados:rados -p foo put existing_57 /etc/fstab 2026-02-26T03:01:38.152 DEBUG:teuthology.orchestra.run.trial197:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p foo put existing_57 /etc/fstab 2026-02-26T03:01:38.229 INFO:tasks.util.rados:rados -p foo put existing_58 /etc/fstab 2026-02-26T03:01:38.229 DEBUG:teuthology.orchestra.run.trial197:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p foo put existing_58 /etc/fstab 2026-02-26T03:01:38.307 INFO:tasks.util.rados:rados -p foo put existing_59 /etc/fstab 2026-02-26T03:01:38.307 DEBUG:teuthology.orchestra.run.trial197:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p foo put existing_59 /etc/fstab 2026-02-26T03:01:38.384 INFO:tasks.util.rados:rados -p foo put existing_60 /etc/fstab 2026-02-26T03:01:38.384 DEBUG:teuthology.orchestra.run.trial197:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p foo put existing_60 /etc/fstab 2026-02-26T03:01:38.462 INFO:tasks.util.rados:rados -p foo put existing_61 /etc/fstab 2026-02-26T03:01:38.462 DEBUG:teuthology.orchestra.run.trial197:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p foo put existing_61 /etc/fstab 2026-02-26T03:01:38.540 INFO:tasks.util.rados:rados -p foo put existing_62 /etc/fstab 2026-02-26T03:01:38.540 DEBUG:teuthology.orchestra.run.trial197:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p foo put existing_62 /etc/fstab 2026-02-26T03:01:38.617 INFO:tasks.util.rados:rados -p foo put existing_63 /etc/fstab 2026-02-26T03:01:38.617 DEBUG:teuthology.orchestra.run.trial197:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p foo put existing_63 /etc/fstab 2026-02-26T03:01:38.696 INFO:tasks.util.rados:rados -p foo put existing_64 /etc/fstab 2026-02-26T03:01:38.696 DEBUG:teuthology.orchestra.run.trial197:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p foo put existing_64 /etc/fstab 2026-02-26T03:01:38.774 INFO:tasks.util.rados:rados -p foo put existing_65 /etc/fstab 2026-02-26T03:01:38.774 DEBUG:teuthology.orchestra.run.trial197:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p foo put existing_65 /etc/fstab 2026-02-26T03:01:38.851 INFO:tasks.util.rados:rados -p foo put existing_66 /etc/fstab 2026-02-26T03:01:38.852 DEBUG:teuthology.orchestra.run.trial197:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p foo put existing_66 /etc/fstab 2026-02-26T03:01:38.928 INFO:tasks.util.rados:rados -p foo put existing_67 /etc/fstab 2026-02-26T03:01:38.928 DEBUG:teuthology.orchestra.run.trial197:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p foo put existing_67 /etc/fstab 2026-02-26T03:01:39.005 INFO:tasks.util.rados:rados -p foo put existing_68 /etc/fstab 2026-02-26T03:01:39.006 DEBUG:teuthology.orchestra.run.trial197:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p foo put existing_68 /etc/fstab 2026-02-26T03:01:39.082 INFO:tasks.util.rados:rados -p foo put existing_69 /etc/fstab 2026-02-26T03:01:39.082 DEBUG:teuthology.orchestra.run.trial197:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p foo put existing_69 /etc/fstab 2026-02-26T03:01:39.160 INFO:tasks.util.rados:rados -p foo put existing_70 /etc/fstab 2026-02-26T03:01:39.160 DEBUG:teuthology.orchestra.run.trial197:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p foo put existing_70 /etc/fstab 2026-02-26T03:01:39.238 INFO:tasks.util.rados:rados -p foo put existing_71 /etc/fstab 2026-02-26T03:01:39.238 DEBUG:teuthology.orchestra.run.trial197:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p foo put existing_71 /etc/fstab 2026-02-26T03:01:39.315 INFO:tasks.util.rados:rados -p foo put existing_72 /etc/fstab 2026-02-26T03:01:39.315 DEBUG:teuthology.orchestra.run.trial197:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p foo put existing_72 /etc/fstab 2026-02-26T03:01:39.392 INFO:tasks.util.rados:rados -p foo put existing_73 /etc/fstab 2026-02-26T03:01:39.392 DEBUG:teuthology.orchestra.run.trial197:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p foo put existing_73 /etc/fstab 2026-02-26T03:01:39.470 INFO:tasks.util.rados:rados -p foo put existing_74 /etc/fstab 2026-02-26T03:01:39.471 DEBUG:teuthology.orchestra.run.trial197:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p foo put existing_74 /etc/fstab 2026-02-26T03:01:39.548 INFO:tasks.util.rados:rados -p foo put existing_75 /etc/fstab 2026-02-26T03:01:39.548 DEBUG:teuthology.orchestra.run.trial197:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p foo put existing_75 /etc/fstab 2026-02-26T03:01:39.626 INFO:tasks.util.rados:rados -p foo put existing_76 /etc/fstab 2026-02-26T03:01:39.627 DEBUG:teuthology.orchestra.run.trial197:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p foo put existing_76 /etc/fstab 2026-02-26T03:01:39.704 INFO:tasks.util.rados:rados -p foo put existing_77 /etc/fstab 2026-02-26T03:01:39.705 DEBUG:teuthology.orchestra.run.trial197:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p foo put existing_77 /etc/fstab 2026-02-26T03:01:39.782 INFO:tasks.util.rados:rados -p foo put existing_78 /etc/fstab 2026-02-26T03:01:39.783 DEBUG:teuthology.orchestra.run.trial197:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p foo put existing_78 /etc/fstab 2026-02-26T03:01:39.861 INFO:tasks.util.rados:rados -p foo put existing_79 /etc/fstab 2026-02-26T03:01:39.862 DEBUG:teuthology.orchestra.run.trial197:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p foo put existing_79 /etc/fstab 2026-02-26T03:01:39.938 INFO:tasks.util.rados:rados -p foo put existing_80 /etc/fstab 2026-02-26T03:01:39.938 DEBUG:teuthology.orchestra.run.trial197:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p foo put existing_80 /etc/fstab 2026-02-26T03:01:40.016 INFO:tasks.util.rados:rados -p foo put existing_81 /etc/fstab 2026-02-26T03:01:40.016 DEBUG:teuthology.orchestra.run.trial197:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p foo put existing_81 /etc/fstab 2026-02-26T03:01:40.094 INFO:tasks.util.rados:rados -p foo put existing_82 /etc/fstab 2026-02-26T03:01:40.094 DEBUG:teuthology.orchestra.run.trial197:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p foo put existing_82 /etc/fstab 2026-02-26T03:01:40.173 INFO:tasks.util.rados:rados -p foo put existing_83 /etc/fstab 2026-02-26T03:01:40.173 DEBUG:teuthology.orchestra.run.trial197:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p foo put existing_83 /etc/fstab 2026-02-26T03:01:40.250 INFO:tasks.util.rados:rados -p foo put existing_84 /etc/fstab 2026-02-26T03:01:40.251 DEBUG:teuthology.orchestra.run.trial197:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p foo put existing_84 /etc/fstab 2026-02-26T03:01:40.327 INFO:tasks.util.rados:rados -p foo put existing_85 /etc/fstab 2026-02-26T03:01:40.327 DEBUG:teuthology.orchestra.run.trial197:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p foo put existing_85 /etc/fstab 2026-02-26T03:01:40.405 INFO:tasks.util.rados:rados -p foo put existing_86 /etc/fstab 2026-02-26T03:01:40.405 DEBUG:teuthology.orchestra.run.trial197:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p foo put existing_86 /etc/fstab 2026-02-26T03:01:40.482 INFO:tasks.util.rados:rados -p foo put existing_87 /etc/fstab 2026-02-26T03:01:40.482 DEBUG:teuthology.orchestra.run.trial197:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p foo put existing_87 /etc/fstab 2026-02-26T03:01:40.560 INFO:tasks.util.rados:rados -p foo put existing_88 /etc/fstab 2026-02-26T03:01:40.560 DEBUG:teuthology.orchestra.run.trial197:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p foo put existing_88 /etc/fstab 2026-02-26T03:01:40.638 INFO:tasks.util.rados:rados -p foo put existing_89 /etc/fstab 2026-02-26T03:01:40.638 DEBUG:teuthology.orchestra.run.trial197:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p foo put existing_89 /etc/fstab 2026-02-26T03:01:40.714 INFO:tasks.util.rados:rados -p foo put existing_90 /etc/fstab 2026-02-26T03:01:40.714 DEBUG:teuthology.orchestra.run.trial197:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p foo put existing_90 /etc/fstab 2026-02-26T03:01:40.791 INFO:tasks.util.rados:rados -p foo put existing_91 /etc/fstab 2026-02-26T03:01:40.791 DEBUG:teuthology.orchestra.run.trial197:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p foo put existing_91 /etc/fstab 2026-02-26T03:01:40.870 INFO:tasks.util.rados:rados -p foo put existing_92 /etc/fstab 2026-02-26T03:01:40.870 DEBUG:teuthology.orchestra.run.trial197:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p foo put existing_92 /etc/fstab 2026-02-26T03:01:40.947 INFO:tasks.util.rados:rados -p foo put existing_93 /etc/fstab 2026-02-26T03:01:40.947 DEBUG:teuthology.orchestra.run.trial197:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p foo put existing_93 /etc/fstab 2026-02-26T03:01:41.026 INFO:tasks.util.rados:rados -p foo put existing_94 /etc/fstab 2026-02-26T03:01:41.026 DEBUG:teuthology.orchestra.run.trial197:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p foo put existing_94 /etc/fstab 2026-02-26T03:01:41.104 INFO:tasks.util.rados:rados -p foo put existing_95 /etc/fstab 2026-02-26T03:01:41.104 DEBUG:teuthology.orchestra.run.trial197:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p foo put existing_95 /etc/fstab 2026-02-26T03:01:41.182 INFO:tasks.util.rados:rados -p foo put existing_96 /etc/fstab 2026-02-26T03:01:41.182 DEBUG:teuthology.orchestra.run.trial197:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p foo put existing_96 /etc/fstab 2026-02-26T03:01:41.260 INFO:tasks.util.rados:rados -p foo put existing_97 /etc/fstab 2026-02-26T03:01:41.260 DEBUG:teuthology.orchestra.run.trial197:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p foo put existing_97 /etc/fstab 2026-02-26T03:01:41.338 INFO:tasks.util.rados:rados -p foo put existing_98 /etc/fstab 2026-02-26T03:01:41.339 DEBUG:teuthology.orchestra.run.trial197:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p foo put existing_98 /etc/fstab 2026-02-26T03:01:41.612 INFO:teuthology.orchestra.run.trial197.stderr:2026-02-26T03:01:41.611+0000 7f9431cf4640 0 --2- 10.20.193.197:0/4069381747 >> [v2:10.20.193.197:3300/0,v1:10.20.193.197:6789/0] conn(0x56135369a5a0 0x561353683ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request get_initial_auth_request returned -2 2026-02-26T03:01:41.619 INFO:tasks.util.rados:rados -p foo put existing_99 /etc/fstab 2026-02-26T03:01:41.619 DEBUG:teuthology.orchestra.run.trial197:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p foo put existing_99 /etc/fstab 2026-02-26T03:01:41.853 INFO:teuthology.orchestra.run.trial197.stderr:2026-02-26T03:01:41.852+0000 7f6b97fff640 0 --2- 10.20.193.197:0/151406957 >> [v2:10.20.193.197:3300/0,v1:10.20.193.197:6789/0] conn(0x55c88500a5a0 0x55c884fff4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request get_initial_auth_request returned -2 2026-02-26T03:01:41.859 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2026-02-26T03:01:41.860 DEBUG:teuthology.orchestra.run.trial197:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-26T03:01:42.011 INFO:teuthology.orchestra.run.trial197.stdout: 2026-02-26T03:01:42.012 INFO:teuthology.orchestra.run.trial197.stderr:dumped all 2026-02-26T03:01:42.019 INFO:teuthology.orchestra.run.trial197.stdout:{"pg_ready":true,"pg_map":{"version":32,"stamp":"2026-02-26T03:01:40.382061+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":31918,"num_objects":51,"num_object_clones":0,"num_object_copies":102,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":51,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":51,"num_write_kb":51,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":81804,"kb_used_data":920,"kb_used_omap":4,"kb_used_meta":80827,"kb_avail":283033716,"statfs":{"total":289910292480,"available":289826525184,"internally_reserved":0,"allocated":942080,"data_stored":219683,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4782,"internal_metadata":82767186},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":31899,"num_objects":49,"num_object_clones":0,"num_object_copies":98,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":49,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":49,"num_write_kb":49,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.000890"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":31,"reported_epoch":20,"state":"active+clean","last_fresh":"2026-02-26T03:01:34.165011+0000","last_change":"2026-02-26T03:01:21.671776+0000","last_active":"2026-02-26T03:01:34.165011+0000","last_peered":"2026-02-26T03:01:34.165011+0000","last_clean":"2026-02-26T03:01:34.165011+0000","last_became_active":"2026-02-26T03:01:19.668398+0000","last_became_peered":"2026-02-26T03:01:19.668398+0000","last_unstale":"2026-02-26T03:01:34.165011+0000","last_undegraded":"2026-02-26T03:01:34.165011+0000","last_fullsized":"2026-02-26T03:01:34.165011+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-26T03:01:18.659251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-26T03:01:18.659251+0000","last_clean_scrub_stamp":"2026-02-26T03:01:18.659251+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-27T04:08:38.487713+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00060011299999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":31,"reported_epoch":20,"state":"active+clean","last_fresh":"2026-02-26T03:01:34.165008+0000","last_change":"2026-02-26T03:01:21.671905+0000","last_active":"2026-02-26T03:01:34.165008+0000","last_peered":"2026-02-26T03:01:34.165008+0000","last_clean":"2026-02-26T03:01:34.165008+0000","last_became_active":"2026-02-26T03:01:19.669316+0000","last_became_peered":"2026-02-26T03:01:19.669316+0000","last_unstale":"2026-02-26T03:01:34.165008+0000","last_undegraded":"2026-02-26T03:01:34.165008+0000","last_fullsized":"2026-02-26T03:01:34.165008+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-26T03:01:18.659251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-26T03:01:18.659251+0000","last_clean_scrub_stamp":"2026-02-26T03:01:18.659251+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-27T13:43:20.762574+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00068042299999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":31,"reported_epoch":20,"state":"active+clean","last_fresh":"2026-02-26T03:01:34.183323+0000","last_change":"2026-02-26T03:01:21.669362+0000","last_active":"2026-02-26T03:01:34.183323+0000","last_peered":"2026-02-26T03:01:34.183323+0000","last_clean":"2026-02-26T03:01:34.183323+0000","last_became_active":"2026-02-26T03:01:19.669313+0000","last_became_peered":"2026-02-26T03:01:19.669313+0000","last_unstale":"2026-02-26T03:01:34.183323+0000","last_undegraded":"2026-02-26T03:01:34.183323+0000","last_fullsized":"2026-02-26T03:01:34.183323+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-26T03:01:18.659251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-26T03:01:18.659251+0000","last_clean_scrub_stamp":"2026-02-26T03:01:18.659251+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-27T10:49:28.489885+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00034704899999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":31,"reported_epoch":20,"state":"active+clean","last_fresh":"2026-02-26T03:01:34.164997+0000","last_change":"2026-02-26T03:01:21.671647+0000","last_active":"2026-02-26T03:01:34.164997+0000","last_peered":"2026-02-26T03:01:34.164997+0000","last_clean":"2026-02-26T03:01:34.164997+0000","last_became_active":"2026-02-26T03:01:19.669171+0000","last_became_peered":"2026-02-26T03:01:19.669171+0000","last_unstale":"2026-02-26T03:01:34.164997+0000","last_undegraded":"2026-02-26T03:01:34.164997+0000","last_fullsized":"2026-02-26T03:01:34.164997+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-26T03:01:18.659251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-26T03:01:18.659251+0000","last_clean_scrub_stamp":"2026-02-26T03:01:18.659251+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-27T04:13:58.662319+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000482733,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"20'49","reported_seq":61,"reported_epoch":20,"state":"active+clean","last_fresh":"2026-02-26T03:01:37.288552+0000","last_change":"2026-02-26T03:01:33.675008+0000","last_active":"2026-02-26T03:01:37.288552+0000","last_peered":"2026-02-26T03:01:37.288552+0000","last_clean":"2026-02-26T03:01:37.288552+0000","last_became_active":"2026-02-26T03:01:33.674358+0000","last_became_peered":"2026-02-26T03:01:33.674358+0000","last_unstale":"2026-02-26T03:01:37.288552+0000","last_undegraded":"2026-02-26T03:01:37.288552+0000","last_fullsized":"2026-02-26T03:01:37.288552+0000","mapping_epoch":18,"log_start":"20'35","ondisk_log_start":"20'35","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-26T03:01:32.157530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-26T03:01:32.157530+0000","last_clean_scrub_stamp":"2026-02-26T03:01:32.157530+0000","objects_scrubbed":0,"log_size":14,"log_dups_size":35,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-27T13:36:02.852922+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31899,"num_objects":49,"num_object_clones":0,"num_object_copies":98,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":49,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":49,"num_write_kb":49,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"11'1","reported_seq":32,"reported_epoch":20,"state":"active+clean","last_fresh":"2026-02-26T03:01:34.164988+0000","last_change":"2026-02-26T03:01:21.671498+0000","last_active":"2026-02-26T03:01:34.164988+0000","last_peered":"2026-02-26T03:01:34.164988+0000","last_clean":"2026-02-26T03:01:34.164988+0000","last_became_active":"2026-02-26T03:01:19.668677+0000","last_became_peered":"2026-02-26T03:01:19.668677+0000","last_unstale":"2026-02-26T03:01:34.164988+0000","last_undegraded":"2026-02-26T03:01:34.164988+0000","last_fullsized":"2026-02-26T03:01:34.164988+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-26T03:01:18.659251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-26T03:01:18.659251+0000","last_clean_scrub_stamp":"2026-02-26T03:01:18.659251+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-27T09:30:44.521018+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00034461499999999999,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"13'2","reported_seq":34,"reported_epoch":20,"state":"active+clean","last_fresh":"2026-02-26T03:01:34.716107+0000","last_change":"2026-02-26T03:01:21.670052+0000","last_active":"2026-02-26T03:01:34.716107+0000","last_peered":"2026-02-26T03:01:34.716107+0000","last_clean":"2026-02-26T03:01:34.716107+0000","last_became_active":"2026-02-26T03:01:19.668415+0000","last_became_peered":"2026-02-26T03:01:19.668415+0000","last_unstale":"2026-02-26T03:01:34.716107+0000","last_undegraded":"2026-02-26T03:01:34.716107+0000","last_fullsized":"2026-02-26T03:01:34.716107+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-26T03:01:18.659251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-26T03:01:18.659251+0000","last_clean_scrub_stamp":"2026-02-26T03:01:18.659251+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-27T12:03:13.020042+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00054467999999999997,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":31,"reported_epoch":20,"state":"active+clean","last_fresh":"2026-02-26T03:01:34.183322+0000","last_change":"2026-02-26T03:01:21.669370+0000","last_active":"2026-02-26T03:01:34.183322+0000","last_peered":"2026-02-26T03:01:34.183322+0000","last_clean":"2026-02-26T03:01:34.183322+0000","last_became_active":"2026-02-26T03:01:19.669112+0000","last_became_peered":"2026-02-26T03:01:19.669112+0000","last_unstale":"2026-02-26T03:01:34.183322+0000","last_undegraded":"2026-02-26T03:01:34.183322+0000","last_fullsized":"2026-02-26T03:01:34.183322+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-26T03:01:18.659251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-26T03:01:18.659251+0000","last_clean_scrub_stamp":"2026-02-26T03:01:18.659251+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-27T14:09:53.100227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00037735600000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":31,"reported_epoch":20,"state":"active+clean","last_fresh":"2026-02-26T03:01:34.164969+0000","last_change":"2026-02-26T03:01:21.671520+0000","last_active":"2026-02-26T03:01:34.164969+0000","last_peered":"2026-02-26T03:01:34.164969+0000","last_clean":"2026-02-26T03:01:34.164969+0000","last_became_active":"2026-02-26T03:01:19.669362+0000","last_became_peered":"2026-02-26T03:01:19.669362+0000","last_unstale":"2026-02-26T03:01:34.164969+0000","last_undegraded":"2026-02-26T03:01:34.164969+0000","last_fullsized":"2026-02-26T03:01:34.164969+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-26T03:01:18.659251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-26T03:01:18.659251+0000","last_clean_scrub_stamp":"2026-02-26T03:01:18.659251+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-27T03:50:25.293079+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00037062400000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":31899,"num_objects":49,"num_object_clones":0,"num_object_copies":98,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":49,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":49,"num_write_kb":49,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":495616,"data_stored":78771,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":14,"ondisk_log_size":14,"up":2,"acting":2,"num_store_stats":2},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":1,"up_from":9,"seq":38654705673,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27384,"kb_used_data":424,"kb_used_omap":1,"kb_used_meta":26942,"kb_avail":94344456,"statfs":{"total":96636764160,"available":96608722944,"internally_reserved":0,"allocated":434176,"data_stored":92113,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27589062},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":8,"seq":34359738377,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27284,"kb_used_data":324,"kb_used_omap":1,"kb_used_meta":26942,"kb_avail":94344556,"statfs":{"total":96636764160,"available":96608825344,"internally_reserved":0,"allocated":331776,"data_stored":76470,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27589062},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738377,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27136,"kb_used_data":172,"kb_used_omap":1,"kb_used_meta":26942,"kb_avail":94344704,"statfs":{"total":96636764160,"available":96608976896,"internally_reserved":0,"allocated":176128,"data_stored":51100,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27589062},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":294912,"data_stored":46872,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":200704,"data_stored":31899,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-26T03:01:42.021 DEBUG:teuthology.orchestra.run.trial197:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-26T03:01:42.178 INFO:teuthology.orchestra.run.trial197.stdout: 2026-02-26T03:01:42.179 INFO:teuthology.orchestra.run.trial197.stderr:dumped all 2026-02-26T03:01:42.187 INFO:teuthology.orchestra.run.trial197.stdout:{"pg_ready":true,"pg_map":{"version":32,"stamp":"2026-02-26T03:01:40.382061+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":31918,"num_objects":51,"num_object_clones":0,"num_object_copies":102,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":51,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":51,"num_write_kb":51,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":81804,"kb_used_data":920,"kb_used_omap":4,"kb_used_meta":80827,"kb_avail":283033716,"statfs":{"total":289910292480,"available":289826525184,"internally_reserved":0,"allocated":942080,"data_stored":219683,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4782,"internal_metadata":82767186},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":31899,"num_objects":49,"num_object_clones":0,"num_object_copies":98,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":49,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":49,"num_write_kb":49,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.000890"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":31,"reported_epoch":20,"state":"active+clean","last_fresh":"2026-02-26T03:01:34.165011+0000","last_change":"2026-02-26T03:01:21.671776+0000","last_active":"2026-02-26T03:01:34.165011+0000","last_peered":"2026-02-26T03:01:34.165011+0000","last_clean":"2026-02-26T03:01:34.165011+0000","last_became_active":"2026-02-26T03:01:19.668398+0000","last_became_peered":"2026-02-26T03:01:19.668398+0000","last_unstale":"2026-02-26T03:01:34.165011+0000","last_undegraded":"2026-02-26T03:01:34.165011+0000","last_fullsized":"2026-02-26T03:01:34.165011+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-26T03:01:18.659251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-26T03:01:18.659251+0000","last_clean_scrub_stamp":"2026-02-26T03:01:18.659251+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-27T04:08:38.487713+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00060011299999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":31,"reported_epoch":20,"state":"active+clean","last_fresh":"2026-02-26T03:01:34.165008+0000","last_change":"2026-02-26T03:01:21.671905+0000","last_active":"2026-02-26T03:01:34.165008+0000","last_peered":"2026-02-26T03:01:34.165008+0000","last_clean":"2026-02-26T03:01:34.165008+0000","last_became_active":"2026-02-26T03:01:19.669316+0000","last_became_peered":"2026-02-26T03:01:19.669316+0000","last_unstale":"2026-02-26T03:01:34.165008+0000","last_undegraded":"2026-02-26T03:01:34.165008+0000","last_fullsized":"2026-02-26T03:01:34.165008+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-26T03:01:18.659251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-26T03:01:18.659251+0000","last_clean_scrub_stamp":"2026-02-26T03:01:18.659251+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-27T13:43:20.762574+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00068042299999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":31,"reported_epoch":20,"state":"active+clean","last_fresh":"2026-02-26T03:01:34.183323+0000","last_change":"2026-02-26T03:01:21.669362+0000","last_active":"2026-02-26T03:01:34.183323+0000","last_peered":"2026-02-26T03:01:34.183323+0000","last_clean":"2026-02-26T03:01:34.183323+0000","last_became_active":"2026-02-26T03:01:19.669313+0000","last_became_peered":"2026-02-26T03:01:19.669313+0000","last_unstale":"2026-02-26T03:01:34.183323+0000","last_undegraded":"2026-02-26T03:01:34.183323+0000","last_fullsized":"2026-02-26T03:01:34.183323+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-26T03:01:18.659251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-26T03:01:18.659251+0000","last_clean_scrub_stamp":"2026-02-26T03:01:18.659251+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-27T10:49:28.489885+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00034704899999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":31,"reported_epoch":20,"state":"active+clean","last_fresh":"2026-02-26T03:01:34.164997+0000","last_change":"2026-02-26T03:01:21.671647+0000","last_active":"2026-02-26T03:01:34.164997+0000","last_peered":"2026-02-26T03:01:34.164997+0000","last_clean":"2026-02-26T03:01:34.164997+0000","last_became_active":"2026-02-26T03:01:19.669171+0000","last_became_peered":"2026-02-26T03:01:19.669171+0000","last_unstale":"2026-02-26T03:01:34.164997+0000","last_undegraded":"2026-02-26T03:01:34.164997+0000","last_fullsized":"2026-02-26T03:01:34.164997+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-26T03:01:18.659251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-26T03:01:18.659251+0000","last_clean_scrub_stamp":"2026-02-26T03:01:18.659251+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-27T04:13:58.662319+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000482733,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"20'49","reported_seq":61,"reported_epoch":20,"state":"active+clean","last_fresh":"2026-02-26T03:01:37.288552+0000","last_change":"2026-02-26T03:01:33.675008+0000","last_active":"2026-02-26T03:01:37.288552+0000","last_peered":"2026-02-26T03:01:37.288552+0000","last_clean":"2026-02-26T03:01:37.288552+0000","last_became_active":"2026-02-26T03:01:33.674358+0000","last_became_peered":"2026-02-26T03:01:33.674358+0000","last_unstale":"2026-02-26T03:01:37.288552+0000","last_undegraded":"2026-02-26T03:01:37.288552+0000","last_fullsized":"2026-02-26T03:01:37.288552+0000","mapping_epoch":18,"log_start":"20'35","ondisk_log_start":"20'35","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-26T03:01:32.157530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-26T03:01:32.157530+0000","last_clean_scrub_stamp":"2026-02-26T03:01:32.157530+0000","objects_scrubbed":0,"log_size":14,"log_dups_size":35,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-27T13:36:02.852922+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31899,"num_objects":49,"num_object_clones":0,"num_object_copies":98,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":49,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":49,"num_write_kb":49,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"11'1","reported_seq":32,"reported_epoch":20,"state":"active+clean","last_fresh":"2026-02-26T03:01:34.164988+0000","last_change":"2026-02-26T03:01:21.671498+0000","last_active":"2026-02-26T03:01:34.164988+0000","last_peered":"2026-02-26T03:01:34.164988+0000","last_clean":"2026-02-26T03:01:34.164988+0000","last_became_active":"2026-02-26T03:01:19.668677+0000","last_became_peered":"2026-02-26T03:01:19.668677+0000","last_unstale":"2026-02-26T03:01:34.164988+0000","last_undegraded":"2026-02-26T03:01:34.164988+0000","last_fullsized":"2026-02-26T03:01:34.164988+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-26T03:01:18.659251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-26T03:01:18.659251+0000","last_clean_scrub_stamp":"2026-02-26T03:01:18.659251+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-27T09:30:44.521018+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00034461499999999999,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"13'2","reported_seq":34,"reported_epoch":20,"state":"active+clean","last_fresh":"2026-02-26T03:01:34.716107+0000","last_change":"2026-02-26T03:01:21.670052+0000","last_active":"2026-02-26T03:01:34.716107+0000","last_peered":"2026-02-26T03:01:34.716107+0000","last_clean":"2026-02-26T03:01:34.716107+0000","last_became_active":"2026-02-26T03:01:19.668415+0000","last_became_peered":"2026-02-26T03:01:19.668415+0000","last_unstale":"2026-02-26T03:01:34.716107+0000","last_undegraded":"2026-02-26T03:01:34.716107+0000","last_fullsized":"2026-02-26T03:01:34.716107+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-26T03:01:18.659251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-26T03:01:18.659251+0000","last_clean_scrub_stamp":"2026-02-26T03:01:18.659251+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-27T12:03:13.020042+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00054467999999999997,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":31,"reported_epoch":20,"state":"active+clean","last_fresh":"2026-02-26T03:01:34.183322+0000","last_change":"2026-02-26T03:01:21.669370+0000","last_active":"2026-02-26T03:01:34.183322+0000","last_peered":"2026-02-26T03:01:34.183322+0000","last_clean":"2026-02-26T03:01:34.183322+0000","last_became_active":"2026-02-26T03:01:19.669112+0000","last_became_peered":"2026-02-26T03:01:19.669112+0000","last_unstale":"2026-02-26T03:01:34.183322+0000","last_undegraded":"2026-02-26T03:01:34.183322+0000","last_fullsized":"2026-02-26T03:01:34.183322+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-26T03:01:18.659251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-26T03:01:18.659251+0000","last_clean_scrub_stamp":"2026-02-26T03:01:18.659251+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-27T14:09:53.100227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00037735600000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":31,"reported_epoch":20,"state":"active+clean","last_fresh":"2026-02-26T03:01:34.164969+0000","last_change":"2026-02-26T03:01:21.671520+0000","last_active":"2026-02-26T03:01:34.164969+0000","last_peered":"2026-02-26T03:01:34.164969+0000","last_clean":"2026-02-26T03:01:34.164969+0000","last_became_active":"2026-02-26T03:01:19.669362+0000","last_became_peered":"2026-02-26T03:01:19.669362+0000","last_unstale":"2026-02-26T03:01:34.164969+0000","last_undegraded":"2026-02-26T03:01:34.164969+0000","last_fullsized":"2026-02-26T03:01:34.164969+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-26T03:01:18.659251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-26T03:01:18.659251+0000","last_clean_scrub_stamp":"2026-02-26T03:01:18.659251+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-27T03:50:25.293079+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00037062400000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":31899,"num_objects":49,"num_object_clones":0,"num_object_copies":98,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":49,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":49,"num_write_kb":49,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":495616,"data_stored":78771,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":14,"ondisk_log_size":14,"up":2,"acting":2,"num_store_stats":2},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":1,"up_from":9,"seq":38654705673,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27384,"kb_used_data":424,"kb_used_omap":1,"kb_used_meta":26942,"kb_avail":94344456,"statfs":{"total":96636764160,"available":96608722944,"internally_reserved":0,"allocated":434176,"data_stored":92113,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27589062},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":8,"seq":34359738377,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27284,"kb_used_data":324,"kb_used_omap":1,"kb_used_meta":26942,"kb_avail":94344556,"statfs":{"total":96636764160,"available":96608825344,"internally_reserved":0,"allocated":331776,"data_stored":76470,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27589062},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738377,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27136,"kb_used_data":172,"kb_used_omap":1,"kb_used_meta":26942,"kb_avail":94344704,"statfs":{"total":96636764160,"available":96608976896,"internally_reserved":0,"allocated":176128,"data_stored":51100,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27589062},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":294912,"data_stored":46872,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":200704,"data_stored":31899,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-26T03:01:42.189 INFO:tasks.ceph.ceph_manager.ceph:clean! 2026-02-26T03:01:42.189 INFO:tasks.divergent_priors2:blackholing osds [0, 1] 2026-02-26T03:01:42.189 DEBUG:teuthology.orchestra.run.trial197:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok config set objectstore_blackhole 1 2026-02-26T03:01:42.259 INFO:teuthology.orchestra.run.trial197.stdout:{ 2026-02-26T03:01:42.260 INFO:teuthology.orchestra.run.trial197.stdout: "success": "objectstore_blackhole = '' (not observed, change may require restart) " 2026-02-26T03:01:42.260 INFO:teuthology.orchestra.run.trial197.stdout:} 2026-02-26T03:01:42.265 DEBUG:teuthology.orchestra.run.trial197:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok config set objectstore_blackhole 1 2026-02-26T03:01:42.333 INFO:teuthology.orchestra.run.trial197.stdout:{ 2026-02-26T03:01:42.333 INFO:teuthology.orchestra.run.trial197.stdout: "success": "objectstore_blackhole = '' (not observed, change may require restart) " 2026-02-26T03:01:42.333 INFO:teuthology.orchestra.run.trial197.stdout:} 2026-02-26T03:01:42.339 INFO:tasks.divergent_priors2:writing divergent objects 2026-02-26T03:01:42.339 INFO:tasks.util.rados:rados -p foo put existing_0 /etc/resolv.conf 2026-02-26T03:01:42.339 DEBUG:teuthology.orchestra.run.trial197:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p foo put existing_0 /etc/resolv.conf 2026-02-26T03:01:42.341 INFO:tasks.util.rados:rados -p foo put existing_1 /etc/resolv.conf 2026-02-26T03:01:42.341 DEBUG:teuthology.orchestra.run.trial197:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p foo put existing_1 /etc/resolv.conf 2026-02-26T03:01:42.375 INFO:tasks.ceph.osd.1.trial197.stderr:2026-02-26T03:01:42.373+0000 7fa6b0914640 -1 bdev(0x55dc8e5fa000 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2026-02-26T03:01:42.375 INFO:tasks.ceph.osd.1.trial197.stderr:2026-02-26T03:01:42.373+0000 7fa6b0914640 -1 bdev(0x55dc8e5fa000 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2026-02-26T03:01:42.376 INFO:tasks.util.rados:rados -p foo put existing_2 /etc/resolv.conf 2026-02-26T03:01:42.376 DEBUG:teuthology.orchestra.run.trial197:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p foo put existing_2 /etc/resolv.conf 2026-02-26T03:01:42.420 INFO:tasks.util.rados:rados -p foo put existing_3 /etc/resolv.conf 2026-02-26T03:01:42.421 DEBUG:teuthology.orchestra.run.trial197:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p foo put existing_3 /etc/resolv.conf 2026-02-26T03:01:42.453 INFO:tasks.ceph.osd.1.trial197.stderr:2026-02-26T03:01:42.452+0000 7fa6b0914640 -1 bdev(0x55dc8e5fa000 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2026-02-26T03:01:42.453 INFO:tasks.ceph.osd.1.trial197.stderr:2026-02-26T03:01:42.452+0000 7fa6b0914640 -1 bdev(0x55dc8e5fa000 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2026-02-26T03:01:42.454 INFO:tasks.util.rados:rados -p foo put existing_4 /etc/resolv.conf 2026-02-26T03:01:42.454 DEBUG:teuthology.orchestra.run.trial197:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p foo put existing_4 /etc/resolv.conf 2026-02-26T03:01:42.488 INFO:tasks.ceph.osd.1.trial197.stderr:2026-02-26T03:01:42.487+0000 7fa6b0914640 -1 bdev(0x55dc8e5fa000 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2026-02-26T03:01:42.488 INFO:tasks.ceph.osd.1.trial197.stderr:2026-02-26T03:01:42.487+0000 7fa6b0914640 -1 bdev(0x55dc8e5fa000 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2026-02-26T03:01:42.489 INFO:tasks.divergent_priors2:remove divergent objects 2026-02-26T03:01:42.489 INFO:tasks.util.rados:rados -p foo rm existing_5 2026-02-26T03:01:42.489 DEBUG:teuthology.orchestra.run.trial197:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p foo rm existing_5 2026-02-26T03:01:42.523 INFO:tasks.ceph.osd.1.trial197.stderr:2026-02-26T03:01:42.521+0000 7fa6b0914640 -1 bdev(0x55dc8e5fa000 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2026-02-26T03:01:42.523 INFO:tasks.ceph.osd.1.trial197.stderr:2026-02-26T03:01:42.521+0000 7fa6b0914640 -1 bdev(0x55dc8e5fa000 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2026-02-26T03:01:42.524 INFO:tasks.util.rados:rados -p foo rm existing_6 2026-02-26T03:01:42.524 DEBUG:teuthology.orchestra.run.trial197:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p foo rm existing_6 2026-02-26T03:01:42.557 INFO:tasks.ceph.osd.1.trial197.stderr:2026-02-26T03:01:42.556+0000 7fa6b0914640 -1 bdev(0x55dc8e5fa000 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2026-02-26T03:01:42.558 INFO:tasks.ceph.osd.1.trial197.stderr:2026-02-26T03:01:42.556+0000 7fa6b0914640 -1 bdev(0x55dc8e5fa000 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2026-02-26T03:01:42.559 INFO:tasks.util.rados:rados -p foo rm existing_7 2026-02-26T03:01:42.559 DEBUG:teuthology.orchestra.run.trial197:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p foo rm existing_7 2026-02-26T03:01:42.591 INFO:tasks.ceph.osd.1.trial197.stderr:2026-02-26T03:01:42.589+0000 7fa6b0914640 -1 bdev(0x55dc8e5fa000 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2026-02-26T03:01:42.591 INFO:tasks.ceph.osd.1.trial197.stderr:2026-02-26T03:01:42.589+0000 7fa6b0914640 -1 bdev(0x55dc8e5fa000 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2026-02-26T03:01:42.591 INFO:tasks.util.rados:rados -p foo rm existing_8 2026-02-26T03:01:42.591 DEBUG:teuthology.orchestra.run.trial197:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p foo rm existing_8 2026-02-26T03:01:42.610 INFO:tasks.ceph.osd.1.trial197.stderr:2026-02-26T03:01:42.608+0000 7fa6b0914640 -1 bdev(0x55dc8e5fa000 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2026-02-26T03:01:42.610 INFO:tasks.ceph.osd.1.trial197.stderr:2026-02-26T03:01:42.608+0000 7fa6b0914640 -1 bdev(0x55dc8e5fa000 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2026-02-26T03:01:42.611 INFO:tasks.util.rados:rados -p foo rm existing_9 2026-02-26T03:01:42.611 DEBUG:teuthology.orchestra.run.trial197:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p foo rm existing_9 2026-02-26T03:01:42.626 INFO:tasks.ceph.osd.1.trial197.stderr:2026-02-26T03:01:42.624+0000 7fa6b0914640 -1 bdev(0x55dc8e5fa000 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2026-02-26T03:01:42.626 INFO:tasks.ceph.osd.1.trial197.stderr:2026-02-26T03:01:42.624+0000 7fa6b0914640 -1 bdev(0x55dc8e5fa000 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2026-02-26T03:01:42.643 INFO:tasks.ceph.osd.1.trial197.stderr:2026-02-26T03:01:42.641+0000 7fa6b0914640 -1 bdev(0x55dc8e5fa000 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2026-02-26T03:01:42.643 INFO:tasks.ceph.osd.1.trial197.stderr:2026-02-26T03:01:42.641+0000 7fa6b0914640 -1 bdev(0x55dc8e5fa000 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2026-02-26T03:01:42.648 INFO:tasks.ceph.osd.1.trial197.stderr:2026-02-26T03:01:42.647+0000 7fa6b491c640 -1 bdev(0x55dc8e404000 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2026-02-26T03:01:42.659 INFO:tasks.ceph.osd.1.trial197.stderr:2026-02-26T03:01:42.658+0000 7fa6b0914640 -1 bdev(0x55dc8e5fa000 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2026-02-26T03:01:42.660 INFO:tasks.ceph.osd.1.trial197.stderr:2026-02-26T03:01:42.658+0000 7fa6b0914640 -1 bdev(0x55dc8e5fa000 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2026-02-26T03:01:52.627 DEBUG:teuthology.orchestra.run.trial197:> killall -9 rados 2026-02-26T03:01:52.657 DEBUG:teuthology.orchestra.run:got remote process result: 1 2026-02-26T03:01:52.657 INFO:teuthology.orchestra.run.trial197.stderr:rados: no process found 2026-02-26T03:01:52.658 INFO:tasks.divergent_priors2:killing all the osds 2026-02-26T03:01:52.658 DEBUG:tasks.ceph.osd.0:waiting for process to exit 2026-02-26T03:01:52.658 INFO:teuthology.orchestra.run:waiting for 300 2026-02-26T03:01:52.734 INFO:tasks.ceph.osd.0:Stopped 2026-02-26T03:01:52.735 DEBUG:tasks.ceph.osd.1:waiting for process to exit 2026-02-26T03:01:52.735 INFO:teuthology.orchestra.run:waiting for 300 2026-02-26T03:01:52.807 INFO:tasks.ceph.osd.1:Stopped 2026-02-26T03:01:52.808 DEBUG:tasks.ceph.osd.2:waiting for process to exit 2026-02-26T03:01:52.808 INFO:teuthology.orchestra.run:waiting for 300 2026-02-26T03:01:52.882 INFO:tasks.ceph.osd.2:Stopped 2026-02-26T03:01:52.883 DEBUG:teuthology.orchestra.run.trial197:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd down 0 2026-02-26T03:01:53.080 INFO:tasks.ceph.mon.a.trial197.stderr:2026-02-26T03:01:53.078+0000 7fba8b7fa640 -1 mon.a@0(leader).osd e20 definitely_dead 0 2026-02-26T03:01:53.177 INFO:teuthology.orchestra.run.trial197.stderr:marked down osd.0. 2026-02-26T03:01:53.186 DEBUG:teuthology.orchestra.run.trial197:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd down 1 2026-02-26T03:01:53.351 INFO:tasks.ceph.mon.a.trial197.stderr:2026-02-26T03:01:53.349+0000 7fba8b7fa640 -1 mon.a@0(leader).osd e21 definitely_dead 0 2026-02-26T03:01:54.180 INFO:teuthology.orchestra.run.trial197.stderr:marked down osd.1. 2026-02-26T03:01:54.192 DEBUG:teuthology.orchestra.run.trial197:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd down 2 2026-02-26T03:01:54.351 INFO:tasks.ceph.mon.a.trial197.stderr:2026-02-26T03:01:54.350+0000 7fba8b7fa640 -1 mon.a@0(leader).osd e22 definitely_dead 0 2026-02-26T03:01:55.182 INFO:teuthology.orchestra.run.trial197.stderr:marked down osd.2. 2026-02-26T03:01:55.193 DEBUG:teuthology.orchestra.run.trial197:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd out 0 2026-02-26T03:01:55.355 INFO:tasks.ceph.mon.a.trial197.stderr:2026-02-26T03:01:55.354+0000 7fba8b7fa640 -1 mon.a@0(leader).osd e23 definitely_dead 0 2026-02-26T03:01:56.185 INFO:teuthology.orchestra.run.trial197.stderr:marked out osd.0. 2026-02-26T03:01:56.198 DEBUG:teuthology.orchestra.run.trial197:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd out 1 2026-02-26T03:01:56.359 INFO:tasks.ceph.mon.a.trial197.stderr:2026-02-26T03:01:56.357+0000 7fba8b7fa640 -1 mon.a@0(leader).osd e24 definitely_dead 0 2026-02-26T03:01:57.188 INFO:teuthology.orchestra.run.trial197.stderr:marked out osd.1. 2026-02-26T03:01:57.199 INFO:tasks.divergent_priors2:bringing up non_divergent [0, 1] 2026-02-26T03:01:57.199 INFO:tasks.ceph.osd.0:Restarting daemon 2026-02-26T03:01:57.200 DEBUG:teuthology.orchestra.run.trial197:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 0 2026-02-26T03:01:57.201 INFO:tasks.ceph.osd.0:Started 2026-02-26T03:01:57.202 DEBUG:teuthology.orchestra.run.trial197:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2026-02-26T03:01:57.268 INFO:teuthology.orchestra.run.trial197.stderr:no valid command found; 10 closest matches: 2026-02-26T03:01:57.268 INFO:teuthology.orchestra.run.trial197.stderr:0 2026-02-26T03:01:57.268 INFO:teuthology.orchestra.run.trial197.stderr:1 2026-02-26T03:01:57.268 INFO:teuthology.orchestra.run.trial197.stderr:2 2026-02-26T03:01:57.268 INFO:teuthology.orchestra.run.trial197.stderr:abort 2026-02-26T03:01:57.269 INFO:teuthology.orchestra.run.trial197.stderr:assert 2026-02-26T03:01:57.269 INFO:teuthology.orchestra.run.trial197.stderr:config diff 2026-02-26T03:01:57.269 INFO:teuthology.orchestra.run.trial197.stderr:config diff get 2026-02-26T03:01:57.269 INFO:teuthology.orchestra.run.trial197.stderr:config get 2026-02-26T03:01:57.269 INFO:teuthology.orchestra.run.trial197.stderr:config help [] 2026-02-26T03:01:57.269 INFO:teuthology.orchestra.run.trial197.stderr:config set ... 2026-02-26T03:01:57.269 INFO:teuthology.orchestra.run.trial197.stderr:admin_socket: invalid command 2026-02-26T03:01:57.270 DEBUG:teuthology.orchestra.run:got remote process result: 22 2026-02-26T03:01:57.271 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-0, ['dump_ops_in_flight'] 2026-02-26T03:01:58.340 INFO:tasks.ceph.osd.0.trial197.stderr:2026-02-26T03:01:58.338+0000 7f95a27b4780 -1 Falling back to public interface 2026-02-26T03:02:02.057 INFO:tasks.ceph.osd.0.trial197.stderr:2026-02-26T03:02:02.055+0000 7f95a27b4780 -1 osd.0 20 log_to_monitors true 2026-02-26T03:02:02.272 DEBUG:teuthology.orchestra.run.trial197:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2026-02-26T03:02:02.344 INFO:teuthology.orchestra.run.trial197.stderr:no valid command found; 10 closest matches: 2026-02-26T03:02:02.345 INFO:teuthology.orchestra.run.trial197.stderr:0 2026-02-26T03:02:02.345 INFO:teuthology.orchestra.run.trial197.stderr:1 2026-02-26T03:02:02.345 INFO:teuthology.orchestra.run.trial197.stderr:2 2026-02-26T03:02:02.345 INFO:teuthology.orchestra.run.trial197.stderr:abort 2026-02-26T03:02:02.345 INFO:teuthology.orchestra.run.trial197.stderr:assert 2026-02-26T03:02:02.345 INFO:teuthology.orchestra.run.trial197.stderr:bluefs debug_inject_read_zeros 2026-02-26T03:02:02.345 INFO:teuthology.orchestra.run.trial197.stderr:bluefs files list 2026-02-26T03:02:02.345 INFO:teuthology.orchestra.run.trial197.stderr:bluefs stats 2026-02-26T03:02:02.345 INFO:teuthology.orchestra.run.trial197.stderr:bluestore allocator dump block 2026-02-26T03:02:02.345 INFO:teuthology.orchestra.run.trial197.stderr:bluestore allocator fragmentation block 2026-02-26T03:02:02.346 INFO:teuthology.orchestra.run.trial197.stderr:admin_socket: invalid command 2026-02-26T03:02:02.347 DEBUG:teuthology.orchestra.run:got remote process result: 22 2026-02-26T03:02:02.347 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-0, ['dump_ops_in_flight'] 2026-02-26T03:02:02.476 INFO:tasks.ceph.osd.0.trial197.stderr:2026-02-26T03:02:02.475+0000 7f9596563640 -1 osd.0 20 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2026-02-26T03:02:06.386 INFO:tasks.ceph.mgr.x.trial197.stderr:2026-02-26T03:02:06.385+0000 7f5f31995640 -1 calc_pg_upmaps abort due to max <= 0 2026-02-26T03:02:06.386 INFO:tasks.ceph.mgr.x.trial197.stderr:2026-02-26T03:02:06.385+0000 7f5f31995640 -1 calc_pg_upmaps abort due to max <= 0 2026-02-26T03:02:07.348 DEBUG:teuthology.orchestra.run.trial197:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2026-02-26T03:02:07.420 INFO:teuthology.orchestra.run.trial197.stdout:{ 2026-02-26T03:02:07.420 INFO:teuthology.orchestra.run.trial197.stdout: "ops": [], 2026-02-26T03:02:07.420 INFO:teuthology.orchestra.run.trial197.stdout: "num_ops": 0 2026-02-26T03:02:07.421 INFO:teuthology.orchestra.run.trial197.stdout:} 2026-02-26T03:02:07.426 INFO:tasks.ceph.osd.1:Restarting daemon 2026-02-26T03:02:07.426 DEBUG:teuthology.orchestra.run.trial197:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 1 2026-02-26T03:02:07.428 INFO:tasks.ceph.osd.1:Started 2026-02-26T03:02:07.428 DEBUG:teuthology.orchestra.run.trial197:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2026-02-26T03:02:07.496 INFO:teuthology.orchestra.run.trial197.stderr:no valid command found; 10 closest matches: 2026-02-26T03:02:07.496 INFO:teuthology.orchestra.run.trial197.stderr:0 2026-02-26T03:02:07.496 INFO:teuthology.orchestra.run.trial197.stderr:1 2026-02-26T03:02:07.496 INFO:teuthology.orchestra.run.trial197.stderr:2 2026-02-26T03:02:07.497 INFO:teuthology.orchestra.run.trial197.stderr:abort 2026-02-26T03:02:07.497 INFO:teuthology.orchestra.run.trial197.stderr:assert 2026-02-26T03:02:07.497 INFO:teuthology.orchestra.run.trial197.stderr:config diff 2026-02-26T03:02:07.497 INFO:teuthology.orchestra.run.trial197.stderr:config diff get 2026-02-26T03:02:07.497 INFO:teuthology.orchestra.run.trial197.stderr:config get 2026-02-26T03:02:07.497 INFO:teuthology.orchestra.run.trial197.stderr:config help [] 2026-02-26T03:02:07.497 INFO:teuthology.orchestra.run.trial197.stderr:config set ... 2026-02-26T03:02:07.497 INFO:teuthology.orchestra.run.trial197.stderr:admin_socket: invalid command 2026-02-26T03:02:07.498 DEBUG:teuthology.orchestra.run:got remote process result: 22 2026-02-26T03:02:07.498 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-1, ['dump_ops_in_flight'] 2026-02-26T03:02:08.567 INFO:tasks.ceph.osd.1.trial197.stderr:2026-02-26T03:02:08.565+0000 7fe00c72c780 -1 Falling back to public interface 2026-02-26T03:02:12.100 INFO:tasks.ceph.osd.1.trial197.stderr:2026-02-26T03:02:12.098+0000 7fe00c72c780 -1 osd.1 20 log_to_monitors true 2026-02-26T03:02:12.214 INFO:tasks.ceph.osd.1.trial197.stderr:2026-02-26T03:02:12.212+0000 7fe0004db640 -1 osd.1 20 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2026-02-26T03:02:12.499 DEBUG:teuthology.orchestra.run.trial197:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2026-02-26T03:02:12.571 INFO:teuthology.orchestra.run.trial197.stdout:{ 2026-02-26T03:02:12.571 INFO:teuthology.orchestra.run.trial197.stdout: "ops": [], 2026-02-26T03:02:12.571 INFO:teuthology.orchestra.run.trial197.stdout: "num_ops": 0 2026-02-26T03:02:12.571 INFO:teuthology.orchestra.run.trial197.stdout:} 2026-02-26T03:02:12.576 DEBUG:teuthology.orchestra.run.trial197:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd in 0 2026-02-26T03:02:12.737 INFO:tasks.ceph.mon.a.trial197.stderr:2026-02-26T03:02:12.735+0000 7fba8b7fa640 -1 mon.a@0(leader).osd e28 definitely_dead 0 2026-02-26T03:02:13.205 INFO:teuthology.orchestra.run.trial197.stderr:marked in osd.0. 2026-02-26T03:02:13.216 DEBUG:teuthology.orchestra.run.trial197:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd in 1 2026-02-26T03:02:13.379 INFO:tasks.ceph.mon.a.trial197.stderr:2026-02-26T03:02:13.377+0000 7fba8b7fa640 -1 mon.a@0(leader).osd e29 definitely_dead 0 2026-02-26T03:02:14.147 INFO:teuthology.orchestra.run.trial197.stderr:marked in osd.1. 2026-02-26T03:02:14.159 INFO:tasks.divergent_priors2:writing non-divergent object existing_10 2026-02-26T03:02:14.159 INFO:tasks.util.rados:rados -p foo put existing_10 /etc/resolv.conf 2026-02-26T03:02:14.159 DEBUG:teuthology.orchestra.run.trial197:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p foo put existing_10 /etc/resolv.conf 2026-02-26T03:02:15.152 INFO:tasks.ceph.ceph_manager.ceph:waiting for recovery to complete 2026-02-26T03:02:15.152 DEBUG:teuthology.orchestra.run.trial197:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-26T03:02:15.308 INFO:teuthology.orchestra.run.trial197.stdout: 2026-02-26T03:02:15.308 INFO:teuthology.orchestra.run.trial197.stderr:dumped all 2026-02-26T03:02:15.316 INFO:teuthology.orchestra.run.trial197.stdout:{"pg_ready":true,"pg_map":{"version":60,"stamp":"2026-02-26T03:02:15.146808+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":59159,"num_objects":97,"num_object_clones":0,"num_object_copies":194,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":97,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":112,"num_write_kb":107,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":18,"ondisk_log_size":18,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27528,"kb_used_data":548,"kb_used_omap":1,"kb_used_meta":26942,"kb_avail":94344312,"statfs":{"total":96636764160,"available":96608575488,"internally_reserved":0,"allocated":561152,"data_stored":110221,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27589062},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.761860"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":31,"reported_epoch":20,"state":"stale+active+clean","last_fresh":"2026-02-26T03:01:34.165011+0000","last_change":"2026-02-26T03:01:21.671776+0000","last_active":"2026-02-26T03:01:34.165011+0000","last_peered":"2026-02-26T03:01:34.165011+0000","last_clean":"2026-02-26T03:01:34.165011+0000","last_became_active":"2026-02-26T03:01:19.668398+0000","last_became_peered":"2026-02-26T03:01:19.668398+0000","last_unstale":"2026-02-26T03:01:54.180116+0000","last_undegraded":"2026-02-26T03:01:34.165011+0000","last_fullsized":"2026-02-26T03:01:34.165011+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-26T03:01:18.659251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-26T03:01:18.659251+0000","last_clean_scrub_stamp":"2026-02-26T03:01:18.659251+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-27T04:08:38.487713+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00060011299999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":31,"reported_epoch":20,"state":"stale+active+clean","last_fresh":"2026-02-26T03:01:34.165008+0000","last_change":"2026-02-26T03:01:21.671905+0000","last_active":"2026-02-26T03:01:34.165008+0000","last_peered":"2026-02-26T03:01:34.165008+0000","last_clean":"2026-02-26T03:01:34.165008+0000","last_became_active":"2026-02-26T03:01:19.669316+0000","last_became_peered":"2026-02-26T03:01:19.669316+0000","last_unstale":"2026-02-26T03:01:54.180117+0000","last_undegraded":"2026-02-26T03:01:34.165008+0000","last_fullsized":"2026-02-26T03:01:34.165008+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-26T03:01:18.659251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-26T03:01:18.659251+0000","last_clean_scrub_stamp":"2026-02-26T03:01:18.659251+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-27T13:43:20.762574+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00068042299999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":31,"reported_epoch":20,"state":"stale+active+clean","last_fresh":"2026-02-26T03:01:34.183323+0000","last_change":"2026-02-26T03:01:21.669362+0000","last_active":"2026-02-26T03:01:34.183323+0000","last_peered":"2026-02-26T03:01:34.183323+0000","last_clean":"2026-02-26T03:01:34.183323+0000","last_became_active":"2026-02-26T03:01:19.669313+0000","last_became_peered":"2026-02-26T03:01:19.669313+0000","last_unstale":"2026-02-26T03:01:55.182890+0000","last_undegraded":"2026-02-26T03:01:34.183323+0000","last_fullsized":"2026-02-26T03:01:34.183323+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-26T03:01:18.659251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-26T03:01:18.659251+0000","last_clean_scrub_stamp":"2026-02-26T03:01:18.659251+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-27T10:49:28.489885+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00034704899999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":31,"reported_epoch":20,"state":"stale+active+clean","last_fresh":"2026-02-26T03:01:34.164997+0000","last_change":"2026-02-26T03:01:21.671647+0000","last_active":"2026-02-26T03:01:34.164997+0000","last_peered":"2026-02-26T03:01:34.164997+0000","last_clean":"2026-02-26T03:01:34.164997+0000","last_became_active":"2026-02-26T03:01:19.669171+0000","last_became_peered":"2026-02-26T03:01:19.669171+0000","last_unstale":"2026-02-26T03:01:54.180119+0000","last_undegraded":"2026-02-26T03:01:34.164997+0000","last_fullsized":"2026-02-26T03:01:34.164997+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-26T03:01:18.659251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-26T03:01:18.659251+0000","last_clean_scrub_stamp":"2026-02-26T03:01:18.659251+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-27T04:13:58.662319+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000482733,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"20'110","reported_seq":122,"reported_epoch":20,"state":"stale+active+clean","last_fresh":"2026-02-26T03:01:42.643204+0000","last_change":"2026-02-26T03:01:33.675008+0000","last_active":"2026-02-26T03:01:42.643204+0000","last_peered":"2026-02-26T03:01:42.643204+0000","last_clean":"2026-02-26T03:01:42.643204+0000","last_became_active":"2026-02-26T03:01:33.674358+0000","last_became_peered":"2026-02-26T03:01:33.674358+0000","last_unstale":"2026-02-26T03:01:55.182891+0000","last_undegraded":"2026-02-26T03:01:42.643204+0000","last_fullsized":"2026-02-26T03:01:42.643204+0000","mapping_epoch":18,"log_start":"20'95","ondisk_log_start":"20'95","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-26T03:01:32.157530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-26T03:01:32.157530+0000","last_clean_scrub_stamp":"2026-02-26T03:01:32.157530+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":95,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-27T13:36:02.852922+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":59140,"num_objects":95,"num_object_clones":0,"num_object_copies":190,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":95,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":110,"num_write_kb":105,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"11'1","reported_seq":32,"reported_epoch":20,"state":"stale+active+clean","last_fresh":"2026-02-26T03:01:34.164988+0000","last_change":"2026-02-26T03:01:21.671498+0000","last_active":"2026-02-26T03:01:34.164988+0000","last_peered":"2026-02-26T03:01:34.164988+0000","last_clean":"2026-02-26T03:01:34.164988+0000","last_became_active":"2026-02-26T03:01:19.668677+0000","last_became_peered":"2026-02-26T03:01:19.668677+0000","last_unstale":"2026-02-26T03:01:54.180123+0000","last_undegraded":"2026-02-26T03:01:34.164988+0000","last_fullsized":"2026-02-26T03:01:34.164988+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-26T03:01:18.659251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-26T03:01:18.659251+0000","last_clean_scrub_stamp":"2026-02-26T03:01:18.659251+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-27T09:30:44.521018+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00034461499999999999,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"13'2","reported_seq":34,"reported_epoch":20,"state":"stale+active+clean","last_fresh":"2026-02-26T03:01:34.716107+0000","last_change":"2026-02-26T03:01:21.670052+0000","last_active":"2026-02-26T03:01:34.716107+0000","last_peered":"2026-02-26T03:01:34.716107+0000","last_clean":"2026-02-26T03:01:34.716107+0000","last_became_active":"2026-02-26T03:01:19.668415+0000","last_became_peered":"2026-02-26T03:01:19.668415+0000","last_unstale":"2026-02-26T03:01:53.177025+0000","last_undegraded":"2026-02-26T03:01:34.716107+0000","last_fullsized":"2026-02-26T03:01:34.716107+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-26T03:01:18.659251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-26T03:01:18.659251+0000","last_clean_scrub_stamp":"2026-02-26T03:01:18.659251+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-27T12:03:13.020042+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00054467999999999997,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":31,"reported_epoch":20,"state":"stale+active+clean","last_fresh":"2026-02-26T03:01:34.183322+0000","last_change":"2026-02-26T03:01:21.669370+0000","last_active":"2026-02-26T03:01:34.183322+0000","last_peered":"2026-02-26T03:01:34.183322+0000","last_clean":"2026-02-26T03:01:34.183322+0000","last_became_active":"2026-02-26T03:01:19.669112+0000","last_became_peered":"2026-02-26T03:01:19.669112+0000","last_unstale":"2026-02-26T03:01:55.182896+0000","last_undegraded":"2026-02-26T03:01:34.183322+0000","last_fullsized":"2026-02-26T03:01:34.183322+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-26T03:01:18.659251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-26T03:01:18.659251+0000","last_clean_scrub_stamp":"2026-02-26T03:01:18.659251+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-27T14:09:53.100227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00037735600000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":31,"reported_epoch":20,"state":"stale+active+clean","last_fresh":"2026-02-26T03:01:34.164969+0000","last_change":"2026-02-26T03:01:21.671520+0000","last_active":"2026-02-26T03:01:34.164969+0000","last_peered":"2026-02-26T03:01:34.164969+0000","last_clean":"2026-02-26T03:01:34.164969+0000","last_became_active":"2026-02-26T03:01:19.669362+0000","last_became_peered":"2026-02-26T03:01:19.669362+0000","last_unstale":"2026-02-26T03:01:54.180125+0000","last_undegraded":"2026-02-26T03:01:34.164969+0000","last_fullsized":"2026-02-26T03:01:34.164969+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-26T03:01:18.659251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-26T03:01:18.659251+0000","last_clean_scrub_stamp":"2026-02-26T03:01:18.659251+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-27T03:50:25.293079+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00037062400000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":59140,"num_objects":95,"num_object_clones":0,"num_object_copies":190,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":95,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":110,"num_write_kb":105,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":798720,"data_stored":124240,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":15,"ondisk_log_size":15,"up":2,"acting":2,"num_store_stats":2},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":1,"up_from":0,"seq":2,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":8,"seq":34359738380,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27528,"kb_used_data":548,"kb_used_omap":1,"kb_used_meta":26942,"kb_avail":94344312,"statfs":{"total":96636764160,"available":96608575488,"internally_reserved":0,"allocated":561152,"data_stored":110221,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27589062},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":115964116996,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":409600,"data_stored":65100,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":389120,"data_stored":59140,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-26T03:02:15.316 DEBUG:teuthology.orchestra.run.trial197:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-26T03:02:15.469 INFO:teuthology.orchestra.run.trial197.stdout: 2026-02-26T03:02:15.469 INFO:teuthology.orchestra.run.trial197.stderr:dumped all 2026-02-26T03:02:15.476 INFO:teuthology.orchestra.run.trial197.stdout:{"pg_ready":true,"pg_map":{"version":60,"stamp":"2026-02-26T03:02:15.146808+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":59159,"num_objects":97,"num_object_clones":0,"num_object_copies":194,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":97,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":112,"num_write_kb":107,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":18,"ondisk_log_size":18,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27528,"kb_used_data":548,"kb_used_omap":1,"kb_used_meta":26942,"kb_avail":94344312,"statfs":{"total":96636764160,"available":96608575488,"internally_reserved":0,"allocated":561152,"data_stored":110221,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27589062},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.761860"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":31,"reported_epoch":20,"state":"stale+active+clean","last_fresh":"2026-02-26T03:01:34.165011+0000","last_change":"2026-02-26T03:01:21.671776+0000","last_active":"2026-02-26T03:01:34.165011+0000","last_peered":"2026-02-26T03:01:34.165011+0000","last_clean":"2026-02-26T03:01:34.165011+0000","last_became_active":"2026-02-26T03:01:19.668398+0000","last_became_peered":"2026-02-26T03:01:19.668398+0000","last_unstale":"2026-02-26T03:01:54.180116+0000","last_undegraded":"2026-02-26T03:01:34.165011+0000","last_fullsized":"2026-02-26T03:01:34.165011+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-26T03:01:18.659251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-26T03:01:18.659251+0000","last_clean_scrub_stamp":"2026-02-26T03:01:18.659251+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-27T04:08:38.487713+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00060011299999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":31,"reported_epoch":20,"state":"stale+active+clean","last_fresh":"2026-02-26T03:01:34.165008+0000","last_change":"2026-02-26T03:01:21.671905+0000","last_active":"2026-02-26T03:01:34.165008+0000","last_peered":"2026-02-26T03:01:34.165008+0000","last_clean":"2026-02-26T03:01:34.165008+0000","last_became_active":"2026-02-26T03:01:19.669316+0000","last_became_peered":"2026-02-26T03:01:19.669316+0000","last_unstale":"2026-02-26T03:01:54.180117+0000","last_undegraded":"2026-02-26T03:01:34.165008+0000","last_fullsized":"2026-02-26T03:01:34.165008+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-26T03:01:18.659251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-26T03:01:18.659251+0000","last_clean_scrub_stamp":"2026-02-26T03:01:18.659251+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-27T13:43:20.762574+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00068042299999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":31,"reported_epoch":20,"state":"stale+active+clean","last_fresh":"2026-02-26T03:01:34.183323+0000","last_change":"2026-02-26T03:01:21.669362+0000","last_active":"2026-02-26T03:01:34.183323+0000","last_peered":"2026-02-26T03:01:34.183323+0000","last_clean":"2026-02-26T03:01:34.183323+0000","last_became_active":"2026-02-26T03:01:19.669313+0000","last_became_peered":"2026-02-26T03:01:19.669313+0000","last_unstale":"2026-02-26T03:01:55.182890+0000","last_undegraded":"2026-02-26T03:01:34.183323+0000","last_fullsized":"2026-02-26T03:01:34.183323+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-26T03:01:18.659251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-26T03:01:18.659251+0000","last_clean_scrub_stamp":"2026-02-26T03:01:18.659251+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-27T10:49:28.489885+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00034704899999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":31,"reported_epoch":20,"state":"stale+active+clean","last_fresh":"2026-02-26T03:01:34.164997+0000","last_change":"2026-02-26T03:01:21.671647+0000","last_active":"2026-02-26T03:01:34.164997+0000","last_peered":"2026-02-26T03:01:34.164997+0000","last_clean":"2026-02-26T03:01:34.164997+0000","last_became_active":"2026-02-26T03:01:19.669171+0000","last_became_peered":"2026-02-26T03:01:19.669171+0000","last_unstale":"2026-02-26T03:01:54.180119+0000","last_undegraded":"2026-02-26T03:01:34.164997+0000","last_fullsized":"2026-02-26T03:01:34.164997+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-26T03:01:18.659251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-26T03:01:18.659251+0000","last_clean_scrub_stamp":"2026-02-26T03:01:18.659251+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-27T04:13:58.662319+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000482733,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"20'110","reported_seq":122,"reported_epoch":20,"state":"stale+active+clean","last_fresh":"2026-02-26T03:01:42.643204+0000","last_change":"2026-02-26T03:01:33.675008+0000","last_active":"2026-02-26T03:01:42.643204+0000","last_peered":"2026-02-26T03:01:42.643204+0000","last_clean":"2026-02-26T03:01:42.643204+0000","last_became_active":"2026-02-26T03:01:33.674358+0000","last_became_peered":"2026-02-26T03:01:33.674358+0000","last_unstale":"2026-02-26T03:01:55.182891+0000","last_undegraded":"2026-02-26T03:01:42.643204+0000","last_fullsized":"2026-02-26T03:01:42.643204+0000","mapping_epoch":18,"log_start":"20'95","ondisk_log_start":"20'95","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-26T03:01:32.157530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-26T03:01:32.157530+0000","last_clean_scrub_stamp":"2026-02-26T03:01:32.157530+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":95,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-27T13:36:02.852922+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":59140,"num_objects":95,"num_object_clones":0,"num_object_copies":190,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":95,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":110,"num_write_kb":105,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"11'1","reported_seq":32,"reported_epoch":20,"state":"stale+active+clean","last_fresh":"2026-02-26T03:01:34.164988+0000","last_change":"2026-02-26T03:01:21.671498+0000","last_active":"2026-02-26T03:01:34.164988+0000","last_peered":"2026-02-26T03:01:34.164988+0000","last_clean":"2026-02-26T03:01:34.164988+0000","last_became_active":"2026-02-26T03:01:19.668677+0000","last_became_peered":"2026-02-26T03:01:19.668677+0000","last_unstale":"2026-02-26T03:01:54.180123+0000","last_undegraded":"2026-02-26T03:01:34.164988+0000","last_fullsized":"2026-02-26T03:01:34.164988+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-26T03:01:18.659251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-26T03:01:18.659251+0000","last_clean_scrub_stamp":"2026-02-26T03:01:18.659251+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-27T09:30:44.521018+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00034461499999999999,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"13'2","reported_seq":34,"reported_epoch":20,"state":"stale+active+clean","last_fresh":"2026-02-26T03:01:34.716107+0000","last_change":"2026-02-26T03:01:21.670052+0000","last_active":"2026-02-26T03:01:34.716107+0000","last_peered":"2026-02-26T03:01:34.716107+0000","last_clean":"2026-02-26T03:01:34.716107+0000","last_became_active":"2026-02-26T03:01:19.668415+0000","last_became_peered":"2026-02-26T03:01:19.668415+0000","last_unstale":"2026-02-26T03:01:53.177025+0000","last_undegraded":"2026-02-26T03:01:34.716107+0000","last_fullsized":"2026-02-26T03:01:34.716107+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-26T03:01:18.659251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-26T03:01:18.659251+0000","last_clean_scrub_stamp":"2026-02-26T03:01:18.659251+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-27T12:03:13.020042+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00054467999999999997,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":31,"reported_epoch":20,"state":"stale+active+clean","last_fresh":"2026-02-26T03:01:34.183322+0000","last_change":"2026-02-26T03:01:21.669370+0000","last_active":"2026-02-26T03:01:34.183322+0000","last_peered":"2026-02-26T03:01:34.183322+0000","last_clean":"2026-02-26T03:01:34.183322+0000","last_became_active":"2026-02-26T03:01:19.669112+0000","last_became_peered":"2026-02-26T03:01:19.669112+0000","last_unstale":"2026-02-26T03:01:55.182896+0000","last_undegraded":"2026-02-26T03:01:34.183322+0000","last_fullsized":"2026-02-26T03:01:34.183322+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-26T03:01:18.659251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-26T03:01:18.659251+0000","last_clean_scrub_stamp":"2026-02-26T03:01:18.659251+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-27T14:09:53.100227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00037735600000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":31,"reported_epoch":20,"state":"stale+active+clean","last_fresh":"2026-02-26T03:01:34.164969+0000","last_change":"2026-02-26T03:01:21.671520+0000","last_active":"2026-02-26T03:01:34.164969+0000","last_peered":"2026-02-26T03:01:34.164969+0000","last_clean":"2026-02-26T03:01:34.164969+0000","last_became_active":"2026-02-26T03:01:19.669362+0000","last_became_peered":"2026-02-26T03:01:19.669362+0000","last_unstale":"2026-02-26T03:01:54.180125+0000","last_undegraded":"2026-02-26T03:01:34.164969+0000","last_fullsized":"2026-02-26T03:01:34.164969+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-26T03:01:18.659251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-26T03:01:18.659251+0000","last_clean_scrub_stamp":"2026-02-26T03:01:18.659251+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-27T03:50:25.293079+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00037062400000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":59140,"num_objects":95,"num_object_clones":0,"num_object_copies":190,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":95,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":110,"num_write_kb":105,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":798720,"data_stored":124240,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":15,"ondisk_log_size":15,"up":2,"acting":2,"num_store_stats":2},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":1,"up_from":0,"seq":2,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":8,"seq":34359738380,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27528,"kb_used_data":548,"kb_used_omap":1,"kb_used_meta":26942,"kb_avail":94344312,"statfs":{"total":96636764160,"available":96608575488,"internally_reserved":0,"allocated":561152,"data_stored":110221,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27589062},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":115964116996,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":409600,"data_stored":65100,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":389120,"data_stored":59140,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-26T03:02:15.478 DEBUG:teuthology.orchestra.run.trial197:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-26T03:02:15.635 INFO:teuthology.orchestra.run.trial197.stdout: 2026-02-26T03:02:15.636 INFO:teuthology.orchestra.run.trial197.stderr:dumped all 2026-02-26T03:02:15.642 INFO:teuthology.orchestra.run.trial197.stdout:{"pg_ready":true,"pg_map":{"version":60,"stamp":"2026-02-26T03:02:15.146808+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":59159,"num_objects":97,"num_object_clones":0,"num_object_copies":194,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":97,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":112,"num_write_kb":107,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":18,"ondisk_log_size":18,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27528,"kb_used_data":548,"kb_used_omap":1,"kb_used_meta":26942,"kb_avail":94344312,"statfs":{"total":96636764160,"available":96608575488,"internally_reserved":0,"allocated":561152,"data_stored":110221,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27589062},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.761860"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":31,"reported_epoch":20,"state":"stale+active+clean","last_fresh":"2026-02-26T03:01:34.165011+0000","last_change":"2026-02-26T03:01:21.671776+0000","last_active":"2026-02-26T03:01:34.165011+0000","last_peered":"2026-02-26T03:01:34.165011+0000","last_clean":"2026-02-26T03:01:34.165011+0000","last_became_active":"2026-02-26T03:01:19.668398+0000","last_became_peered":"2026-02-26T03:01:19.668398+0000","last_unstale":"2026-02-26T03:01:54.180116+0000","last_undegraded":"2026-02-26T03:01:34.165011+0000","last_fullsized":"2026-02-26T03:01:34.165011+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-26T03:01:18.659251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-26T03:01:18.659251+0000","last_clean_scrub_stamp":"2026-02-26T03:01:18.659251+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-27T04:08:38.487713+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00060011299999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":31,"reported_epoch":20,"state":"stale+active+clean","last_fresh":"2026-02-26T03:01:34.165008+0000","last_change":"2026-02-26T03:01:21.671905+0000","last_active":"2026-02-26T03:01:34.165008+0000","last_peered":"2026-02-26T03:01:34.165008+0000","last_clean":"2026-02-26T03:01:34.165008+0000","last_became_active":"2026-02-26T03:01:19.669316+0000","last_became_peered":"2026-02-26T03:01:19.669316+0000","last_unstale":"2026-02-26T03:01:54.180117+0000","last_undegraded":"2026-02-26T03:01:34.165008+0000","last_fullsized":"2026-02-26T03:01:34.165008+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-26T03:01:18.659251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-26T03:01:18.659251+0000","last_clean_scrub_stamp":"2026-02-26T03:01:18.659251+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-27T13:43:20.762574+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00068042299999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":31,"reported_epoch":20,"state":"stale+active+clean","last_fresh":"2026-02-26T03:01:34.183323+0000","last_change":"2026-02-26T03:01:21.669362+0000","last_active":"2026-02-26T03:01:34.183323+0000","last_peered":"2026-02-26T03:01:34.183323+0000","last_clean":"2026-02-26T03:01:34.183323+0000","last_became_active":"2026-02-26T03:01:19.669313+0000","last_became_peered":"2026-02-26T03:01:19.669313+0000","last_unstale":"2026-02-26T03:01:55.182890+0000","last_undegraded":"2026-02-26T03:01:34.183323+0000","last_fullsized":"2026-02-26T03:01:34.183323+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-26T03:01:18.659251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-26T03:01:18.659251+0000","last_clean_scrub_stamp":"2026-02-26T03:01:18.659251+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-27T10:49:28.489885+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00034704899999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":31,"reported_epoch":20,"state":"stale+active+clean","last_fresh":"2026-02-26T03:01:34.164997+0000","last_change":"2026-02-26T03:01:21.671647+0000","last_active":"2026-02-26T03:01:34.164997+0000","last_peered":"2026-02-26T03:01:34.164997+0000","last_clean":"2026-02-26T03:01:34.164997+0000","last_became_active":"2026-02-26T03:01:19.669171+0000","last_became_peered":"2026-02-26T03:01:19.669171+0000","last_unstale":"2026-02-26T03:01:54.180119+0000","last_undegraded":"2026-02-26T03:01:34.164997+0000","last_fullsized":"2026-02-26T03:01:34.164997+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-26T03:01:18.659251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-26T03:01:18.659251+0000","last_clean_scrub_stamp":"2026-02-26T03:01:18.659251+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-27T04:13:58.662319+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000482733,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"20'110","reported_seq":122,"reported_epoch":20,"state":"stale+active+clean","last_fresh":"2026-02-26T03:01:42.643204+0000","last_change":"2026-02-26T03:01:33.675008+0000","last_active":"2026-02-26T03:01:42.643204+0000","last_peered":"2026-02-26T03:01:42.643204+0000","last_clean":"2026-02-26T03:01:42.643204+0000","last_became_active":"2026-02-26T03:01:33.674358+0000","last_became_peered":"2026-02-26T03:01:33.674358+0000","last_unstale":"2026-02-26T03:01:55.182891+0000","last_undegraded":"2026-02-26T03:01:42.643204+0000","last_fullsized":"2026-02-26T03:01:42.643204+0000","mapping_epoch":18,"log_start":"20'95","ondisk_log_start":"20'95","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-26T03:01:32.157530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-26T03:01:32.157530+0000","last_clean_scrub_stamp":"2026-02-26T03:01:32.157530+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":95,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-27T13:36:02.852922+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":59140,"num_objects":95,"num_object_clones":0,"num_object_copies":190,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":95,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":110,"num_write_kb":105,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"11'1","reported_seq":32,"reported_epoch":20,"state":"stale+active+clean","last_fresh":"2026-02-26T03:01:34.164988+0000","last_change":"2026-02-26T03:01:21.671498+0000","last_active":"2026-02-26T03:01:34.164988+0000","last_peered":"2026-02-26T03:01:34.164988+0000","last_clean":"2026-02-26T03:01:34.164988+0000","last_became_active":"2026-02-26T03:01:19.668677+0000","last_became_peered":"2026-02-26T03:01:19.668677+0000","last_unstale":"2026-02-26T03:01:54.180123+0000","last_undegraded":"2026-02-26T03:01:34.164988+0000","last_fullsized":"2026-02-26T03:01:34.164988+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-26T03:01:18.659251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-26T03:01:18.659251+0000","last_clean_scrub_stamp":"2026-02-26T03:01:18.659251+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-27T09:30:44.521018+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00034461499999999999,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"13'2","reported_seq":34,"reported_epoch":20,"state":"stale+active+clean","last_fresh":"2026-02-26T03:01:34.716107+0000","last_change":"2026-02-26T03:01:21.670052+0000","last_active":"2026-02-26T03:01:34.716107+0000","last_peered":"2026-02-26T03:01:34.716107+0000","last_clean":"2026-02-26T03:01:34.716107+0000","last_became_active":"2026-02-26T03:01:19.668415+0000","last_became_peered":"2026-02-26T03:01:19.668415+0000","last_unstale":"2026-02-26T03:01:53.177025+0000","last_undegraded":"2026-02-26T03:01:34.716107+0000","last_fullsized":"2026-02-26T03:01:34.716107+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-26T03:01:18.659251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-26T03:01:18.659251+0000","last_clean_scrub_stamp":"2026-02-26T03:01:18.659251+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-27T12:03:13.020042+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00054467999999999997,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":31,"reported_epoch":20,"state":"stale+active+clean","last_fresh":"2026-02-26T03:01:34.183322+0000","last_change":"2026-02-26T03:01:21.669370+0000","last_active":"2026-02-26T03:01:34.183322+0000","last_peered":"2026-02-26T03:01:34.183322+0000","last_clean":"2026-02-26T03:01:34.183322+0000","last_became_active":"2026-02-26T03:01:19.669112+0000","last_became_peered":"2026-02-26T03:01:19.669112+0000","last_unstale":"2026-02-26T03:01:55.182896+0000","last_undegraded":"2026-02-26T03:01:34.183322+0000","last_fullsized":"2026-02-26T03:01:34.183322+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-26T03:01:18.659251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-26T03:01:18.659251+0000","last_clean_scrub_stamp":"2026-02-26T03:01:18.659251+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-27T14:09:53.100227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00037735600000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":31,"reported_epoch":20,"state":"stale+active+clean","last_fresh":"2026-02-26T03:01:34.164969+0000","last_change":"2026-02-26T03:01:21.671520+0000","last_active":"2026-02-26T03:01:34.164969+0000","last_peered":"2026-02-26T03:01:34.164969+0000","last_clean":"2026-02-26T03:01:34.164969+0000","last_became_active":"2026-02-26T03:01:19.669362+0000","last_became_peered":"2026-02-26T03:01:19.669362+0000","last_unstale":"2026-02-26T03:01:54.180125+0000","last_undegraded":"2026-02-26T03:01:34.164969+0000","last_fullsized":"2026-02-26T03:01:34.164969+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-26T03:01:18.659251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-26T03:01:18.659251+0000","last_clean_scrub_stamp":"2026-02-26T03:01:18.659251+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-27T03:50:25.293079+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00037062400000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":59140,"num_objects":95,"num_object_clones":0,"num_object_copies":190,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":95,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":110,"num_write_kb":105,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":798720,"data_stored":124240,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":15,"ondisk_log_size":15,"up":2,"acting":2,"num_store_stats":2},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":1,"up_from":0,"seq":2,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":8,"seq":34359738380,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27528,"kb_used_data":548,"kb_used_omap":1,"kb_used_meta":26942,"kb_avail":94344312,"statfs":{"total":96636764160,"available":96608575488,"internally_reserved":0,"allocated":561152,"data_stored":110221,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27589062},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":115964116996,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":409600,"data_stored":65100,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":389120,"data_stored":59140,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-26T03:02:18.645 DEBUG:teuthology.orchestra.run.trial197:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-26T03:02:18.801 INFO:teuthology.orchestra.run.trial197.stdout: 2026-02-26T03:02:18.801 INFO:teuthology.orchestra.run.trial197.stderr:dumped all 2026-02-26T03:02:18.809 INFO:teuthology.orchestra.run.trial197.stdout:{"pg_ready":true,"pg_map":{"version":63,"stamp":"2026-02-26T03:02:18.385895+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":64578,"num_objects":102,"num_object_clones":0,"num_object_copies":204,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":100,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":102,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":103,"num_write_kb":103,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":19,"ondisk_log_size":19,"up":13,"acting":15,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":21,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":87524,"kb_used_data":1484,"kb_used_omap":10,"kb_used_meta":85941,"kb_avail":283027996,"statfs":{"total":289910292480,"available":289820667904,"internally_reserved":0,"allocated":1519616,"data_stored":357730,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":10460,"internal_metadata":88004388},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":5419,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":100,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":-9,"num_write_kb":-4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.180783"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":32,"reported_epoch":32,"state":"active+clean+remapped","last_fresh":"2026-02-26T03:02:16.406189+0000","last_change":"2026-02-26T03:02:16.406189+0000","last_active":"2026-02-26T03:02:16.406189+0000","last_peered":"2026-02-26T03:02:16.406189+0000","last_clean":"2026-02-26T03:02:16.406189+0000","last_became_active":"2026-02-26T03:02:16.405955+0000","last_became_peered":"2026-02-26T03:02:16.405955+0000","last_unstale":"2026-02-26T03:02:16.406189+0000","last_undegraded":"2026-02-26T03:02:16.406189+0000","last_fullsized":"2026-02-26T03:02:16.406189+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-26T03:01:18.659251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-26T03:01:18.659251+0000","last_clean_scrub_stamp":"2026-02-26T03:01:18.659251+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-27T04:51:09.223889+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00060011299999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1,0],"avail_no_missing":["1","0"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":32,"reported_epoch":32,"state":"active+clean","last_fresh":"2026-02-26T03:02:16.405639+0000","last_change":"2026-02-26T03:02:16.405639+0000","last_active":"2026-02-26T03:02:16.405639+0000","last_peered":"2026-02-26T03:02:16.405639+0000","last_clean":"2026-02-26T03:02:16.405639+0000","last_became_active":"2026-02-26T03:02:16.405505+0000","last_became_peered":"2026-02-26T03:02:16.405505+0000","last_unstale":"2026-02-26T03:02:16.405639+0000","last_undegraded":"2026-02-26T03:02:16.405639+0000","last_fullsized":"2026-02-26T03:02:16.405639+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-26T03:01:18.659251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-26T03:01:18.659251+0000","last_clean_scrub_stamp":"2026-02-26T03:01:18.659251+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-27T14:28:28.066117+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00068042299999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":12,"reported_epoch":32,"state":"active+undersized","last_fresh":"2026-02-26T03:02:16.405009+0000","last_change":"2026-02-26T03:02:14.149472+0000","last_active":"2026-02-26T03:02:16.405009+0000","last_peered":"2026-02-26T03:02:16.405009+0000","last_clean":"2026-02-26T03:01:18.659251+0000","last_became_active":"2026-02-26T03:02:14.149472+0000","last_became_peered":"2026-02-26T03:02:14.149472+0000","last_unstale":"2026-02-26T03:02:16.405009+0000","last_undegraded":"2026-02-26T03:02:16.405009+0000","last_fullsized":"2026-02-26T03:02:14.148088+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-26T03:01:18.659251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-26T03:01:18.659251+0000","last_clean_scrub_stamp":"2026-02-26T03:01:18.659251+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-27T06:49:39.700070+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0],"acting":[0],"avail_no_missing":["0"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":32,"reported_epoch":32,"state":"active+clean","last_fresh":"2026-02-26T03:02:16.406146+0000","last_change":"2026-02-26T03:02:16.406146+0000","last_active":"2026-02-26T03:02:16.406146+0000","last_peered":"2026-02-26T03:02:16.406146+0000","last_clean":"2026-02-26T03:02:16.406146+0000","last_became_active":"2026-02-26T03:02:16.405921+0000","last_became_peered":"2026-02-26T03:02:16.405921+0000","last_unstale":"2026-02-26T03:02:16.406146+0000","last_undegraded":"2026-02-26T03:02:16.406146+0000","last_fullsized":"2026-02-26T03:02:16.406146+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-26T03:01:18.659251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-26T03:01:18.659251+0000","last_clean_scrub_stamp":"2026-02-26T03:01:18.659251+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-27T08:15:04.934373+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000482733,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"31'101","reported_seq":121,"reported_epoch":32,"state":"active+undersized+degraded","last_fresh":"2026-02-26T03:02:16.151582+0000","last_change":"2026-02-26T03:02:15.149270+0000","last_active":"2026-02-26T03:02:16.151582+0000","last_peered":"2026-02-26T03:02:16.151582+0000","last_clean":"2026-02-26T03:01:41.616735+0000","last_became_active":"2026-02-26T03:02:15.149270+0000","last_became_peered":"2026-02-26T03:02:15.149270+0000","last_unstale":"2026-02-26T03:02:16.151582+0000","last_undegraded":"2026-02-26T03:02:15.148613+0000","last_fullsized":"2026-02-26T03:02:15.148549+0000","mapping_epoch":30,"log_start":"20'85","ondisk_log_start":"20'85","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-26T03:01:32.157530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-26T03:01:32.157530+0000","last_clean_scrub_stamp":"2026-02-26T03:01:32.157530+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":85,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-27T04:17:32.398297+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":64559,"num_objects":100,"num_object_clones":0,"num_object_copies":200,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":100,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":100,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":101,"num_write_kb":101,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":100}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.3","version":"11'1","reported_seq":37,"reported_epoch":32,"state":"active+clean+remapped","last_fresh":"2026-02-26T03:02:16.407407+0000","last_change":"2026-02-26T03:02:16.407407+0000","last_active":"2026-02-26T03:02:16.407407+0000","last_peered":"2026-02-26T03:02:16.407407+0000","last_clean":"2026-02-26T03:02:16.407407+0000","last_became_active":"2026-02-26T03:02:16.405936+0000","last_became_peered":"2026-02-26T03:02:16.405936+0000","last_unstale":"2026-02-26T03:02:16.407407+0000","last_undegraded":"2026-02-26T03:02:16.407407+0000","last_fullsized":"2026-02-26T03:02:16.407407+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-26T03:01:18.659251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-26T03:01:18.659251+0000","last_clean_scrub_stamp":"2026-02-26T03:01:18.659251+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-27T06:14:23.045829+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00034461499999999999,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1,0],"avail_no_missing":["1","0"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"13'2","reported_seq":37,"reported_epoch":32,"state":"active+clean","last_fresh":"2026-02-26T03:02:16.404856+0000","last_change":"2026-02-26T03:02:15.149310+0000","last_active":"2026-02-26T03:02:16.404856+0000","last_peered":"2026-02-26T03:02:16.404856+0000","last_clean":"2026-02-26T03:02:16.404856+0000","last_became_active":"2026-02-26T03:02:15.149144+0000","last_became_peered":"2026-02-26T03:02:15.149144+0000","last_unstale":"2026-02-26T03:02:16.404856+0000","last_undegraded":"2026-02-26T03:02:16.404856+0000","last_fullsized":"2026-02-26T03:02:16.404856+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-26T03:01:18.659251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-26T03:01:18.659251+0000","last_clean_scrub_stamp":"2026-02-26T03:01:18.659251+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-27T11:49:30.505493+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00054467999999999997,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":12,"reported_epoch":32,"state":"active+undersized","last_fresh":"2026-02-26T03:02:16.404997+0000","last_change":"2026-02-26T03:02:14.149162+0000","last_active":"2026-02-26T03:02:16.404997+0000","last_peered":"2026-02-26T03:02:16.404997+0000","last_clean":"2026-02-26T03:01:18.659251+0000","last_became_active":"2026-02-26T03:02:14.149162+0000","last_became_peered":"2026-02-26T03:02:14.149162+0000","last_unstale":"2026-02-26T03:02:16.404997+0000","last_undegraded":"2026-02-26T03:02:16.404997+0000","last_fullsized":"2026-02-26T03:02:14.147876+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-26T03:01:18.659251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-26T03:01:18.659251+0000","last_clean_scrub_stamp":"2026-02-26T03:01:18.659251+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-27T05:39:28.378674+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0],"acting":[0],"avail_no_missing":["0"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":32,"reported_epoch":32,"state":"active+clean","last_fresh":"2026-02-26T03:02:16.406053+0000","last_change":"2026-02-26T03:02:16.406053+0000","last_active":"2026-02-26T03:02:16.406053+0000","last_peered":"2026-02-26T03:02:16.406053+0000","last_clean":"2026-02-26T03:02:16.406053+0000","last_became_active":"2026-02-26T03:02:16.405837+0000","last_became_peered":"2026-02-26T03:02:16.405837+0000","last_unstale":"2026-02-26T03:02:16.406053+0000","last_undegraded":"2026-02-26T03:02:16.406053+0000","last_fullsized":"2026-02-26T03:02:16.406053+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-26T03:01:18.659251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-26T03:01:18.659251+0000","last_clean_scrub_stamp":"2026-02-26T03:01:18.659251+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-27T14:23:01.074046+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00037062400000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":64559,"num_objects":100,"num_object_clones":0,"num_object_copies":200,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":100,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":100,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":101,"num_write_kb":101,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":798720,"data_stored":123699,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":16,"ondisk_log_size":16,"up":1,"acting":1,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":12,"acting":14,"num_store_stats":3}],"osd_stats":[{"osd":1,"up_from":29,"seq":124554051587,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32244,"kb_used_data":668,"kb_used_omap":4,"kb_used_meta":31547,"kb_avail":94339596,"statfs":{"total":96636764160,"available":96603746304,"internally_reserved":0,"allocated":684032,"data_stored":156034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4434,"internal_metadata":32304814},"hb_peers":[0],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":8,"seq":34359738380,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27528,"kb_used_data":548,"kb_used_omap":1,"kb_used_meta":26942,"kb_avail":94344312,"statfs":{"total":96636764160,"available":96608575488,"internally_reserved":0,"allocated":561152,"data_stored":110221,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27589062},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":27,"seq":115964116997,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27752,"kb_used_data":268,"kb_used_omap":4,"kb_used_meta":27451,"kb_avail":94344088,"statfs":{"total":96636764160,"available":96608346112,"internally_reserved":0,"allocated":274432,"data_stored":91475,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4432,"internal_metadata":28110512},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":409600,"data_stored":64559,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":389120,"data_stored":59140,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-26T03:02:18.810 INFO:tasks.ceph.ceph_manager.ceph:recovered! 2026-02-26T03:02:18.810 INFO:tasks.divergent_priors2:delay recovery 2026-02-26T03:02:18.810 DEBUG:teuthology.orchestra.run.trial197:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok set_recovery_delay 100000 2026-02-26T03:02:18.882 INFO:teuthology.orchestra.run.trial197.stdout:set_recovery_delay: set osd_recovery_delay_start to 100000 2026-02-26T03:02:18.887 DEBUG:teuthology.orchestra.run.trial197:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok set_recovery_delay 100000 2026-02-26T03:02:18.959 INFO:teuthology.orchestra.run.trial197.stdout:set_recovery_delay: set osd_recovery_delay_start to 100000 2026-02-26T03:02:18.965 INFO:tasks.divergent_priors2:revive divergent 2 2026-02-26T03:02:18.965 DEBUG:teuthology.orchestra.run.trial197:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set noup 2026-02-26T03:02:19.154 INFO:teuthology.orchestra.run.trial197.stderr:noup is set 2026-02-26T03:02:19.163 INFO:tasks.ceph.osd.2:Restarting daemon 2026-02-26T03:02:19.164 DEBUG:teuthology.orchestra.run.trial197:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 2 2026-02-26T03:02:19.166 INFO:tasks.ceph.osd.2:Started 2026-02-26T03:02:19.167 DEBUG:teuthology.orchestra.run.trial197:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2026-02-26T03:02:19.231 INFO:teuthology.orchestra.run.trial197.stderr:no valid command found; 10 closest matches: 2026-02-26T03:02:19.231 INFO:teuthology.orchestra.run.trial197.stderr:0 2026-02-26T03:02:19.231 INFO:teuthology.orchestra.run.trial197.stderr:1 2026-02-26T03:02:19.231 INFO:teuthology.orchestra.run.trial197.stderr:2 2026-02-26T03:02:19.231 INFO:teuthology.orchestra.run.trial197.stderr:abort 2026-02-26T03:02:19.231 INFO:teuthology.orchestra.run.trial197.stderr:assert 2026-02-26T03:02:19.231 INFO:teuthology.orchestra.run.trial197.stderr:config diff 2026-02-26T03:02:19.231 INFO:teuthology.orchestra.run.trial197.stderr:config diff get 2026-02-26T03:02:19.231 INFO:teuthology.orchestra.run.trial197.stderr:config get 2026-02-26T03:02:19.231 INFO:teuthology.orchestra.run.trial197.stderr:config help [] 2026-02-26T03:02:19.231 INFO:teuthology.orchestra.run.trial197.stderr:config set ... 2026-02-26T03:02:19.231 INFO:teuthology.orchestra.run.trial197.stderr:admin_socket: invalid command 2026-02-26T03:02:19.233 DEBUG:teuthology.orchestra.run:got remote process result: 22 2026-02-26T03:02:19.233 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-2, ['dump_ops_in_flight'] 2026-02-26T03:02:20.556 INFO:tasks.ceph.osd.2.trial197.stderr:2026-02-26T03:02:20.554+0000 7f0c3b031780 -1 Falling back to public interface 2026-02-26T03:02:24.052 INFO:tasks.ceph.osd.2.trial197.stderr:2026-02-26T03:02:24.050+0000 7f0c3b031780 -1 osd.2 20 log_to_monitors true 2026-02-26T03:02:24.167 INFO:tasks.ceph.osd.2.trial197.stderr:2026-02-26T03:02:24.165+0000 7f0c32393640 -1 osd.2 20 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2026-02-26T03:02:24.169 INFO:tasks.ceph.osd.2.trial197.stderr:2026-02-26T03:02:24.167+0000 7f0c36d7c640 -1 osd.2 34 osdmap NOUP flag is set, waiting for it to clear 2026-02-26T03:02:24.234 DEBUG:teuthology.orchestra.run.trial197:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2026-02-26T03:02:24.306 INFO:teuthology.orchestra.run.trial197.stdout:{ 2026-02-26T03:02:24.306 INFO:teuthology.orchestra.run.trial197.stdout: "ops": [], 2026-02-26T03:02:24.306 INFO:teuthology.orchestra.run.trial197.stdout: "num_ops": 0 2026-02-26T03:02:24.306 INFO:teuthology.orchestra.run.trial197.stdout:} 2026-02-26T03:02:24.312 INFO:tasks.divergent_priors2:delay recovery divergent 2026-02-26T03:02:24.312 DEBUG:teuthology.orchestra.run.trial197:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok set_recovery_delay 100000 2026-02-26T03:02:24.383 INFO:teuthology.orchestra.run.trial197.stdout:set_recovery_delay: set osd_recovery_delay_start to 100000 2026-02-26T03:02:24.389 DEBUG:teuthology.orchestra.run.trial197:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset noup 2026-02-26T03:02:25.160 INFO:teuthology.orchestra.run.trial197.stderr:noup is unset 2026-02-26T03:02:25.162 INFO:tasks.ceph.osd.2.trial197.stderr:2026-02-26T03:02:25.160+0000 7f0c32393640 -1 osd.2 35 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2026-02-26T03:02:25.172 DEBUG:teuthology.orchestra.run.trial197:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump 2026-02-26T03:02:25.328 INFO:teuthology.orchestra.run.trial197.stdout:epoch 35 2026-02-26T03:02:25.329 INFO:teuthology.orchestra.run.trial197.stdout:fsid 658087b3-2758-487d-ba7c-f8b171a8f312 2026-02-26T03:02:25.329 INFO:teuthology.orchestra.run.trial197.stdout:created 2026-02-26T03:01:04.144741+0000 2026-02-26T03:02:25.329 INFO:teuthology.orchestra.run.trial197.stdout:modified 2026-02-26T03:02:25.157388+0000 2026-02-26T03:02:25.329 INFO:teuthology.orchestra.run.trial197.stdout:flags nodown,noout,noin,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit 2026-02-26T03:02:25.329 INFO:teuthology.orchestra.run.trial197.stdout:crush_version 8 2026-02-26T03:02:25.329 INFO:teuthology.orchestra.run.trial197.stdout:full_ratio 0.95 2026-02-26T03:02:25.329 INFO:teuthology.orchestra.run.trial197.stdout:backfillfull_ratio 0.9 2026-02-26T03:02:25.329 INFO:teuthology.orchestra.run.trial197.stdout:nearfull_ratio 0.85 2026-02-26T03:02:25.329 INFO:teuthology.orchestra.run.trial197.stdout:require_min_compat_client luminous 2026-02-26T03:02:25.329 INFO:teuthology.orchestra.run.trial197.stdout:min_compat_client jewel 2026-02-26T03:02:25.330 INFO:teuthology.orchestra.run.trial197.stdout:require_osd_release squid 2026-02-26T03:02:25.330 INFO:teuthology.orchestra.run.trial197.stdout:stretch_mode_enabled false 2026-02-26T03:02:25.330 INFO:teuthology.orchestra.run.trial197.stdout:pool 1 'rbd' replicated size 2 min_size 1 crush_rule 0 object_hash rjenkins pg_num 8 pgp_num 8 autoscale_mode off last_change 13 flags hashpspool,selfmanaged_snaps stripe_width 0 application rbd read_balance_score 1.87 2026-02-26T03:02:25.330 INFO:teuthology.orchestra.run.trial197.stdout:pool 2 'foo' replicated size 2 min_size 1 crush_rule 0 object_hash rjenkins pg_num 1 pgp_num 1 autoscale_mode off last_change 20 flags hashpspool stripe_width 0 read_balance_score 3.03 2026-02-26T03:02:25.330 INFO:teuthology.orchestra.run.trial197.stdout:max_osd 3 2026-02-26T03:02:25.330 INFO:teuthology.orchestra.run.trial197.stdout:osd.0 up in weight 1 up_from 27 up_thru 30 down_at 21 last_clean_interval [8,20) [v2:10.20.193.197:6800/1385562264,v1:10.20.193.197:6801/1385562264] [v2:10.20.193.197:6802/1385562264,v1:10.20.193.197:6803/1385562264] exists,up 696e32c7-b2ad-41e8-9c32-f7e12eb617ee 2026-02-26T03:02:25.330 INFO:teuthology.orchestra.run.trial197.stdout:osd.1 up in weight 1 up_from 29 up_thru 31 down_at 22 last_clean_interval [9,20) [v2:10.20.193.197:6810/4100957723,v1:10.20.193.197:6811/4100957723] [v2:10.20.193.197:6812/4100957723,v1:10.20.193.197:6813/4100957723] exists,up 606c6d1b-0452-4860-a1dc-ca58763aa4e9 2026-02-26T03:02:25.330 INFO:teuthology.orchestra.run.trial197.stdout:osd.2 down in weight 1 up_from 8 up_thru 18 down_at 23 last_clean_interval [0,0) [v2:10.20.193.197:6810/2617788373,v1:10.20.193.197:6811/2617788373] [v2:10.20.193.197:6812/2617788373,v1:10.20.193.197:6813/2617788373] exists 4a54c53b-ac96-4483-97dc-bf76ae085e1b 2026-02-26T03:02:25.330 INFO:teuthology.orchestra.run.trial197.stdout:pg_temp 1.3 [1,0] 2026-02-26T03:02:25.330 INFO:teuthology.orchestra.run.trial197.stdout:pg_temp 1.7 [1,0] 2026-02-26T03:02:25.336 INFO:tasks.ceph.ceph_manager.ceph:['osd.0 up in weight 1 up_from 27 up_thru 30 down_at 21 last_clean_interval [8,20) [v2:10.20.193.197:6800/1385562264,v1:10.20.193.197:6801/1385562264] [v2:10.20.193.197:6802/1385562264,v1:10.20.193.197:6803/1385562264] exists,up 696e32c7-b2ad-41e8-9c32-f7e12eb617ee', 'osd.1 up in weight 1 up_from 29 up_thru 31 down_at 22 last_clean_interval [9,20) [v2:10.20.193.197:6810/4100957723,v1:10.20.193.197:6811/4100957723] [v2:10.20.193.197:6812/4100957723,v1:10.20.193.197:6813/4100957723] exists,up 606c6d1b-0452-4860-a1dc-ca58763aa4e9', 'osd.2 down in weight 1 up_from 8 up_thru 18 down_at 23 last_clean_interval [0,0) [v2:10.20.193.197:6810/2617788373,v1:10.20.193.197:6811/2617788373] [v2:10.20.193.197:6812/2617788373,v1:10.20.193.197:6813/2617788373] exists 4a54c53b-ac96-4483-97dc-bf76ae085e1b'] 2026-02-26T03:02:35.338 DEBUG:teuthology.orchestra.run.trial197:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump 2026-02-26T03:02:35.497 INFO:teuthology.orchestra.run.trial197.stdout:epoch 38 2026-02-26T03:02:35.497 INFO:teuthology.orchestra.run.trial197.stdout:fsid 658087b3-2758-487d-ba7c-f8b171a8f312 2026-02-26T03:02:35.497 INFO:teuthology.orchestra.run.trial197.stdout:created 2026-02-26T03:01:04.144741+0000 2026-02-26T03:02:35.497 INFO:teuthology.orchestra.run.trial197.stdout:modified 2026-02-26T03:02:28.165358+0000 2026-02-26T03:02:35.497 INFO:teuthology.orchestra.run.trial197.stdout:flags nodown,noout,noin,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit 2026-02-26T03:02:35.497 INFO:teuthology.orchestra.run.trial197.stdout:crush_version 8 2026-02-26T03:02:35.497 INFO:teuthology.orchestra.run.trial197.stdout:full_ratio 0.95 2026-02-26T03:02:35.497 INFO:teuthology.orchestra.run.trial197.stdout:backfillfull_ratio 0.9 2026-02-26T03:02:35.497 INFO:teuthology.orchestra.run.trial197.stdout:nearfull_ratio 0.85 2026-02-26T03:02:35.497 INFO:teuthology.orchestra.run.trial197.stdout:require_min_compat_client luminous 2026-02-26T03:02:35.497 INFO:teuthology.orchestra.run.trial197.stdout:min_compat_client jewel 2026-02-26T03:02:35.497 INFO:teuthology.orchestra.run.trial197.stdout:require_osd_release squid 2026-02-26T03:02:35.498 INFO:teuthology.orchestra.run.trial197.stdout:stretch_mode_enabled false 2026-02-26T03:02:35.498 INFO:teuthology.orchestra.run.trial197.stdout:pool 1 'rbd' replicated size 2 min_size 1 crush_rule 0 object_hash rjenkins pg_num 8 pgp_num 8 autoscale_mode off last_change 13 flags hashpspool,selfmanaged_snaps stripe_width 0 application rbd read_balance_score 1.88 2026-02-26T03:02:35.498 INFO:teuthology.orchestra.run.trial197.stdout:pool 2 'foo' replicated size 2 min_size 1 crush_rule 0 object_hash rjenkins pg_num 1 pgp_num 1 autoscale_mode off last_change 20 flags hashpspool stripe_width 0 read_balance_score 2.99 2026-02-26T03:02:35.498 INFO:teuthology.orchestra.run.trial197.stdout:max_osd 3 2026-02-26T03:02:35.498 INFO:teuthology.orchestra.run.trial197.stdout:osd.0 up in weight 1 up_from 27 up_thru 30 down_at 21 last_clean_interval [8,20) [v2:10.20.193.197:6800/1385562264,v1:10.20.193.197:6801/1385562264] [v2:10.20.193.197:6802/1385562264,v1:10.20.193.197:6803/1385562264] exists,up 696e32c7-b2ad-41e8-9c32-f7e12eb617ee 2026-02-26T03:02:35.498 INFO:teuthology.orchestra.run.trial197.stdout:osd.1 up in weight 1 up_from 29 up_thru 37 down_at 22 last_clean_interval [9,20) [v2:10.20.193.197:6810/4100957723,v1:10.20.193.197:6811/4100957723] [v2:10.20.193.197:6812/4100957723,v1:10.20.193.197:6813/4100957723] exists,up 606c6d1b-0452-4860-a1dc-ca58763aa4e9 2026-02-26T03:02:35.498 INFO:teuthology.orchestra.run.trial197.stdout:osd.2 up in weight 1 up_from 36 up_thru 36 down_at 23 last_clean_interval [8,20) [v2:10.20.193.197:6818/3326824975,v1:10.20.193.197:6819/3326824975] [v2:10.20.193.197:6820/3326824975,v1:10.20.193.197:6821/3326824975] exists,up 4a54c53b-ac96-4483-97dc-bf76ae085e1b 2026-02-26T03:02:35.505 INFO:tasks.ceph.ceph_manager.ceph:['osd.0 up in weight 1 up_from 27 up_thru 30 down_at 21 last_clean_interval [8,20) [v2:10.20.193.197:6800/1385562264,v1:10.20.193.197:6801/1385562264] [v2:10.20.193.197:6802/1385562264,v1:10.20.193.197:6803/1385562264] exists,up 696e32c7-b2ad-41e8-9c32-f7e12eb617ee', 'osd.1 up in weight 1 up_from 29 up_thru 37 down_at 22 last_clean_interval [9,20) [v2:10.20.193.197:6810/4100957723,v1:10.20.193.197:6811/4100957723] [v2:10.20.193.197:6812/4100957723,v1:10.20.193.197:6813/4100957723] exists,up 606c6d1b-0452-4860-a1dc-ca58763aa4e9', 'osd.2 up in weight 1 up_from 36 up_thru 36 down_at 23 last_clean_interval [8,20) [v2:10.20.193.197:6818/3326824975,v1:10.20.193.197:6819/3326824975] [v2:10.20.193.197:6820/3326824975,v1:10.20.193.197:6821/3326824975] exists,up 4a54c53b-ac96-4483-97dc-bf76ae085e1b'] 2026-02-26T03:02:35.505 INFO:tasks.divergent_priors2:wait for peering 2026-02-26T03:02:35.505 INFO:tasks.util.rados:rados -p foo put foo /etc/fstab 2026-02-26T03:02:35.505 DEBUG:teuthology.orchestra.run.trial197:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p foo put foo /etc/fstab 2026-02-26T03:02:35.543 INFO:tasks.divergent_priors2:killing divergent 2 2026-02-26T03:02:35.543 DEBUG:tasks.ceph.osd.2:waiting for process to exit 2026-02-26T03:02:35.543 INFO:teuthology.orchestra.run:waiting for 300 2026-02-26T03:02:35.621 INFO:tasks.ceph.osd.2:Stopped 2026-02-26T03:02:35.621 DEBUG:teuthology.orchestra.run.trial197:> sudo adjust-ulimits ceph-objectstore-tool --data-path /var/lib/ceph/osd/ceph-2 --journal-path /var/lib/ceph/osd/ceph-2/journal --log-file=/var/log/ceph/objectstore_tool.$$.log --op export-remove --pgid 2.0 --file /home/ubuntu/cephtest/exp.771639.out 2026-02-26T03:02:37.294 INFO:teuthology.orchestra.run.trial197.stderr:Exporting 2.0 info 2.0( v 38'102 lc 0'0 (20'95,38'102] local-lis/les=36/37 n=101 ec=18/18 lis/c=36/18 les/c/f=37/19/0 sis=36) 2026-02-26T03:02:37.318 INFO:teuthology.orchestra.run.trial197.stderr:Read #2:023dd3d2:::existing_12:head# 2026-02-26T03:02:37.319 INFO:teuthology.orchestra.run.trial197.stderr:Read #2:04c40a0c:::existing_93:head# 2026-02-26T03:02:37.319 INFO:teuthology.orchestra.run.trial197.stderr:Read #2:064ff5fb:::existing_36:head# 2026-02-26T03:02:37.319 INFO:teuthology.orchestra.run.trial197.stderr:Read #2:091f9f1d:::existing_94:head# 2026-02-26T03:02:37.319 INFO:teuthology.orchestra.run.trial197.stderr:Read #2:0a865a93:::existing_54:head# 2026-02-26T03:02:37.319 INFO:teuthology.orchestra.run.trial197.stderr:Read #2:0fed513c:::existing_41:head# 2026-02-26T03:02:37.319 INFO:teuthology.orchestra.run.trial197.stderr:Read #2:13b9e0b8:::existing_92:head# 2026-02-26T03:02:37.319 INFO:teuthology.orchestra.run.trial197.stderr:Read #2:169157fa:::existing_81:head# 2026-02-26T03:02:37.319 INFO:teuthology.orchestra.run.trial197.stderr:Read #2:17731382:::existing_58:head# 2026-02-26T03:02:37.319 INFO:teuthology.orchestra.run.trial197.stderr:Read #2:18a45a39:::existing_43:head# 2026-02-26T03:02:37.320 INFO:teuthology.orchestra.run.trial197.stderr:Read #2:1c160f01:::existing_66:head# 2026-02-26T03:02:37.320 INFO:teuthology.orchestra.run.trial197.stderr:Read #2:1dd6c249:::existing_97:head# 2026-02-26T03:02:37.320 INFO:teuthology.orchestra.run.trial197.stderr:Read #2:23e1b063:::existing_59:head# 2026-02-26T03:02:37.320 INFO:teuthology.orchestra.run.trial197.stderr:Read #2:28566edb:::existing_78:head# 2026-02-26T03:02:37.320 INFO:teuthology.orchestra.run.trial197.stderr:Read #2:2a101182:::existing_20:head# 2026-02-26T03:02:37.320 INFO:teuthology.orchestra.run.trial197.stderr:Read #2:2d1519da:::existing_96:head# 2026-02-26T03:02:37.320 INFO:teuthology.orchestra.run.trial197.stderr:Read #2:3078b87a:::existing_86:head# 2026-02-26T03:02:37.320 INFO:teuthology.orchestra.run.trial197.stderr:Read #2:30bde01b:::existing_15:head# 2026-02-26T03:02:37.320 INFO:teuthology.orchestra.run.trial197.stderr:Read #2:317a7e19:::existing_56:head# 2026-02-26T03:02:37.320 INFO:teuthology.orchestra.run.trial197.stderr:Read #2:38b9c7fa:::existing_33:head# 2026-02-26T03:02:37.320 INFO:teuthology.orchestra.run.trial197.stderr:Read #2:3e16d84b:::existing_48:head# 2026-02-26T03:02:37.321 INFO:teuthology.orchestra.run.trial197.stderr:Read #2:45fe8203:::existing_75:head# 2026-02-26T03:02:37.321 INFO:teuthology.orchestra.run.trial197.stderr:Read #2:4e82c9ba:::existing_50:head# 2026-02-26T03:02:37.321 INFO:teuthology.orchestra.run.trial197.stderr:Read #2:5298dcdb:::existing_57:head# 2026-02-26T03:02:37.321 INFO:teuthology.orchestra.run.trial197.stderr:Read #2:52ebaa44:::existing_32:head# 2026-02-26T03:02:37.321 INFO:teuthology.orchestra.run.trial197.stderr:Read #2:5b7e2de6:::existing_84:head# 2026-02-26T03:02:37.321 INFO:teuthology.orchestra.run.trial197.stderr:Read #2:5da0255c:::existing_13:head# 2026-02-26T03:02:37.321 INFO:teuthology.orchestra.run.trial197.stderr:Read #2:5da2dab8:::existing_53:head# 2026-02-26T03:02:37.321 INFO:teuthology.orchestra.run.trial197.stderr:Read #2:5ee3514b:::existing_30:head# 2026-02-26T03:02:37.321 INFO:teuthology.orchestra.run.trial197.stderr:Read #2:602f83fe:::foo:head# 2026-02-26T03:02:37.321 INFO:teuthology.orchestra.run.trial197.stderr:Read #2:60426cc7:::existing_62:head# 2026-02-26T03:02:37.322 INFO:teuthology.orchestra.run.trial197.stderr:Read #2:628918ca:::existing_79:head# 2026-02-26T03:02:37.322 INFO:teuthology.orchestra.run.trial197.stderr:Read #2:63f8ab2d:::existing_77:head# 2026-02-26T03:02:37.322 INFO:teuthology.orchestra.run.trial197.stderr:Read #2:67e1780f:::existing_27:head# 2026-02-26T03:02:37.322 INFO:teuthology.orchestra.run.trial197.stderr:Read #2:68143ca1:::existing_71:head# 2026-02-26T03:02:37.322 INFO:teuthology.orchestra.run.trial197.stderr:Read #2:6bac9509:::existing_29:head# 2026-02-26T03:02:37.322 INFO:teuthology.orchestra.run.trial197.stderr:Read #2:6bbec798:::existing_45:head# 2026-02-26T03:02:37.322 INFO:teuthology.orchestra.run.trial197.stderr:Read #2:6cb23a40:::existing_65:head# 2026-02-26T03:02:37.322 INFO:teuthology.orchestra.run.trial197.stderr:Read #2:7055ec5e:::existing_83:head# 2026-02-26T03:02:37.322 INFO:teuthology.orchestra.run.trial197.stderr:Read #2:70a54e2b:::existing_87:head# 2026-02-26T03:02:37.322 INFO:teuthology.orchestra.run.trial197.stderr:Read #2:750ace21:::existing_19:head# 2026-02-26T03:02:37.322 INFO:teuthology.orchestra.run.trial197.stderr:Read #2:79881152:::existing_63:head# 2026-02-26T03:02:37.323 INFO:teuthology.orchestra.run.trial197.stderr:Read #2:7a80c0d5:::existing_99:head# 2026-02-26T03:02:37.323 INFO:teuthology.orchestra.run.trial197.stderr:Read #2:81490677:::existing_76:head# 2026-02-26T03:02:37.323 INFO:teuthology.orchestra.run.trial197.stderr:Read #2:85ec8e5d:::existing_85:head# 2026-02-26T03:02:37.323 INFO:teuthology.orchestra.run.trial197.stderr:Read #2:8e2abbf0:::existing_49:head# 2026-02-26T03:02:37.323 INFO:teuthology.orchestra.run.trial197.stderr:Read #2:8ee0fe0f:::existing_52:head# 2026-02-26T03:02:37.323 INFO:teuthology.orchestra.run.trial197.stderr:Read #2:92731e71:::existing_89:head# 2026-02-26T03:02:37.323 INFO:teuthology.orchestra.run.trial197.stderr:Read #2:928205cf:::existing_82:head# 2026-02-26T03:02:37.323 INFO:teuthology.orchestra.run.trial197.stderr:Read #2:93be2267:::existing_24:head# 2026-02-26T03:02:37.323 INFO:teuthology.orchestra.run.trial197.stderr:Read #2:99ceda2c:::existing_44:head# 2026-02-26T03:02:37.323 INFO:teuthology.orchestra.run.trial197.stderr:Read #2:9dd917ee:::existing_26:head# 2026-02-26T03:02:37.323 INFO:teuthology.orchestra.run.trial197.stderr:Read #2:a28805a6:::existing_72:head# 2026-02-26T03:02:37.324 INFO:teuthology.orchestra.run.trial197.stderr:Read #2:a294a8f6:::existing_16:head# 2026-02-26T03:02:37.324 INFO:teuthology.orchestra.run.trial197.stderr:Read #2:a2f5473a:::existing_42:head# 2026-02-26T03:02:37.324 INFO:teuthology.orchestra.run.trial197.stderr:Read #2:a504a533:::existing_90:head# 2026-02-26T03:02:37.324 INFO:teuthology.orchestra.run.trial197.stderr:Read #2:a691337e:::existing_61:head# 2026-02-26T03:02:37.324 INFO:teuthology.orchestra.run.trial197.stderr:Read #2:a6e8d7ac:::existing_28:head# 2026-02-26T03:02:37.324 INFO:teuthology.orchestra.run.trial197.stderr:Read #2:a9fde86e:::existing_18:head# 2026-02-26T03:02:37.324 INFO:teuthology.orchestra.run.trial197.stderr:Read #2:aa309faa:::existing_68:head# 2026-02-26T03:02:37.324 INFO:teuthology.orchestra.run.trial197.stderr:Read #2:ae40b0ed:::existing_31:head# 2026-02-26T03:02:37.324 INFO:teuthology.orchestra.run.trial197.stderr:Read #2:b279b1fe:::existing_10:head# 2026-02-26T03:02:37.324 INFO:teuthology.orchestra.run.trial197.stderr:Read #2:b343e277:::existing_88:head# 2026-02-26T03:02:37.324 INFO:teuthology.orchestra.run.trial197.stderr:Read #2:b7130e90:::existing_80:head# 2026-02-26T03:02:37.325 INFO:teuthology.orchestra.run.trial197.stderr:Read #2:bc6f7309:::existing_70:head# 2026-02-26T03:02:37.325 INFO:teuthology.orchestra.run.trial197.stderr:Read #2:be8489b7:::existing_38:head# 2026-02-26T03:02:37.325 INFO:teuthology.orchestra.run.trial197.stderr:Read #2:beb4e5e3:::existing_47:head# 2026-02-26T03:02:37.325 INFO:teuthology.orchestra.run.trial197.stderr:Read #2:bebec877:::existing_74:head# 2026-02-26T03:02:37.325 INFO:teuthology.orchestra.run.trial197.stderr:Read #2:c04347d8:::existing_60:head# 2026-02-26T03:02:37.325 INFO:teuthology.orchestra.run.trial197.stderr:Read #2:c3e6ebe4:::existing_17:head# 2026-02-26T03:02:37.325 INFO:teuthology.orchestra.run.trial197.stderr:Read #2:c6c5a7c2:::existing_39:head# 2026-02-26T03:02:37.325 INFO:teuthology.orchestra.run.trial197.stderr:Read #2:cc1d0abf:::existing_95:head# 2026-02-26T03:02:37.325 INFO:teuthology.orchestra.run.trial197.stderr:Read #2:ccb3865b:::existing_73:head# 2026-02-26T03:02:37.325 INFO:teuthology.orchestra.run.trial197.stderr:Read #2:cd84e19e:::existing_40:head# 2026-02-26T03:02:37.325 INFO:teuthology.orchestra.run.trial197.stderr:Read #2:d2cd64ce:::existing_37:head# 2026-02-26T03:02:37.326 INFO:teuthology.orchestra.run.trial197.stderr:Read #2:d41a25c9:::existing_22:head# 2026-02-26T03:02:37.326 INFO:teuthology.orchestra.run.trial197.stderr:Read #2:d7f088da:::existing_91:head# 2026-02-26T03:02:37.326 INFO:teuthology.orchestra.run.trial197.stderr:Read #2:d9cbd1d6:::existing_64:head# 2026-02-26T03:02:37.326 INFO:teuthology.orchestra.run.trial197.stderr:Read #2:dcc4ff82:::existing_98:head# 2026-02-26T03:02:37.326 INFO:teuthology.orchestra.run.trial197.stderr:Read #2:dd677bd3:::existing_35:head# 2026-02-26T03:02:37.326 INFO:teuthology.orchestra.run.trial197.stderr:Read #2:dd6d90e7:::existing_46:head# 2026-02-26T03:02:37.326 INFO:teuthology.orchestra.run.trial197.stderr:Read #2:e0e22077:::existing_21:head# 2026-02-26T03:02:37.326 INFO:teuthology.orchestra.run.trial197.stderr:Read #2:e255b710:::existing_55:head# 2026-02-26T03:02:37.326 INFO:teuthology.orchestra.run.trial197.stderr:Read #2:eff1d5db:::existing_14:head# 2026-02-26T03:02:37.326 INFO:teuthology.orchestra.run.trial197.stderr:Read #2:f1b156bf:::existing_23:head# 2026-02-26T03:02:37.326 INFO:teuthology.orchestra.run.trial197.stderr:Read #2:f5ce4199:::existing_34:head# 2026-02-26T03:02:37.326 INFO:teuthology.orchestra.run.trial197.stderr:Read #2:f6809456:::existing_11:head# 2026-02-26T03:02:37.327 INFO:teuthology.orchestra.run.trial197.stderr:Read #2:f699a2a9:::existing_67:head# 2026-02-26T03:02:37.327 INFO:teuthology.orchestra.run.trial197.stderr:Read #2:f9baac43:::existing_25:head# 2026-02-26T03:02:37.327 INFO:teuthology.orchestra.run.trial197.stderr:Read #2:fb76688e:::existing_51:head# 2026-02-26T03:02:37.327 INFO:teuthology.orchestra.run.trial197.stderr:Read #2:fbb92eec:::existing_69:head# 2026-02-26T03:02:37.327 INFO:teuthology.orchestra.run.trial197.stderr:Export successful 2026-02-26T03:02:37.328 INFO:teuthology.orchestra.run.trial197.stdout: marking collection for removal 2026-02-26T03:02:37.328 INFO:teuthology.orchestra.run.trial197.stdout:setting '_remove' omap key 2026-02-26T03:02:37.328 INFO:teuthology.orchestra.run.trial197.stdout:finish_remove_pgs 2.0_head removing 2.0 2026-02-26T03:02:37.328 INFO:teuthology.orchestra.run.trial197.stderr:Remove successful 2026-02-26T03:02:37.832 DEBUG:teuthology.orchestra.run.trial197:> sudo adjust-ulimits ceph-objectstore-tool --data-path /var/lib/ceph/osd/ceph-2 --journal-path /var/lib/ceph/osd/ceph-2/journal --log-file=/var/log/ceph/objectstore_tool.$$.log --op import --file /home/ubuntu/cephtest/exp.771639.out 2026-02-26T03:02:39.204 INFO:teuthology.orchestra.run.trial197.stdout:get_pg_num_history pg_num_history pg_num_history(e38 pg_nums {1={10=8},2={18=1}} deleted_pools ) 2026-02-26T03:02:39.205 INFO:teuthology.orchestra.run.trial197.stdout:Importing pgid 2.0 2026-02-26T03:02:39.211 INFO:teuthology.orchestra.run.trial197.stdout:Write #2:023dd3d2:::existing_12:head# 2026-02-26T03:02:39.211 INFO:teuthology.orchestra.run.trial197.stdout:snapset 0=[]:{} 2026-02-26T03:02:39.212 INFO:teuthology.orchestra.run.trial197.stdout:Write #2:04c40a0c:::existing_93:head# 2026-02-26T03:02:39.212 INFO:teuthology.orchestra.run.trial197.stdout:snapset 0=[]:{} 2026-02-26T03:02:39.212 INFO:teuthology.orchestra.run.trial197.stdout:Write #2:064ff5fb:::existing_36:head# 2026-02-26T03:02:39.212 INFO:teuthology.orchestra.run.trial197.stdout:snapset 0=[]:{} 2026-02-26T03:02:39.212 INFO:teuthology.orchestra.run.trial197.stdout:Write #2:091f9f1d:::existing_94:head# 2026-02-26T03:02:39.212 INFO:teuthology.orchestra.run.trial197.stdout:snapset 0=[]:{} 2026-02-26T03:02:39.213 INFO:teuthology.orchestra.run.trial197.stdout:Write #2:0a865a93:::existing_54:head# 2026-02-26T03:02:39.213 INFO:teuthology.orchestra.run.trial197.stdout:snapset 0=[]:{} 2026-02-26T03:02:39.213 INFO:teuthology.orchestra.run.trial197.stdout:Write #2:0fed513c:::existing_41:head# 2026-02-26T03:02:39.213 INFO:teuthology.orchestra.run.trial197.stdout:snapset 0=[]:{} 2026-02-26T03:02:39.213 INFO:teuthology.orchestra.run.trial197.stdout:Write #2:13b9e0b8:::existing_92:head# 2026-02-26T03:02:39.213 INFO:teuthology.orchestra.run.trial197.stdout:snapset 0=[]:{} 2026-02-26T03:02:39.213 INFO:teuthology.orchestra.run.trial197.stdout:Write #2:169157fa:::existing_81:head# 2026-02-26T03:02:39.214 INFO:teuthology.orchestra.run.trial197.stdout:snapset 0=[]:{} 2026-02-26T03:02:39.214 INFO:teuthology.orchestra.run.trial197.stdout:Write #2:17731382:::existing_58:head# 2026-02-26T03:02:39.214 INFO:teuthology.orchestra.run.trial197.stdout:snapset 0=[]:{} 2026-02-26T03:02:39.214 INFO:teuthology.orchestra.run.trial197.stdout:Write #2:18a45a39:::existing_43:head# 2026-02-26T03:02:39.214 INFO:teuthology.orchestra.run.trial197.stdout:snapset 0=[]:{} 2026-02-26T03:02:39.214 INFO:teuthology.orchestra.run.trial197.stdout:Write #2:1c160f01:::existing_66:head# 2026-02-26T03:02:39.214 INFO:teuthology.orchestra.run.trial197.stdout:snapset 0=[]:{} 2026-02-26T03:02:39.215 INFO:teuthology.orchestra.run.trial197.stdout:Write #2:1dd6c249:::existing_97:head# 2026-02-26T03:02:39.215 INFO:teuthology.orchestra.run.trial197.stdout:snapset 0=[]:{} 2026-02-26T03:02:39.215 INFO:teuthology.orchestra.run.trial197.stdout:Write #2:23e1b063:::existing_59:head# 2026-02-26T03:02:39.215 INFO:teuthology.orchestra.run.trial197.stdout:snapset 0=[]:{} 2026-02-26T03:02:39.215 INFO:teuthology.orchestra.run.trial197.stdout:Write #2:28566edb:::existing_78:head# 2026-02-26T03:02:39.215 INFO:teuthology.orchestra.run.trial197.stdout:snapset 0=[]:{} 2026-02-26T03:02:39.216 INFO:teuthology.orchestra.run.trial197.stdout:Write #2:2a101182:::existing_20:head# 2026-02-26T03:02:39.216 INFO:teuthology.orchestra.run.trial197.stdout:snapset 0=[]:{} 2026-02-26T03:02:39.216 INFO:teuthology.orchestra.run.trial197.stdout:Write #2:2d1519da:::existing_96:head# 2026-02-26T03:02:39.216 INFO:teuthology.orchestra.run.trial197.stdout:snapset 0=[]:{} 2026-02-26T03:02:39.216 INFO:teuthology.orchestra.run.trial197.stdout:Write #2:3078b87a:::existing_86:head# 2026-02-26T03:02:39.216 INFO:teuthology.orchestra.run.trial197.stdout:snapset 0=[]:{} 2026-02-26T03:02:39.216 INFO:teuthology.orchestra.run.trial197.stdout:Write #2:30bde01b:::existing_15:head# 2026-02-26T03:02:39.216 INFO:teuthology.orchestra.run.trial197.stdout:snapset 0=[]:{} 2026-02-26T03:02:39.217 INFO:teuthology.orchestra.run.trial197.stdout:Write #2:317a7e19:::existing_56:head# 2026-02-26T03:02:39.217 INFO:teuthology.orchestra.run.trial197.stdout:snapset 0=[]:{} 2026-02-26T03:02:39.217 INFO:teuthology.orchestra.run.trial197.stdout:Write #2:38b9c7fa:::existing_33:head# 2026-02-26T03:02:39.217 INFO:teuthology.orchestra.run.trial197.stdout:snapset 0=[]:{} 2026-02-26T03:02:39.217 INFO:teuthology.orchestra.run.trial197.stdout:Write #2:3e16d84b:::existing_48:head# 2026-02-26T03:02:39.217 INFO:teuthology.orchestra.run.trial197.stdout:snapset 0=[]:{} 2026-02-26T03:02:39.217 INFO:teuthology.orchestra.run.trial197.stdout:Write #2:45fe8203:::existing_75:head# 2026-02-26T03:02:39.217 INFO:teuthology.orchestra.run.trial197.stdout:snapset 0=[]:{} 2026-02-26T03:02:39.217 INFO:teuthology.orchestra.run.trial197.stdout:Write #2:4e82c9ba:::existing_50:head# 2026-02-26T03:02:39.217 INFO:teuthology.orchestra.run.trial197.stdout:snapset 0=[]:{} 2026-02-26T03:02:39.218 INFO:teuthology.orchestra.run.trial197.stdout:Write #2:5298dcdb:::existing_57:head# 2026-02-26T03:02:39.218 INFO:teuthology.orchestra.run.trial197.stdout:snapset 0=[]:{} 2026-02-26T03:02:39.218 INFO:teuthology.orchestra.run.trial197.stdout:Write #2:52ebaa44:::existing_32:head# 2026-02-26T03:02:39.218 INFO:teuthology.orchestra.run.trial197.stdout:snapset 0=[]:{} 2026-02-26T03:02:39.218 INFO:teuthology.orchestra.run.trial197.stdout:Write #2:5b7e2de6:::existing_84:head# 2026-02-26T03:02:39.218 INFO:teuthology.orchestra.run.trial197.stdout:snapset 0=[]:{} 2026-02-26T03:02:39.218 INFO:teuthology.orchestra.run.trial197.stdout:Write #2:5da0255c:::existing_13:head# 2026-02-26T03:02:39.218 INFO:teuthology.orchestra.run.trial197.stdout:snapset 0=[]:{} 2026-02-26T03:02:39.219 INFO:teuthology.orchestra.run.trial197.stdout:Write #2:5da2dab8:::existing_53:head# 2026-02-26T03:02:39.219 INFO:teuthology.orchestra.run.trial197.stdout:snapset 0=[]:{} 2026-02-26T03:02:39.219 INFO:teuthology.orchestra.run.trial197.stdout:Write #2:5ee3514b:::existing_30:head# 2026-02-26T03:02:39.219 INFO:teuthology.orchestra.run.trial197.stdout:snapset 0=[]:{} 2026-02-26T03:02:39.219 INFO:teuthology.orchestra.run.trial197.stdout:Write #2:602f83fe:::foo:head# 2026-02-26T03:02:39.219 INFO:teuthology.orchestra.run.trial197.stdout:snapset 0=[]:{} 2026-02-26T03:02:39.219 INFO:teuthology.orchestra.run.trial197.stdout:Write #2:60426cc7:::existing_62:head# 2026-02-26T03:02:39.219 INFO:teuthology.orchestra.run.trial197.stdout:snapset 0=[]:{} 2026-02-26T03:02:39.220 INFO:teuthology.orchestra.run.trial197.stdout:Write #2:628918ca:::existing_79:head# 2026-02-26T03:02:39.220 INFO:teuthology.orchestra.run.trial197.stdout:snapset 0=[]:{} 2026-02-26T03:02:39.220 INFO:teuthology.orchestra.run.trial197.stdout:Write #2:63f8ab2d:::existing_77:head# 2026-02-26T03:02:39.220 INFO:teuthology.orchestra.run.trial197.stdout:snapset 0=[]:{} 2026-02-26T03:02:39.220 INFO:teuthology.orchestra.run.trial197.stdout:Write #2:67e1780f:::existing_27:head# 2026-02-26T03:02:39.220 INFO:teuthology.orchestra.run.trial197.stdout:snapset 0=[]:{} 2026-02-26T03:02:39.221 INFO:teuthology.orchestra.run.trial197.stdout:Write #2:68143ca1:::existing_71:head# 2026-02-26T03:02:39.221 INFO:teuthology.orchestra.run.trial197.stdout:snapset 0=[]:{} 2026-02-26T03:02:39.221 INFO:teuthology.orchestra.run.trial197.stdout:Write #2:6bac9509:::existing_29:head# 2026-02-26T03:02:39.221 INFO:teuthology.orchestra.run.trial197.stdout:snapset 0=[]:{} 2026-02-26T03:02:39.221 INFO:teuthology.orchestra.run.trial197.stdout:Write #2:6bbec798:::existing_45:head# 2026-02-26T03:02:39.221 INFO:teuthology.orchestra.run.trial197.stdout:snapset 0=[]:{} 2026-02-26T03:02:39.222 INFO:teuthology.orchestra.run.trial197.stdout:Write #2:6cb23a40:::existing_65:head# 2026-02-26T03:02:39.222 INFO:teuthology.orchestra.run.trial197.stdout:snapset 0=[]:{} 2026-02-26T03:02:39.222 INFO:teuthology.orchestra.run.trial197.stdout:Write #2:7055ec5e:::existing_83:head# 2026-02-26T03:02:39.222 INFO:teuthology.orchestra.run.trial197.stdout:snapset 0=[]:{} 2026-02-26T03:02:39.222 INFO:teuthology.orchestra.run.trial197.stdout:Write #2:70a54e2b:::existing_87:head# 2026-02-26T03:02:39.222 INFO:teuthology.orchestra.run.trial197.stdout:snapset 0=[]:{} 2026-02-26T03:02:39.222 INFO:teuthology.orchestra.run.trial197.stdout:Write #2:750ace21:::existing_19:head# 2026-02-26T03:02:39.222 INFO:teuthology.orchestra.run.trial197.stdout:snapset 0=[]:{} 2026-02-26T03:02:39.223 INFO:teuthology.orchestra.run.trial197.stdout:Write #2:79881152:::existing_63:head# 2026-02-26T03:02:39.223 INFO:teuthology.orchestra.run.trial197.stdout:snapset 0=[]:{} 2026-02-26T03:02:39.223 INFO:teuthology.orchestra.run.trial197.stdout:Write #2:7a80c0d5:::existing_99:head# 2026-02-26T03:02:39.223 INFO:teuthology.orchestra.run.trial197.stdout:snapset 0=[]:{} 2026-02-26T03:02:39.223 INFO:teuthology.orchestra.run.trial197.stdout:Write #2:81490677:::existing_76:head# 2026-02-26T03:02:39.224 INFO:teuthology.orchestra.run.trial197.stdout:snapset 0=[]:{} 2026-02-26T03:02:39.224 INFO:teuthology.orchestra.run.trial197.stdout:Write #2:85ec8e5d:::existing_85:head# 2026-02-26T03:02:39.224 INFO:teuthology.orchestra.run.trial197.stdout:snapset 0=[]:{} 2026-02-26T03:02:39.224 INFO:teuthology.orchestra.run.trial197.stdout:Write #2:8e2abbf0:::existing_49:head# 2026-02-26T03:02:39.224 INFO:teuthology.orchestra.run.trial197.stdout:snapset 0=[]:{} 2026-02-26T03:02:39.224 INFO:teuthology.orchestra.run.trial197.stdout:Write #2:8ee0fe0f:::existing_52:head# 2026-02-26T03:02:39.225 INFO:teuthology.orchestra.run.trial197.stdout:snapset 0=[]:{} 2026-02-26T03:02:39.225 INFO:teuthology.orchestra.run.trial197.stdout:Write #2:92731e71:::existing_89:head# 2026-02-26T03:02:39.225 INFO:teuthology.orchestra.run.trial197.stdout:snapset 0=[]:{} 2026-02-26T03:02:39.225 INFO:teuthology.orchestra.run.trial197.stdout:Write #2:928205cf:::existing_82:head# 2026-02-26T03:02:39.225 INFO:teuthology.orchestra.run.trial197.stdout:snapset 0=[]:{} 2026-02-26T03:02:39.225 INFO:teuthology.orchestra.run.trial197.stdout:Write #2:93be2267:::existing_24:head# 2026-02-26T03:02:39.226 INFO:teuthology.orchestra.run.trial197.stdout:snapset 0=[]:{} 2026-02-26T03:02:39.226 INFO:teuthology.orchestra.run.trial197.stdout:Write #2:99ceda2c:::existing_44:head# 2026-02-26T03:02:39.226 INFO:teuthology.orchestra.run.trial197.stdout:snapset 0=[]:{} 2026-02-26T03:02:39.226 INFO:teuthology.orchestra.run.trial197.stdout:Write #2:9dd917ee:::existing_26:head# 2026-02-26T03:02:39.226 INFO:teuthology.orchestra.run.trial197.stdout:snapset 0=[]:{} 2026-02-26T03:02:39.226 INFO:teuthology.orchestra.run.trial197.stdout:Write #2:a28805a6:::existing_72:head# 2026-02-26T03:02:39.226 INFO:teuthology.orchestra.run.trial197.stdout:snapset 0=[]:{} 2026-02-26T03:02:39.227 INFO:teuthology.orchestra.run.trial197.stdout:Write #2:a294a8f6:::existing_16:head# 2026-02-26T03:02:39.227 INFO:teuthology.orchestra.run.trial197.stdout:snapset 0=[]:{} 2026-02-26T03:02:39.227 INFO:teuthology.orchestra.run.trial197.stdout:Write #2:a2f5473a:::existing_42:head# 2026-02-26T03:02:39.227 INFO:teuthology.orchestra.run.trial197.stdout:snapset 0=[]:{} 2026-02-26T03:02:39.227 INFO:teuthology.orchestra.run.trial197.stdout:Write #2:a504a533:::existing_90:head# 2026-02-26T03:02:39.227 INFO:teuthology.orchestra.run.trial197.stdout:snapset 0=[]:{} 2026-02-26T03:02:39.228 INFO:teuthology.orchestra.run.trial197.stdout:Write #2:a691337e:::existing_61:head# 2026-02-26T03:02:39.228 INFO:teuthology.orchestra.run.trial197.stdout:snapset 0=[]:{} 2026-02-26T03:02:39.228 INFO:teuthology.orchestra.run.trial197.stdout:Write #2:a6e8d7ac:::existing_28:head# 2026-02-26T03:02:39.228 INFO:teuthology.orchestra.run.trial197.stdout:snapset 0=[]:{} 2026-02-26T03:02:39.228 INFO:teuthology.orchestra.run.trial197.stdout:Write #2:a9fde86e:::existing_18:head# 2026-02-26T03:02:39.228 INFO:teuthology.orchestra.run.trial197.stdout:snapset 0=[]:{} 2026-02-26T03:02:39.229 INFO:teuthology.orchestra.run.trial197.stdout:Write #2:aa309faa:::existing_68:head# 2026-02-26T03:02:39.229 INFO:teuthology.orchestra.run.trial197.stdout:snapset 0=[]:{} 2026-02-26T03:02:39.229 INFO:teuthology.orchestra.run.trial197.stdout:Write #2:ae40b0ed:::existing_31:head# 2026-02-26T03:02:39.229 INFO:teuthology.orchestra.run.trial197.stdout:snapset 0=[]:{} 2026-02-26T03:02:39.229 INFO:teuthology.orchestra.run.trial197.stdout:Write #2:b279b1fe:::existing_10:head# 2026-02-26T03:02:39.229 INFO:teuthology.orchestra.run.trial197.stdout:snapset 0=[]:{} 2026-02-26T03:02:39.230 INFO:teuthology.orchestra.run.trial197.stdout:Write #2:b343e277:::existing_88:head# 2026-02-26T03:02:39.230 INFO:teuthology.orchestra.run.trial197.stdout:snapset 0=[]:{} 2026-02-26T03:02:39.230 INFO:teuthology.orchestra.run.trial197.stdout:Write #2:b7130e90:::existing_80:head# 2026-02-26T03:02:39.230 INFO:teuthology.orchestra.run.trial197.stdout:snapset 0=[]:{} 2026-02-26T03:02:39.230 INFO:teuthology.orchestra.run.trial197.stdout:Write #2:bc6f7309:::existing_70:head# 2026-02-26T03:02:39.230 INFO:teuthology.orchestra.run.trial197.stdout:snapset 0=[]:{} 2026-02-26T03:02:39.231 INFO:teuthology.orchestra.run.trial197.stdout:Write #2:be8489b7:::existing_38:head# 2026-02-26T03:02:39.231 INFO:teuthology.orchestra.run.trial197.stdout:snapset 0=[]:{} 2026-02-26T03:02:39.231 INFO:teuthology.orchestra.run.trial197.stdout:Write #2:beb4e5e3:::existing_47:head# 2026-02-26T03:02:39.231 INFO:teuthology.orchestra.run.trial197.stdout:snapset 0=[]:{} 2026-02-26T03:02:39.231 INFO:teuthology.orchestra.run.trial197.stdout:Write #2:bebec877:::existing_74:head# 2026-02-26T03:02:39.231 INFO:teuthology.orchestra.run.trial197.stdout:snapset 0=[]:{} 2026-02-26T03:02:39.232 INFO:teuthology.orchestra.run.trial197.stdout:Write #2:c04347d8:::existing_60:head# 2026-02-26T03:02:39.232 INFO:teuthology.orchestra.run.trial197.stdout:snapset 0=[]:{} 2026-02-26T03:02:39.232 INFO:teuthology.orchestra.run.trial197.stdout:Write #2:c3e6ebe4:::existing_17:head# 2026-02-26T03:02:39.232 INFO:teuthology.orchestra.run.trial197.stdout:snapset 0=[]:{} 2026-02-26T03:02:39.232 INFO:teuthology.orchestra.run.trial197.stdout:Write #2:c6c5a7c2:::existing_39:head# 2026-02-26T03:02:39.232 INFO:teuthology.orchestra.run.trial197.stdout:snapset 0=[]:{} 2026-02-26T03:02:39.233 INFO:teuthology.orchestra.run.trial197.stdout:Write #2:cc1d0abf:::existing_95:head# 2026-02-26T03:02:39.233 INFO:teuthology.orchestra.run.trial197.stdout:snapset 0=[]:{} 2026-02-26T03:02:39.233 INFO:teuthology.orchestra.run.trial197.stdout:Write #2:ccb3865b:::existing_73:head# 2026-02-26T03:02:39.233 INFO:teuthology.orchestra.run.trial197.stdout:snapset 0=[]:{} 2026-02-26T03:02:39.233 INFO:teuthology.orchestra.run.trial197.stdout:Write #2:cd84e19e:::existing_40:head# 2026-02-26T03:02:39.233 INFO:teuthology.orchestra.run.trial197.stdout:snapset 0=[]:{} 2026-02-26T03:02:39.234 INFO:teuthology.orchestra.run.trial197.stdout:Write #2:d2cd64ce:::existing_37:head# 2026-02-26T03:02:39.234 INFO:teuthology.orchestra.run.trial197.stdout:snapset 0=[]:{} 2026-02-26T03:02:39.234 INFO:teuthology.orchestra.run.trial197.stdout:Write #2:d41a25c9:::existing_22:head# 2026-02-26T03:02:39.234 INFO:teuthology.orchestra.run.trial197.stdout:snapset 0=[]:{} 2026-02-26T03:02:39.234 INFO:teuthology.orchestra.run.trial197.stdout:Write #2:d7f088da:::existing_91:head# 2026-02-26T03:02:39.234 INFO:teuthology.orchestra.run.trial197.stdout:snapset 0=[]:{} 2026-02-26T03:02:39.235 INFO:teuthology.orchestra.run.trial197.stdout:Write #2:d9cbd1d6:::existing_64:head# 2026-02-26T03:02:39.235 INFO:teuthology.orchestra.run.trial197.stdout:snapset 0=[]:{} 2026-02-26T03:02:39.235 INFO:teuthology.orchestra.run.trial197.stdout:Write #2:dcc4ff82:::existing_98:head# 2026-02-26T03:02:39.235 INFO:teuthology.orchestra.run.trial197.stdout:snapset 0=[]:{} 2026-02-26T03:02:39.235 INFO:teuthology.orchestra.run.trial197.stdout:Write #2:dd677bd3:::existing_35:head# 2026-02-26T03:02:39.235 INFO:teuthology.orchestra.run.trial197.stdout:snapset 0=[]:{} 2026-02-26T03:02:39.235 INFO:teuthology.orchestra.run.trial197.stdout:Write #2:dd6d90e7:::existing_46:head# 2026-02-26T03:02:39.236 INFO:teuthology.orchestra.run.trial197.stdout:snapset 0=[]:{} 2026-02-26T03:02:39.236 INFO:teuthology.orchestra.run.trial197.stdout:Write #2:e0e22077:::existing_21:head# 2026-02-26T03:02:39.236 INFO:teuthology.orchestra.run.trial197.stdout:snapset 0=[]:{} 2026-02-26T03:02:39.236 INFO:teuthology.orchestra.run.trial197.stdout:Write #2:e255b710:::existing_55:head# 2026-02-26T03:02:39.236 INFO:teuthology.orchestra.run.trial197.stdout:snapset 0=[]:{} 2026-02-26T03:02:39.236 INFO:teuthology.orchestra.run.trial197.stdout:Write #2:eff1d5db:::existing_14:head# 2026-02-26T03:02:39.237 INFO:teuthology.orchestra.run.trial197.stdout:snapset 0=[]:{} 2026-02-26T03:02:39.237 INFO:teuthology.orchestra.run.trial197.stdout:Write #2:f1b156bf:::existing_23:head# 2026-02-26T03:02:39.237 INFO:teuthology.orchestra.run.trial197.stdout:snapset 0=[]:{} 2026-02-26T03:02:39.237 INFO:teuthology.orchestra.run.trial197.stdout:Write #2:f5ce4199:::existing_34:head# 2026-02-26T03:02:39.237 INFO:teuthology.orchestra.run.trial197.stdout:snapset 0=[]:{} 2026-02-26T03:02:39.237 INFO:teuthology.orchestra.run.trial197.stdout:Write #2:f6809456:::existing_11:head# 2026-02-26T03:02:39.238 INFO:teuthology.orchestra.run.trial197.stdout:snapset 0=[]:{} 2026-02-26T03:02:39.238 INFO:teuthology.orchestra.run.trial197.stdout:Write #2:f699a2a9:::existing_67:head# 2026-02-26T03:02:39.238 INFO:teuthology.orchestra.run.trial197.stdout:snapset 0=[]:{} 2026-02-26T03:02:39.238 INFO:teuthology.orchestra.run.trial197.stdout:Write #2:f9baac43:::existing_25:head# 2026-02-26T03:02:39.238 INFO:teuthology.orchestra.run.trial197.stdout:snapset 0=[]:{} 2026-02-26T03:02:39.238 INFO:teuthology.orchestra.run.trial197.stdout:Write #2:fb76688e:::existing_51:head# 2026-02-26T03:02:39.239 INFO:teuthology.orchestra.run.trial197.stdout:snapset 0=[]:{} 2026-02-26T03:02:39.239 INFO:teuthology.orchestra.run.trial197.stdout:Write #2:fbb92eec:::existing_69:head# 2026-02-26T03:02:39.239 INFO:teuthology.orchestra.run.trial197.stdout:snapset 0=[]:{} 2026-02-26T03:02:39.239 INFO:teuthology.orchestra.run.trial197.stdout:write_pg epoch 38 info 2.0( v 38'102 lc 0'0 (20'95,38'102] local-lis/les=36/37 n=101 ec=18/18 lis/c=36/18 les/c/f=37/19/0 sis=36) 2026-02-26T03:02:39.240 INFO:teuthology.orchestra.run.trial197.stdout:Import successful 2026-02-26T03:02:39.757 INFO:tasks.divergent_priors2:reviving divergent 2 2026-02-26T03:02:39.757 INFO:tasks.ceph.osd.2:Restarting daemon 2026-02-26T03:02:39.757 DEBUG:teuthology.orchestra.run.trial197:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 2 2026-02-26T03:02:39.758 INFO:tasks.ceph.osd.2:Started 2026-02-26T03:02:39.759 DEBUG:teuthology.orchestra.run.trial197:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2026-02-26T03:02:39.825 INFO:teuthology.orchestra.run.trial197.stderr:no valid command found; 10 closest matches: 2026-02-26T03:02:39.826 INFO:teuthology.orchestra.run.trial197.stderr:0 2026-02-26T03:02:39.826 INFO:teuthology.orchestra.run.trial197.stderr:1 2026-02-26T03:02:39.826 INFO:teuthology.orchestra.run.trial197.stderr:2 2026-02-26T03:02:39.826 INFO:teuthology.orchestra.run.trial197.stderr:abort 2026-02-26T03:02:39.826 INFO:teuthology.orchestra.run.trial197.stderr:assert 2026-02-26T03:02:39.826 INFO:teuthology.orchestra.run.trial197.stderr:config diff 2026-02-26T03:02:39.826 INFO:teuthology.orchestra.run.trial197.stderr:config diff get 2026-02-26T03:02:39.826 INFO:teuthology.orchestra.run.trial197.stderr:config get 2026-02-26T03:02:39.826 INFO:teuthology.orchestra.run.trial197.stderr:config help [] 2026-02-26T03:02:39.827 INFO:teuthology.orchestra.run.trial197.stderr:config set ... 2026-02-26T03:02:39.827 INFO:teuthology.orchestra.run.trial197.stderr:admin_socket: invalid command 2026-02-26T03:02:39.827 DEBUG:teuthology.orchestra.run:got remote process result: 22 2026-02-26T03:02:39.828 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-2, ['dump_ops_in_flight'] 2026-02-26T03:02:40.908 INFO:tasks.ceph.osd.2.trial197.stderr:2026-02-26T03:02:40.906+0000 7ff903015780 -1 Falling back to public interface 2026-02-26T03:02:44.660 INFO:tasks.ceph.osd.2.trial197.stderr:2026-02-26T03:02:44.658+0000 7ff903015780 -1 osd.2 38 log_to_monitors true 2026-02-26T03:02:44.830 DEBUG:teuthology.orchestra.run.trial197:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2026-02-26T03:02:44.898 INFO:teuthology.orchestra.run.trial197.stderr:no valid command found; 10 closest matches: 2026-02-26T03:02:44.899 INFO:teuthology.orchestra.run.trial197.stderr:0 2026-02-26T03:02:44.899 INFO:teuthology.orchestra.run.trial197.stderr:1 2026-02-26T03:02:44.899 INFO:teuthology.orchestra.run.trial197.stderr:2 2026-02-26T03:02:44.899 INFO:teuthology.orchestra.run.trial197.stderr:abort 2026-02-26T03:02:44.899 INFO:teuthology.orchestra.run.trial197.stderr:assert 2026-02-26T03:02:44.899 INFO:teuthology.orchestra.run.trial197.stderr:bluefs debug_inject_read_zeros 2026-02-26T03:02:44.899 INFO:teuthology.orchestra.run.trial197.stderr:bluefs files list 2026-02-26T03:02:44.899 INFO:teuthology.orchestra.run.trial197.stderr:bluefs stats 2026-02-26T03:02:44.899 INFO:teuthology.orchestra.run.trial197.stderr:bluestore allocator dump block 2026-02-26T03:02:44.899 INFO:teuthology.orchestra.run.trial197.stderr:bluestore allocator fragmentation block 2026-02-26T03:02:44.900 INFO:teuthology.orchestra.run.trial197.stderr:admin_socket: invalid command 2026-02-26T03:02:44.900 DEBUG:teuthology.orchestra.run:got remote process result: 22 2026-02-26T03:02:44.901 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-2, ['dump_ops_in_flight'] 2026-02-26T03:02:45.235 INFO:tasks.ceph.osd.2.trial197.stderr:2026-02-26T03:02:45.233+0000 7ff8fa6b3640 -1 osd.2 38 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2026-02-26T03:02:49.902 DEBUG:teuthology.orchestra.run.trial197:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2026-02-26T03:02:49.973 INFO:teuthology.orchestra.run.trial197.stdout:{ 2026-02-26T03:02:49.973 INFO:teuthology.orchestra.run.trial197.stdout: "ops": [], 2026-02-26T03:02:49.973 INFO:teuthology.orchestra.run.trial197.stdout: "num_ops": 0 2026-02-26T03:02:49.973 INFO:teuthology.orchestra.run.trial197.stdout:} 2026-02-26T03:02:49.978 DEBUG:teuthology.orchestra.run.trial197:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2026-02-26T03:02:50.047 INFO:teuthology.orchestra.run.trial197.stdout:{ 2026-02-26T03:02:50.048 INFO:teuthology.orchestra.run.trial197.stdout: "ops": [], 2026-02-26T03:02:50.048 INFO:teuthology.orchestra.run.trial197.stdout: "num_ops": 0 2026-02-26T03:02:50.048 INFO:teuthology.orchestra.run.trial197.stdout:} 2026-02-26T03:03:10.052 INFO:tasks.divergent_priors2:allowing recovery 2026-02-26T03:03:10.053 DEBUG:teuthology.orchestra.run.trial197:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 debug kick_recovery_wq ' 0' 2026-02-26T03:03:10.133 INFO:teuthology.orchestra.run.trial197.stderr:kicking recovery queue. set osd_recovery_delay_start to 0 2026-02-26T03:03:10.138 DEBUG:teuthology.orchestra.run.trial197:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 debug kick_recovery_wq ' 0' 2026-02-26T03:03:10.216 INFO:teuthology.orchestra.run.trial197.stderr:kicking recovery queue. set osd_recovery_delay_start to 0 2026-02-26T03:03:10.222 DEBUG:teuthology.orchestra.run.trial197:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 debug kick_recovery_wq ' 0' 2026-02-26T03:03:10.300 INFO:teuthology.orchestra.run.trial197.stderr:kicking recovery queue. set osd_recovery_delay_start to 0 2026-02-26T03:03:10.305 INFO:tasks.divergent_priors2:reading divergent objects 2026-02-26T03:03:10.305 INFO:tasks.util.rados:rados -p foo get existing_0 /tmp/existing 2026-02-26T03:03:10.306 DEBUG:teuthology.orchestra.run.trial197:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p foo get existing_0 /tmp/existing 2026-02-26T03:03:10.344 INFO:tasks.util.rados:rados -p foo get existing_1 /tmp/existing 2026-02-26T03:03:10.344 DEBUG:teuthology.orchestra.run.trial197:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p foo get existing_1 /tmp/existing 2026-02-26T03:03:10.422 INFO:tasks.util.rados:rados -p foo get existing_2 /tmp/existing 2026-02-26T03:03:10.422 DEBUG:teuthology.orchestra.run.trial197:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p foo get existing_2 /tmp/existing 2026-02-26T03:03:10.500 INFO:tasks.util.rados:rados -p foo get existing_3 /tmp/existing 2026-02-26T03:03:10.500 DEBUG:teuthology.orchestra.run.trial197:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p foo get existing_3 /tmp/existing 2026-02-26T03:03:10.578 INFO:tasks.util.rados:rados -p foo get existing_4 /tmp/existing 2026-02-26T03:03:10.578 DEBUG:teuthology.orchestra.run.trial197:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p foo get existing_4 /tmp/existing 2026-02-26T03:03:10.655 INFO:tasks.util.rados:rados -p foo get existing_5 /tmp/existing 2026-02-26T03:03:10.656 DEBUG:teuthology.orchestra.run.trial197:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p foo get existing_5 /tmp/existing 2026-02-26T03:03:10.732 INFO:tasks.util.rados:rados -p foo get existing_6 /tmp/existing 2026-02-26T03:03:10.732 DEBUG:teuthology.orchestra.run.trial197:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p foo get existing_6 /tmp/existing 2026-02-26T03:03:10.808 INFO:tasks.util.rados:rados -p foo get existing_7 /tmp/existing 2026-02-26T03:03:10.809 DEBUG:teuthology.orchestra.run.trial197:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p foo get existing_7 /tmp/existing 2026-02-26T03:03:10.885 INFO:tasks.util.rados:rados -p foo get existing_8 /tmp/existing 2026-02-26T03:03:10.885 DEBUG:teuthology.orchestra.run.trial197:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p foo get existing_8 /tmp/existing 2026-02-26T03:03:10.962 INFO:tasks.util.rados:rados -p foo get existing_9 /tmp/existing 2026-02-26T03:03:10.962 DEBUG:teuthology.orchestra.run.trial197:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p foo get existing_9 /tmp/existing 2026-02-26T03:03:11.040 DEBUG:teuthology.orchestra.run.trial197:> rm /home/ubuntu/cephtest/exp.771639.out 2026-02-26T03:03:11.095 INFO:tasks.divergent_priors2:success 2026-02-26T03:03:11.096 DEBUG:teuthology.run_tasks:Unwinding manager ceph 2026-02-26T03:03:11.103 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2026-02-26T03:03:11.103 DEBUG:teuthology.orchestra.run.trial197:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-26T03:03:11.289 INFO:teuthology.orchestra.run.trial197.stdout: 2026-02-26T03:03:11.290 INFO:teuthology.orchestra.run.trial197.stderr:dumped all 2026-02-26T03:03:11.298 INFO:teuthology.orchestra.run.trial197.stdout:{"pg_ready":true,"pg_map":{"version":99,"stamp":"2026-02-26T03:03:10.391681+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":65229,"num_objects":103,"num_object_clones":0,"num_object_copies":206,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":103,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":104,"num_write_kb":104,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":6620,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":10,"ondisk_log_size":10,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":93816,"kb_used_data":1848,"kb_used_omap":20,"kb_used_meta":91883,"kb_avail":283021704,"statfs":{"total":289910292480,"available":289814224896,"internally_reserved":0,"allocated":1892352,"data_stored":505641,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":21021,"internal_metadata":94088675},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.000944"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":60,"reported_epoch":42,"state":"active+clean","last_fresh":"2026-02-26T03:02:48.232684+0000","last_change":"2026-02-26T03:02:48.232684+0000","last_active":"2026-02-26T03:02:48.232684+0000","last_peered":"2026-02-26T03:02:48.232684+0000","last_clean":"2026-02-26T03:02:48.232684+0000","last_became_active":"2026-02-26T03:02:48.232368+0000","last_became_peered":"2026-02-26T03:02:48.232368+0000","last_unstale":"2026-02-26T03:02:48.232684+0000","last_undegraded":"2026-02-26T03:02:48.232684+0000","last_fullsized":"2026-02-26T03:02:48.232684+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-26T03:01:18.659251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-26T03:01:18.659251+0000","last_clean_scrub_stamp":"2026-02-26T03:01:18.659251+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-27T11:57:01.371827+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00060011299999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":52,"reported_epoch":42,"state":"active+clean","last_fresh":"2026-02-26T03:02:48.230981+0000","last_change":"2026-02-26T03:02:16.405639+0000","last_active":"2026-02-26T03:02:48.230981+0000","last_peered":"2026-02-26T03:02:48.230981+0000","last_clean":"2026-02-26T03:02:48.230981+0000","last_became_active":"2026-02-26T03:02:16.405505+0000","last_became_peered":"2026-02-26T03:02:16.405505+0000","last_unstale":"2026-02-26T03:02:48.230981+0000","last_undegraded":"2026-02-26T03:02:48.230981+0000","last_fullsized":"2026-02-26T03:02:48.230981+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-26T03:01:18.659251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-26T03:01:18.659251+0000","last_clean_scrub_stamp":"2026-02-26T03:01:18.659251+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-27T14:28:28.066117+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00068042299999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":45,"reported_epoch":42,"state":"active+clean","last_fresh":"2026-02-26T03:02:48.762716+0000","last_change":"2026-02-26T03:02:48.762716+0000","last_active":"2026-02-26T03:02:48.762716+0000","last_peered":"2026-02-26T03:02:48.762716+0000","last_clean":"2026-02-26T03:02:48.762716+0000","last_became_active":"2026-02-26T03:02:48.762400+0000","last_became_peered":"2026-02-26T03:02:48.762400+0000","last_unstale":"2026-02-26T03:02:48.762716+0000","last_undegraded":"2026-02-26T03:02:48.762716+0000","last_fullsized":"2026-02-26T03:02:48.762716+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-26T03:01:18.659251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-26T03:01:18.659251+0000","last_clean_scrub_stamp":"2026-02-26T03:01:18.659251+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-27T05:49:10.168518+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":52,"reported_epoch":42,"state":"active+clean","last_fresh":"2026-02-26T03:02:48.230975+0000","last_change":"2026-02-26T03:02:16.406146+0000","last_active":"2026-02-26T03:02:48.230975+0000","last_peered":"2026-02-26T03:02:48.230975+0000","last_clean":"2026-02-26T03:02:48.230975+0000","last_became_active":"2026-02-26T03:02:16.405921+0000","last_became_peered":"2026-02-26T03:02:16.405921+0000","last_unstale":"2026-02-26T03:02:48.230975+0000","last_undegraded":"2026-02-26T03:02:48.230975+0000","last_fullsized":"2026-02-26T03:02:48.230975+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-26T03:01:18.659251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-26T03:01:18.659251+0000","last_clean_scrub_stamp":"2026-02-26T03:01:18.659251+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-27T08:15:04.934373+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000482733,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"38'102","reported_seq":170,"reported_epoch":42,"state":"active+clean","last_fresh":"2026-02-26T03:02:48.241816+0000","last_change":"2026-02-26T03:02:48.241816+0000","last_active":"2026-02-26T03:02:48.241816+0000","last_peered":"2026-02-26T03:02:48.241816+0000","last_clean":"2026-02-26T03:02:48.241816+0000","last_became_active":"2026-02-26T03:02:48.232430+0000","last_became_peered":"2026-02-26T03:02:48.232430+0000","last_unstale":"2026-02-26T03:02:48.241816+0000","last_undegraded":"2026-02-26T03:02:48.241816+0000","last_fullsized":"2026-02-26T03:02:48.241816+0000","mapping_epoch":41,"log_start":"20'95","ondisk_log_start":"20'95","created":18,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-26T03:01:32.157530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-26T03:01:32.157530+0000","last_clean_scrub_stamp":"2026-02-26T03:01:32.157530+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":95,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-27T07:23:32.702444+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":65210,"num_objects":101,"num_object_clones":0,"num_object_copies":202,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":101,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":102,"num_write_kb":102,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":6620,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"11'1","reported_seq":65,"reported_epoch":42,"state":"active+clean","last_fresh":"2026-02-26T03:02:48.232628+0000","last_change":"2026-02-26T03:02:48.232628+0000","last_active":"2026-02-26T03:02:48.232628+0000","last_peered":"2026-02-26T03:02:48.232628+0000","last_clean":"2026-02-26T03:02:48.232628+0000","last_became_active":"2026-02-26T03:02:48.232286+0000","last_became_peered":"2026-02-26T03:02:48.232286+0000","last_unstale":"2026-02-26T03:02:48.232628+0000","last_undegraded":"2026-02-26T03:02:48.232628+0000","last_fullsized":"2026-02-26T03:02:48.232628+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-26T03:01:18.659251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-26T03:01:18.659251+0000","last_clean_scrub_stamp":"2026-02-26T03:01:18.659251+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-27T05:55:53.987752+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00034461499999999999,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"13'2","reported_seq":56,"reported_epoch":42,"state":"active+clean","last_fresh":"2026-02-26T03:02:48.761742+0000","last_change":"2026-02-26T03:02:15.149310+0000","last_active":"2026-02-26T03:02:48.761742+0000","last_peered":"2026-02-26T03:02:48.761742+0000","last_clean":"2026-02-26T03:02:48.761742+0000","last_became_active":"2026-02-26T03:02:15.149144+0000","last_became_peered":"2026-02-26T03:02:15.149144+0000","last_unstale":"2026-02-26T03:02:48.761742+0000","last_undegraded":"2026-02-26T03:02:48.761742+0000","last_fullsized":"2026-02-26T03:02:48.761742+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-26T03:01:18.659251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-26T03:01:18.659251+0000","last_clean_scrub_stamp":"2026-02-26T03:01:18.659251+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-27T11:49:30.505493+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00054467999999999997,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":45,"reported_epoch":42,"state":"active+clean","last_fresh":"2026-02-26T03:02:48.762548+0000","last_change":"2026-02-26T03:02:48.762548+0000","last_active":"2026-02-26T03:02:48.762548+0000","last_peered":"2026-02-26T03:02:48.762548+0000","last_clean":"2026-02-26T03:02:48.762548+0000","last_became_active":"2026-02-26T03:02:48.762315+0000","last_became_peered":"2026-02-26T03:02:48.762315+0000","last_unstale":"2026-02-26T03:02:48.762548+0000","last_undegraded":"2026-02-26T03:02:48.762548+0000","last_fullsized":"2026-02-26T03:02:48.762548+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-26T03:01:18.659251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-26T03:01:18.659251+0000","last_clean_scrub_stamp":"2026-02-26T03:01:18.659251+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-27T10:46:34.526556+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":52,"reported_epoch":42,"state":"active+clean","last_fresh":"2026-02-26T03:02:48.230874+0000","last_change":"2026-02-26T03:02:16.406053+0000","last_active":"2026-02-26T03:02:48.230874+0000","last_peered":"2026-02-26T03:02:48.230874+0000","last_clean":"2026-02-26T03:02:48.230874+0000","last_became_active":"2026-02-26T03:02:16.405837+0000","last_became_peered":"2026-02-26T03:02:16.405837+0000","last_unstale":"2026-02-26T03:02:48.230874+0000","last_undegraded":"2026-02-26T03:02:48.230874+0000","last_fullsized":"2026-02-26T03:02:48.230874+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-26T03:01:18.659251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-26T03:01:18.659251+0000","last_clean_scrub_stamp":"2026-02-26T03:01:18.659251+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-27T14:23:01.074046+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00037062400000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":65210,"num_objects":101,"num_object_clones":0,"num_object_copies":202,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":101,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":102,"num_write_kb":102,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":6620,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":827392,"data_stored":130420,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6077,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":2,"acting":2,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":1,"up_from":29,"seq":124554051597,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32324,"kb_used_data":752,"kb_used_omap":4,"kb_used_meta":31547,"kb_avail":94339516,"statfs":{"total":96636764160,"available":96603664384,"internally_reserved":0,"allocated":770048,"data_stored":190290,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4434,"internal_metadata":32304814},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":41,"seq":176093659142,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":29504,"kb_used_data":748,"kb_used_omap":11,"kb_used_meta":28724,"kb_avail":94342336,"statfs":{"total":96636764160,"available":96606552064,"internally_reserved":0,"allocated":765952,"data_stored":190271,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12155,"internal_metadata":29413509},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":27,"seq":115964117007,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31988,"kb_used_data":348,"kb_used_omap":4,"kb_used_meta":31611,"kb_avail":94339852,"statfs":{"total":96636764160,"available":96604008448,"internally_reserved":0,"allocated":356352,"data_stored":125080,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4432,"internal_metadata":32370352},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":413696,"data_stored":65210,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":413696,"data_stored":65210,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6077,"internal_metadata":0}]}} 2026-02-26T03:03:11.300 DEBUG:teuthology.orchestra.run.trial197:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-26T03:03:11.454 INFO:teuthology.orchestra.run.trial197.stdout: 2026-02-26T03:03:11.454 INFO:teuthology.orchestra.run.trial197.stderr:dumped all 2026-02-26T03:03:11.463 INFO:teuthology.orchestra.run.trial197.stdout:{"pg_ready":true,"pg_map":{"version":99,"stamp":"2026-02-26T03:03:10.391681+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":65229,"num_objects":103,"num_object_clones":0,"num_object_copies":206,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":103,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":104,"num_write_kb":104,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":6620,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":10,"ondisk_log_size":10,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":93816,"kb_used_data":1848,"kb_used_omap":20,"kb_used_meta":91883,"kb_avail":283021704,"statfs":{"total":289910292480,"available":289814224896,"internally_reserved":0,"allocated":1892352,"data_stored":505641,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":21021,"internal_metadata":94088675},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.000944"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":60,"reported_epoch":42,"state":"active+clean","last_fresh":"2026-02-26T03:02:48.232684+0000","last_change":"2026-02-26T03:02:48.232684+0000","last_active":"2026-02-26T03:02:48.232684+0000","last_peered":"2026-02-26T03:02:48.232684+0000","last_clean":"2026-02-26T03:02:48.232684+0000","last_became_active":"2026-02-26T03:02:48.232368+0000","last_became_peered":"2026-02-26T03:02:48.232368+0000","last_unstale":"2026-02-26T03:02:48.232684+0000","last_undegraded":"2026-02-26T03:02:48.232684+0000","last_fullsized":"2026-02-26T03:02:48.232684+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-26T03:01:18.659251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-26T03:01:18.659251+0000","last_clean_scrub_stamp":"2026-02-26T03:01:18.659251+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-27T11:57:01.371827+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00060011299999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":52,"reported_epoch":42,"state":"active+clean","last_fresh":"2026-02-26T03:02:48.230981+0000","last_change":"2026-02-26T03:02:16.405639+0000","last_active":"2026-02-26T03:02:48.230981+0000","last_peered":"2026-02-26T03:02:48.230981+0000","last_clean":"2026-02-26T03:02:48.230981+0000","last_became_active":"2026-02-26T03:02:16.405505+0000","last_became_peered":"2026-02-26T03:02:16.405505+0000","last_unstale":"2026-02-26T03:02:48.230981+0000","last_undegraded":"2026-02-26T03:02:48.230981+0000","last_fullsized":"2026-02-26T03:02:48.230981+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-26T03:01:18.659251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-26T03:01:18.659251+0000","last_clean_scrub_stamp":"2026-02-26T03:01:18.659251+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-27T14:28:28.066117+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00068042299999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":45,"reported_epoch":42,"state":"active+clean","last_fresh":"2026-02-26T03:02:48.762716+0000","last_change":"2026-02-26T03:02:48.762716+0000","last_active":"2026-02-26T03:02:48.762716+0000","last_peered":"2026-02-26T03:02:48.762716+0000","last_clean":"2026-02-26T03:02:48.762716+0000","last_became_active":"2026-02-26T03:02:48.762400+0000","last_became_peered":"2026-02-26T03:02:48.762400+0000","last_unstale":"2026-02-26T03:02:48.762716+0000","last_undegraded":"2026-02-26T03:02:48.762716+0000","last_fullsized":"2026-02-26T03:02:48.762716+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-26T03:01:18.659251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-26T03:01:18.659251+0000","last_clean_scrub_stamp":"2026-02-26T03:01:18.659251+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-27T05:49:10.168518+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":52,"reported_epoch":42,"state":"active+clean","last_fresh":"2026-02-26T03:02:48.230975+0000","last_change":"2026-02-26T03:02:16.406146+0000","last_active":"2026-02-26T03:02:48.230975+0000","last_peered":"2026-02-26T03:02:48.230975+0000","last_clean":"2026-02-26T03:02:48.230975+0000","last_became_active":"2026-02-26T03:02:16.405921+0000","last_became_peered":"2026-02-26T03:02:16.405921+0000","last_unstale":"2026-02-26T03:02:48.230975+0000","last_undegraded":"2026-02-26T03:02:48.230975+0000","last_fullsized":"2026-02-26T03:02:48.230975+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-26T03:01:18.659251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-26T03:01:18.659251+0000","last_clean_scrub_stamp":"2026-02-26T03:01:18.659251+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-27T08:15:04.934373+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000482733,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"38'102","reported_seq":170,"reported_epoch":42,"state":"active+clean","last_fresh":"2026-02-26T03:02:48.241816+0000","last_change":"2026-02-26T03:02:48.241816+0000","last_active":"2026-02-26T03:02:48.241816+0000","last_peered":"2026-02-26T03:02:48.241816+0000","last_clean":"2026-02-26T03:02:48.241816+0000","last_became_active":"2026-02-26T03:02:48.232430+0000","last_became_peered":"2026-02-26T03:02:48.232430+0000","last_unstale":"2026-02-26T03:02:48.241816+0000","last_undegraded":"2026-02-26T03:02:48.241816+0000","last_fullsized":"2026-02-26T03:02:48.241816+0000","mapping_epoch":41,"log_start":"20'95","ondisk_log_start":"20'95","created":18,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-26T03:01:32.157530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-26T03:01:32.157530+0000","last_clean_scrub_stamp":"2026-02-26T03:01:32.157530+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":95,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-27T07:23:32.702444+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":65210,"num_objects":101,"num_object_clones":0,"num_object_copies":202,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":101,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":102,"num_write_kb":102,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":6620,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"11'1","reported_seq":65,"reported_epoch":42,"state":"active+clean","last_fresh":"2026-02-26T03:02:48.232628+0000","last_change":"2026-02-26T03:02:48.232628+0000","last_active":"2026-02-26T03:02:48.232628+0000","last_peered":"2026-02-26T03:02:48.232628+0000","last_clean":"2026-02-26T03:02:48.232628+0000","last_became_active":"2026-02-26T03:02:48.232286+0000","last_became_peered":"2026-02-26T03:02:48.232286+0000","last_unstale":"2026-02-26T03:02:48.232628+0000","last_undegraded":"2026-02-26T03:02:48.232628+0000","last_fullsized":"2026-02-26T03:02:48.232628+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-26T03:01:18.659251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-26T03:01:18.659251+0000","last_clean_scrub_stamp":"2026-02-26T03:01:18.659251+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-27T05:55:53.987752+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00034461499999999999,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"13'2","reported_seq":56,"reported_epoch":42,"state":"active+clean","last_fresh":"2026-02-26T03:02:48.761742+0000","last_change":"2026-02-26T03:02:15.149310+0000","last_active":"2026-02-26T03:02:48.761742+0000","last_peered":"2026-02-26T03:02:48.761742+0000","last_clean":"2026-02-26T03:02:48.761742+0000","last_became_active":"2026-02-26T03:02:15.149144+0000","last_became_peered":"2026-02-26T03:02:15.149144+0000","last_unstale":"2026-02-26T03:02:48.761742+0000","last_undegraded":"2026-02-26T03:02:48.761742+0000","last_fullsized":"2026-02-26T03:02:48.761742+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-26T03:01:18.659251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-26T03:01:18.659251+0000","last_clean_scrub_stamp":"2026-02-26T03:01:18.659251+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-27T11:49:30.505493+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00054467999999999997,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":45,"reported_epoch":42,"state":"active+clean","last_fresh":"2026-02-26T03:02:48.762548+0000","last_change":"2026-02-26T03:02:48.762548+0000","last_active":"2026-02-26T03:02:48.762548+0000","last_peered":"2026-02-26T03:02:48.762548+0000","last_clean":"2026-02-26T03:02:48.762548+0000","last_became_active":"2026-02-26T03:02:48.762315+0000","last_became_peered":"2026-02-26T03:02:48.762315+0000","last_unstale":"2026-02-26T03:02:48.762548+0000","last_undegraded":"2026-02-26T03:02:48.762548+0000","last_fullsized":"2026-02-26T03:02:48.762548+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-26T03:01:18.659251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-26T03:01:18.659251+0000","last_clean_scrub_stamp":"2026-02-26T03:01:18.659251+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-27T10:46:34.526556+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":52,"reported_epoch":42,"state":"active+clean","last_fresh":"2026-02-26T03:02:48.230874+0000","last_change":"2026-02-26T03:02:16.406053+0000","last_active":"2026-02-26T03:02:48.230874+0000","last_peered":"2026-02-26T03:02:48.230874+0000","last_clean":"2026-02-26T03:02:48.230874+0000","last_became_active":"2026-02-26T03:02:16.405837+0000","last_became_peered":"2026-02-26T03:02:16.405837+0000","last_unstale":"2026-02-26T03:02:48.230874+0000","last_undegraded":"2026-02-26T03:02:48.230874+0000","last_fullsized":"2026-02-26T03:02:48.230874+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-26T03:01:18.659251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-26T03:01:18.659251+0000","last_clean_scrub_stamp":"2026-02-26T03:01:18.659251+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-27T14:23:01.074046+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00037062400000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":65210,"num_objects":101,"num_object_clones":0,"num_object_copies":202,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":101,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":102,"num_write_kb":102,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":6620,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":827392,"data_stored":130420,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6077,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":2,"acting":2,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":1,"up_from":29,"seq":124554051597,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32324,"kb_used_data":752,"kb_used_omap":4,"kb_used_meta":31547,"kb_avail":94339516,"statfs":{"total":96636764160,"available":96603664384,"internally_reserved":0,"allocated":770048,"data_stored":190290,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4434,"internal_metadata":32304814},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":41,"seq":176093659142,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":29504,"kb_used_data":748,"kb_used_omap":11,"kb_used_meta":28724,"kb_avail":94342336,"statfs":{"total":96636764160,"available":96606552064,"internally_reserved":0,"allocated":765952,"data_stored":190271,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12155,"internal_metadata":29413509},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":27,"seq":115964117007,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31988,"kb_used_data":348,"kb_used_omap":4,"kb_used_meta":31611,"kb_avail":94339852,"statfs":{"total":96636764160,"available":96604008448,"internally_reserved":0,"allocated":356352,"data_stored":125080,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4432,"internal_metadata":32370352},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":413696,"data_stored":65210,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":413696,"data_stored":65210,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6077,"internal_metadata":0}]}} 2026-02-26T03:03:11.464 INFO:tasks.ceph.ceph_manager.ceph:clean! 2026-02-26T03:03:11.464 DEBUG:teuthology.orchestra.run.trial197:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-26T03:03:11.619 INFO:teuthology.orchestra.run.trial197.stdout: 2026-02-26T03:03:11.619 INFO:teuthology.orchestra.run.trial197.stderr:dumped all 2026-02-26T03:03:11.626 INFO:teuthology.orchestra.run.trial197.stdout:{"pg_ready":true,"pg_map":{"version":99,"stamp":"2026-02-26T03:03:10.391681+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":65229,"num_objects":103,"num_object_clones":0,"num_object_copies":206,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":103,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":104,"num_write_kb":104,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":6620,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":10,"ondisk_log_size":10,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":93816,"kb_used_data":1848,"kb_used_omap":20,"kb_used_meta":91883,"kb_avail":283021704,"statfs":{"total":289910292480,"available":289814224896,"internally_reserved":0,"allocated":1892352,"data_stored":505641,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":21021,"internal_metadata":94088675},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.000944"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":60,"reported_epoch":42,"state":"active+clean","last_fresh":"2026-02-26T03:02:48.232684+0000","last_change":"2026-02-26T03:02:48.232684+0000","last_active":"2026-02-26T03:02:48.232684+0000","last_peered":"2026-02-26T03:02:48.232684+0000","last_clean":"2026-02-26T03:02:48.232684+0000","last_became_active":"2026-02-26T03:02:48.232368+0000","last_became_peered":"2026-02-26T03:02:48.232368+0000","last_unstale":"2026-02-26T03:02:48.232684+0000","last_undegraded":"2026-02-26T03:02:48.232684+0000","last_fullsized":"2026-02-26T03:02:48.232684+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-26T03:01:18.659251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-26T03:01:18.659251+0000","last_clean_scrub_stamp":"2026-02-26T03:01:18.659251+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-27T11:57:01.371827+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00060011299999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":52,"reported_epoch":42,"state":"active+clean","last_fresh":"2026-02-26T03:02:48.230981+0000","last_change":"2026-02-26T03:02:16.405639+0000","last_active":"2026-02-26T03:02:48.230981+0000","last_peered":"2026-02-26T03:02:48.230981+0000","last_clean":"2026-02-26T03:02:48.230981+0000","last_became_active":"2026-02-26T03:02:16.405505+0000","last_became_peered":"2026-02-26T03:02:16.405505+0000","last_unstale":"2026-02-26T03:02:48.230981+0000","last_undegraded":"2026-02-26T03:02:48.230981+0000","last_fullsized":"2026-02-26T03:02:48.230981+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-26T03:01:18.659251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-26T03:01:18.659251+0000","last_clean_scrub_stamp":"2026-02-26T03:01:18.659251+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-27T14:28:28.066117+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00068042299999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":45,"reported_epoch":42,"state":"active+clean","last_fresh":"2026-02-26T03:02:48.762716+0000","last_change":"2026-02-26T03:02:48.762716+0000","last_active":"2026-02-26T03:02:48.762716+0000","last_peered":"2026-02-26T03:02:48.762716+0000","last_clean":"2026-02-26T03:02:48.762716+0000","last_became_active":"2026-02-26T03:02:48.762400+0000","last_became_peered":"2026-02-26T03:02:48.762400+0000","last_unstale":"2026-02-26T03:02:48.762716+0000","last_undegraded":"2026-02-26T03:02:48.762716+0000","last_fullsized":"2026-02-26T03:02:48.762716+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-26T03:01:18.659251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-26T03:01:18.659251+0000","last_clean_scrub_stamp":"2026-02-26T03:01:18.659251+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-27T05:49:10.168518+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":52,"reported_epoch":42,"state":"active+clean","last_fresh":"2026-02-26T03:02:48.230975+0000","last_change":"2026-02-26T03:02:16.406146+0000","last_active":"2026-02-26T03:02:48.230975+0000","last_peered":"2026-02-26T03:02:48.230975+0000","last_clean":"2026-02-26T03:02:48.230975+0000","last_became_active":"2026-02-26T03:02:16.405921+0000","last_became_peered":"2026-02-26T03:02:16.405921+0000","last_unstale":"2026-02-26T03:02:48.230975+0000","last_undegraded":"2026-02-26T03:02:48.230975+0000","last_fullsized":"2026-02-26T03:02:48.230975+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-26T03:01:18.659251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-26T03:01:18.659251+0000","last_clean_scrub_stamp":"2026-02-26T03:01:18.659251+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-27T08:15:04.934373+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000482733,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"38'102","reported_seq":170,"reported_epoch":42,"state":"active+clean","last_fresh":"2026-02-26T03:02:48.241816+0000","last_change":"2026-02-26T03:02:48.241816+0000","last_active":"2026-02-26T03:02:48.241816+0000","last_peered":"2026-02-26T03:02:48.241816+0000","last_clean":"2026-02-26T03:02:48.241816+0000","last_became_active":"2026-02-26T03:02:48.232430+0000","last_became_peered":"2026-02-26T03:02:48.232430+0000","last_unstale":"2026-02-26T03:02:48.241816+0000","last_undegraded":"2026-02-26T03:02:48.241816+0000","last_fullsized":"2026-02-26T03:02:48.241816+0000","mapping_epoch":41,"log_start":"20'95","ondisk_log_start":"20'95","created":18,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-26T03:01:32.157530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-26T03:01:32.157530+0000","last_clean_scrub_stamp":"2026-02-26T03:01:32.157530+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":95,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-27T07:23:32.702444+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":65210,"num_objects":101,"num_object_clones":0,"num_object_copies":202,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":101,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":102,"num_write_kb":102,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":6620,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"11'1","reported_seq":65,"reported_epoch":42,"state":"active+clean","last_fresh":"2026-02-26T03:02:48.232628+0000","last_change":"2026-02-26T03:02:48.232628+0000","last_active":"2026-02-26T03:02:48.232628+0000","last_peered":"2026-02-26T03:02:48.232628+0000","last_clean":"2026-02-26T03:02:48.232628+0000","last_became_active":"2026-02-26T03:02:48.232286+0000","last_became_peered":"2026-02-26T03:02:48.232286+0000","last_unstale":"2026-02-26T03:02:48.232628+0000","last_undegraded":"2026-02-26T03:02:48.232628+0000","last_fullsized":"2026-02-26T03:02:48.232628+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-26T03:01:18.659251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-26T03:01:18.659251+0000","last_clean_scrub_stamp":"2026-02-26T03:01:18.659251+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-27T05:55:53.987752+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00034461499999999999,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"13'2","reported_seq":56,"reported_epoch":42,"state":"active+clean","last_fresh":"2026-02-26T03:02:48.761742+0000","last_change":"2026-02-26T03:02:15.149310+0000","last_active":"2026-02-26T03:02:48.761742+0000","last_peered":"2026-02-26T03:02:48.761742+0000","last_clean":"2026-02-26T03:02:48.761742+0000","last_became_active":"2026-02-26T03:02:15.149144+0000","last_became_peered":"2026-02-26T03:02:15.149144+0000","last_unstale":"2026-02-26T03:02:48.761742+0000","last_undegraded":"2026-02-26T03:02:48.761742+0000","last_fullsized":"2026-02-26T03:02:48.761742+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-26T03:01:18.659251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-26T03:01:18.659251+0000","last_clean_scrub_stamp":"2026-02-26T03:01:18.659251+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-27T11:49:30.505493+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00054467999999999997,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":45,"reported_epoch":42,"state":"active+clean","last_fresh":"2026-02-26T03:02:48.762548+0000","last_change":"2026-02-26T03:02:48.762548+0000","last_active":"2026-02-26T03:02:48.762548+0000","last_peered":"2026-02-26T03:02:48.762548+0000","last_clean":"2026-02-26T03:02:48.762548+0000","last_became_active":"2026-02-26T03:02:48.762315+0000","last_became_peered":"2026-02-26T03:02:48.762315+0000","last_unstale":"2026-02-26T03:02:48.762548+0000","last_undegraded":"2026-02-26T03:02:48.762548+0000","last_fullsized":"2026-02-26T03:02:48.762548+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-26T03:01:18.659251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-26T03:01:18.659251+0000","last_clean_scrub_stamp":"2026-02-26T03:01:18.659251+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-27T10:46:34.526556+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":52,"reported_epoch":42,"state":"active+clean","last_fresh":"2026-02-26T03:02:48.230874+0000","last_change":"2026-02-26T03:02:16.406053+0000","last_active":"2026-02-26T03:02:48.230874+0000","last_peered":"2026-02-26T03:02:48.230874+0000","last_clean":"2026-02-26T03:02:48.230874+0000","last_became_active":"2026-02-26T03:02:16.405837+0000","last_became_peered":"2026-02-26T03:02:16.405837+0000","last_unstale":"2026-02-26T03:02:48.230874+0000","last_undegraded":"2026-02-26T03:02:48.230874+0000","last_fullsized":"2026-02-26T03:02:48.230874+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-26T03:01:18.659251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-26T03:01:18.659251+0000","last_clean_scrub_stamp":"2026-02-26T03:01:18.659251+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-27T14:23:01.074046+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00037062400000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":65210,"num_objects":101,"num_object_clones":0,"num_object_copies":202,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":101,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":102,"num_write_kb":102,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":6620,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":827392,"data_stored":130420,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6077,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":2,"acting":2,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":1,"up_from":29,"seq":124554051597,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32324,"kb_used_data":752,"kb_used_omap":4,"kb_used_meta":31547,"kb_avail":94339516,"statfs":{"total":96636764160,"available":96603664384,"internally_reserved":0,"allocated":770048,"data_stored":190290,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4434,"internal_metadata":32304814},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":41,"seq":176093659142,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":29504,"kb_used_data":748,"kb_used_omap":11,"kb_used_meta":28724,"kb_avail":94342336,"statfs":{"total":96636764160,"available":96606552064,"internally_reserved":0,"allocated":765952,"data_stored":190271,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12155,"internal_metadata":29413509},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":27,"seq":115964117007,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31988,"kb_used_data":348,"kb_used_omap":4,"kb_used_meta":31611,"kb_avail":94339852,"statfs":{"total":96636764160,"available":96604008448,"internally_reserved":0,"allocated":356352,"data_stored":125080,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4432,"internal_metadata":32370352},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":413696,"data_stored":65210,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":413696,"data_stored":65210,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6077,"internal_metadata":0}]}} 2026-02-26T03:03:11.628 DEBUG:teuthology.orchestra.run.trial197:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2026-02-26T03:03:11.785 INFO:teuthology.orchestra.run.trial197.stdout: 2026-02-26T03:03:11.785 INFO:teuthology.orchestra.run.trial197.stdout:{"epoch":42,"fsid":"658087b3-2758-487d-ba7c-f8b171a8f312","created":"2026-02-26T03:01:04.144741+0000","modified":"2026-02-26T03:02:48.229711+0000","last_up_change":"2026-02-26T03:02:47.228819+0000","last_in_change":"2026-02-26T03:02:14.145480+0000","flags":"nodown,noout,noin,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5800384,"flags_set":["nodown","noin","noout","pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":9,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":"rbd","create_time":"2026-02-26T03:01:18.430774+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"13","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":2,"snap_epoch":13,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"foo","create_time":"2026-02-26T03:01:31.663753+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{},"read_balance":{"score_type":"Fair distribution","score_acting":2.9900000095367432,"score_stable":2.9900000095367432,"optimal_score":0.67000001668930054,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"696e32c7-b2ad-41e8-9c32-f7e12eb617ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":8,"last_clean_end":20,"up_from":27,"up_thru":40,"down_at":21,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.197:6800","nonce":1385562264},{"type":"v1","addr":"10.20.193.197:6801","nonce":1385562264}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.197:6802","nonce":1385562264},{"type":"v1","addr":"10.20.193.197:6803","nonce":1385562264}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.197:6806","nonce":1385562264},{"type":"v1","addr":"10.20.193.197:6807","nonce":1385562264}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.197:6804","nonce":1385562264},{"type":"v1","addr":"10.20.193.197:6805","nonce":1385562264}]},"public_addr":"10.20.193.197:6801/1385562264","cluster_addr":"10.20.193.197:6803/1385562264","heartbeat_back_addr":"10.20.193.197:6807/1385562264","heartbeat_front_addr":"10.20.193.197:6805/1385562264","state":["exists","up"]},{"osd":1,"uuid":"606c6d1b-0452-4860-a1dc-ca58763aa4e9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":20,"up_from":29,"up_thru":41,"down_at":22,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.197:6810","nonce":4100957723},{"type":"v1","addr":"10.20.193.197:6811","nonce":4100957723}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.197:6812","nonce":4100957723},{"type":"v1","addr":"10.20.193.197:6813","nonce":4100957723}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.197:6816","nonce":4100957723},{"type":"v1","addr":"10.20.193.197:6817","nonce":4100957723}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.197:6814","nonce":4100957723},{"type":"v1","addr":"10.20.193.197:6815","nonce":4100957723}]},"public_addr":"10.20.193.197:6811/4100957723","cluster_addr":"10.20.193.197:6813/4100957723","heartbeat_back_addr":"10.20.193.197:6817/4100957723","heartbeat_front_addr":"10.20.193.197:6815/4100957723","state":["exists","up"]},{"osd":2,"uuid":"4a54c53b-ac96-4483-97dc-bf76ae085e1b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":36,"last_clean_end":38,"up_from":41,"up_thru":41,"down_at":40,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.197:6818","nonce":2048171066},{"type":"v1","addr":"10.20.193.197:6819","nonce":2048171066}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.197:6820","nonce":2048171066},{"type":"v1","addr":"10.20.193.197:6821","nonce":2048171066}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.197:6824","nonce":2048171066},{"type":"v1","addr":"10.20.193.197:6825","nonce":2048171066}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.197:6822","nonce":2048171066},{"type":"v1","addr":"10.20.193.197:6823","nonce":2048171066}]},"public_addr":"10.20.193.197:6819/2048171066","cluster_addr":"10.20.193.197:6821/2048171066","heartbeat_back_addr":"10.20.193.197:6825/2048171066","heartbeat_front_addr":"10.20.193.197:6823/2048171066","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"2026-02-26T03:01:53.175240+0000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2026-02-26T03:01:11.283523+0000","dead_epoch":0},{"osd":1,"down_stamp":"2026-02-26T03:01:54.177094+0000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2026-02-26T03:01:11.607888+0000","dead_epoch":0},{"osd":2,"down_stamp":"2026-02-26T03:02:46.227230+0000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2026-02-26T03:01:11.277139+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-26T03:03:12.793 INFO:tasks.ceph:Scrubbing osd.0 2026-02-26T03:03:12.794 DEBUG:teuthology.orchestra.run.trial197:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 config set osd_debug_deep_scrub_sleep 0 2026-02-26T03:03:12.872 INFO:teuthology.orchestra.run.trial197.stdout:{ 2026-02-26T03:03:12.872 INFO:teuthology.orchestra.run.trial197.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2026-02-26T03:03:12.872 INFO:teuthology.orchestra.run.trial197.stdout:} 2026-02-26T03:03:12.877 DEBUG:teuthology.orchestra.run.trial197:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 0 2026-02-26T03:03:13.033 INFO:teuthology.orchestra.run.trial197.stderr:instructed osd(s) 0 to deep-scrub 2026-02-26T03:03:13.040 INFO:tasks.ceph:Scrubbing osd.1 2026-02-26T03:03:13.040 DEBUG:teuthology.orchestra.run.trial197:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 config set osd_debug_deep_scrub_sleep 0 2026-02-26T03:03:13.121 INFO:teuthology.orchestra.run.trial197.stdout:{ 2026-02-26T03:03:13.121 INFO:teuthology.orchestra.run.trial197.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2026-02-26T03:03:13.121 INFO:teuthology.orchestra.run.trial197.stdout:} 2026-02-26T03:03:13.126 DEBUG:teuthology.orchestra.run.trial197:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 1 2026-02-26T03:03:13.281 INFO:teuthology.orchestra.run.trial197.stderr:instructed osd(s) 1 to deep-scrub 2026-02-26T03:03:13.289 INFO:tasks.ceph:Scrubbing osd.2 2026-02-26T03:03:13.289 DEBUG:teuthology.orchestra.run.trial197:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 config set osd_debug_deep_scrub_sleep 0 2026-02-26T03:03:13.370 INFO:teuthology.orchestra.run.trial197.stdout:{ 2026-02-26T03:03:13.370 INFO:teuthology.orchestra.run.trial197.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2026-02-26T03:03:13.370 INFO:teuthology.orchestra.run.trial197.stdout:} 2026-02-26T03:03:13.375 DEBUG:teuthology.orchestra.run.trial197:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 2 2026-02-26T03:03:13.534 INFO:teuthology.orchestra.run.trial197.stderr:instructed osd(s) 2 to deep-scrub 2026-02-26T03:03:13.543 DEBUG:teuthology.orchestra.run.trial197:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-26T03:03:13.704 INFO:teuthology.orchestra.run.trial197.stdout: 2026-02-26T03:03:13.704 INFO:teuthology.orchestra.run.trial197.stderr:dumped all 2026-02-26T03:03:13.713 INFO:teuthology.orchestra.run.trial197.stdout:{"pg_ready":true,"pg_map":{"version":100,"stamp":"2026-02-26T03:03:12.391860+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":65229,"num_objects":103,"num_object_clones":0,"num_object_copies":206,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":103,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":104,"num_write_kb":104,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":6620,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":10,"ondisk_log_size":10,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":93816,"kb_used_data":1848,"kb_used_omap":20,"kb_used_meta":91883,"kb_avail":283021704,"statfs":{"total":289910292480,"available":289814224896,"internally_reserved":0,"allocated":1892352,"data_stored":505641,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":21021,"internal_metadata":94088675},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.000967"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":60,"reported_epoch":42,"state":"active+clean","last_fresh":"2026-02-26T03:02:48.232684+0000","last_change":"2026-02-26T03:02:48.232684+0000","last_active":"2026-02-26T03:02:48.232684+0000","last_peered":"2026-02-26T03:02:48.232684+0000","last_clean":"2026-02-26T03:02:48.232684+0000","last_became_active":"2026-02-26T03:02:48.232368+0000","last_became_peered":"2026-02-26T03:02:48.232368+0000","last_unstale":"2026-02-26T03:02:48.232684+0000","last_undegraded":"2026-02-26T03:02:48.232684+0000","last_fullsized":"2026-02-26T03:02:48.232684+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-26T03:01:18.659251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-26T03:01:18.659251+0000","last_clean_scrub_stamp":"2026-02-26T03:01:18.659251+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-27T11:57:01.371827+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00060011299999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":52,"reported_epoch":42,"state":"active+clean","last_fresh":"2026-02-26T03:02:48.230981+0000","last_change":"2026-02-26T03:02:16.405639+0000","last_active":"2026-02-26T03:02:48.230981+0000","last_peered":"2026-02-26T03:02:48.230981+0000","last_clean":"2026-02-26T03:02:48.230981+0000","last_became_active":"2026-02-26T03:02:16.405505+0000","last_became_peered":"2026-02-26T03:02:16.405505+0000","last_unstale":"2026-02-26T03:02:48.230981+0000","last_undegraded":"2026-02-26T03:02:48.230981+0000","last_fullsized":"2026-02-26T03:02:48.230981+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-26T03:01:18.659251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-26T03:01:18.659251+0000","last_clean_scrub_stamp":"2026-02-26T03:01:18.659251+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-27T14:28:28.066117+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00068042299999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":45,"reported_epoch":42,"state":"active+clean","last_fresh":"2026-02-26T03:02:48.762716+0000","last_change":"2026-02-26T03:02:48.762716+0000","last_active":"2026-02-26T03:02:48.762716+0000","last_peered":"2026-02-26T03:02:48.762716+0000","last_clean":"2026-02-26T03:02:48.762716+0000","last_became_active":"2026-02-26T03:02:48.762400+0000","last_became_peered":"2026-02-26T03:02:48.762400+0000","last_unstale":"2026-02-26T03:02:48.762716+0000","last_undegraded":"2026-02-26T03:02:48.762716+0000","last_fullsized":"2026-02-26T03:02:48.762716+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-26T03:01:18.659251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-26T03:01:18.659251+0000","last_clean_scrub_stamp":"2026-02-26T03:01:18.659251+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-27T05:49:10.168518+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":52,"reported_epoch":42,"state":"active+clean","last_fresh":"2026-02-26T03:02:48.230975+0000","last_change":"2026-02-26T03:02:16.406146+0000","last_active":"2026-02-26T03:02:48.230975+0000","last_peered":"2026-02-26T03:02:48.230975+0000","last_clean":"2026-02-26T03:02:48.230975+0000","last_became_active":"2026-02-26T03:02:16.405921+0000","last_became_peered":"2026-02-26T03:02:16.405921+0000","last_unstale":"2026-02-26T03:02:48.230975+0000","last_undegraded":"2026-02-26T03:02:48.230975+0000","last_fullsized":"2026-02-26T03:02:48.230975+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-26T03:01:18.659251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-26T03:01:18.659251+0000","last_clean_scrub_stamp":"2026-02-26T03:01:18.659251+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-27T08:15:04.934373+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000482733,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"38'102","reported_seq":170,"reported_epoch":42,"state":"active+clean","last_fresh":"2026-02-26T03:02:48.241816+0000","last_change":"2026-02-26T03:02:48.241816+0000","last_active":"2026-02-26T03:02:48.241816+0000","last_peered":"2026-02-26T03:02:48.241816+0000","last_clean":"2026-02-26T03:02:48.241816+0000","last_became_active":"2026-02-26T03:02:48.232430+0000","last_became_peered":"2026-02-26T03:02:48.232430+0000","last_unstale":"2026-02-26T03:02:48.241816+0000","last_undegraded":"2026-02-26T03:02:48.241816+0000","last_fullsized":"2026-02-26T03:02:48.241816+0000","mapping_epoch":41,"log_start":"20'95","ondisk_log_start":"20'95","created":18,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-26T03:01:32.157530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-26T03:01:32.157530+0000","last_clean_scrub_stamp":"2026-02-26T03:01:32.157530+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":95,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-27T07:23:32.702444+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":65210,"num_objects":101,"num_object_clones":0,"num_object_copies":202,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":101,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":102,"num_write_kb":102,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":6620,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"11'1","reported_seq":65,"reported_epoch":42,"state":"active+clean","last_fresh":"2026-02-26T03:02:48.232628+0000","last_change":"2026-02-26T03:02:48.232628+0000","last_active":"2026-02-26T03:02:48.232628+0000","last_peered":"2026-02-26T03:02:48.232628+0000","last_clean":"2026-02-26T03:02:48.232628+0000","last_became_active":"2026-02-26T03:02:48.232286+0000","last_became_peered":"2026-02-26T03:02:48.232286+0000","last_unstale":"2026-02-26T03:02:48.232628+0000","last_undegraded":"2026-02-26T03:02:48.232628+0000","last_fullsized":"2026-02-26T03:02:48.232628+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-26T03:01:18.659251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-26T03:01:18.659251+0000","last_clean_scrub_stamp":"2026-02-26T03:01:18.659251+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-27T05:55:53.987752+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00034461499999999999,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"13'2","reported_seq":56,"reported_epoch":42,"state":"active+clean","last_fresh":"2026-02-26T03:02:48.761742+0000","last_change":"2026-02-26T03:02:15.149310+0000","last_active":"2026-02-26T03:02:48.761742+0000","last_peered":"2026-02-26T03:02:48.761742+0000","last_clean":"2026-02-26T03:02:48.761742+0000","last_became_active":"2026-02-26T03:02:15.149144+0000","last_became_peered":"2026-02-26T03:02:15.149144+0000","last_unstale":"2026-02-26T03:02:48.761742+0000","last_undegraded":"2026-02-26T03:02:48.761742+0000","last_fullsized":"2026-02-26T03:02:48.761742+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-26T03:01:18.659251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-26T03:01:18.659251+0000","last_clean_scrub_stamp":"2026-02-26T03:01:18.659251+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-27T11:49:30.505493+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00054467999999999997,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":45,"reported_epoch":42,"state":"active+clean","last_fresh":"2026-02-26T03:02:48.762548+0000","last_change":"2026-02-26T03:02:48.762548+0000","last_active":"2026-02-26T03:02:48.762548+0000","last_peered":"2026-02-26T03:02:48.762548+0000","last_clean":"2026-02-26T03:02:48.762548+0000","last_became_active":"2026-02-26T03:02:48.762315+0000","last_became_peered":"2026-02-26T03:02:48.762315+0000","last_unstale":"2026-02-26T03:02:48.762548+0000","last_undegraded":"2026-02-26T03:02:48.762548+0000","last_fullsized":"2026-02-26T03:02:48.762548+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-26T03:01:18.659251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-26T03:01:18.659251+0000","last_clean_scrub_stamp":"2026-02-26T03:01:18.659251+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-27T10:46:34.526556+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":52,"reported_epoch":42,"state":"active+clean","last_fresh":"2026-02-26T03:02:48.230874+0000","last_change":"2026-02-26T03:02:16.406053+0000","last_active":"2026-02-26T03:02:48.230874+0000","last_peered":"2026-02-26T03:02:48.230874+0000","last_clean":"2026-02-26T03:02:48.230874+0000","last_became_active":"2026-02-26T03:02:16.405837+0000","last_became_peered":"2026-02-26T03:02:16.405837+0000","last_unstale":"2026-02-26T03:02:48.230874+0000","last_undegraded":"2026-02-26T03:02:48.230874+0000","last_fullsized":"2026-02-26T03:02:48.230874+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-26T03:01:18.659251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-26T03:01:18.659251+0000","last_clean_scrub_stamp":"2026-02-26T03:01:18.659251+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-27T14:23:01.074046+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00037062400000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":65210,"num_objects":101,"num_object_clones":0,"num_object_copies":202,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":101,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":102,"num_write_kb":102,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":6620,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":827392,"data_stored":130420,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6077,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":2,"acting":2,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":1,"up_from":29,"seq":124554051597,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32324,"kb_used_data":752,"kb_used_omap":4,"kb_used_meta":31547,"kb_avail":94339516,"statfs":{"total":96636764160,"available":96603664384,"internally_reserved":0,"allocated":770048,"data_stored":190290,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4434,"internal_metadata":32304814},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":41,"seq":176093659142,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":29504,"kb_used_data":748,"kb_used_omap":11,"kb_used_meta":28724,"kb_avail":94342336,"statfs":{"total":96636764160,"available":96606552064,"internally_reserved":0,"allocated":765952,"data_stored":190271,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12155,"internal_metadata":29413509},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":27,"seq":115964117007,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31988,"kb_used_data":348,"kb_used_omap":4,"kb_used_meta":31611,"kb_avail":94339852,"statfs":{"total":96636764160,"available":96604008448,"internally_reserved":0,"allocated":356352,"data_stored":125080,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4432,"internal_metadata":32370352},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":413696,"data_stored":65210,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":413696,"data_stored":65210,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6077,"internal_metadata":0}]}} 2026-02-26T03:03:13.720 INFO:tasks.ceph:pgid 1.7 last_scrub_stamp 2026-02-26T03:01:18.659251+0000 time.struct_time(tm_year=2026, tm_mon=2, tm_mday=26, tm_hour=3, tm_min=1, tm_sec=18, tm_wday=3, tm_yday=57, tm_isdst=-1) <= time.struct_time(tm_year=2026, tm_mon=2, tm_mday=26, tm_hour=3, tm_min=3, tm_sec=11, tm_wday=3, tm_yday=57, tm_isdst=0) 2026-02-26T03:03:13.720 INFO:tasks.ceph:pgid 1.6 last_scrub_stamp 2026-02-26T03:01:18.659251+0000 time.struct_time(tm_year=2026, tm_mon=2, tm_mday=26, tm_hour=3, tm_min=1, tm_sec=18, tm_wday=3, tm_yday=57, tm_isdst=-1) <= time.struct_time(tm_year=2026, tm_mon=2, tm_mday=26, tm_hour=3, tm_min=3, tm_sec=11, tm_wday=3, tm_yday=57, tm_isdst=0) 2026-02-26T03:03:13.720 INFO:tasks.ceph:pgid 1.5 last_scrub_stamp 2026-02-26T03:01:18.659251+0000 time.struct_time(tm_year=2026, tm_mon=2, tm_mday=26, tm_hour=3, tm_min=1, tm_sec=18, tm_wday=3, tm_yday=57, tm_isdst=-1) <= time.struct_time(tm_year=2026, tm_mon=2, tm_mday=26, tm_hour=3, tm_min=3, tm_sec=11, tm_wday=3, tm_yday=57, tm_isdst=0) 2026-02-26T03:03:13.720 INFO:tasks.ceph:pgid 1.4 last_scrub_stamp 2026-02-26T03:01:18.659251+0000 time.struct_time(tm_year=2026, tm_mon=2, tm_mday=26, tm_hour=3, tm_min=1, tm_sec=18, tm_wday=3, tm_yday=57, tm_isdst=-1) <= time.struct_time(tm_year=2026, tm_mon=2, tm_mday=26, tm_hour=3, tm_min=3, tm_sec=11, tm_wday=3, tm_yday=57, tm_isdst=0) 2026-02-26T03:03:13.721 INFO:tasks.ceph:pgid 2.0 last_scrub_stamp 2026-02-26T03:01:32.157530+0000 time.struct_time(tm_year=2026, tm_mon=2, tm_mday=26, tm_hour=3, tm_min=1, tm_sec=32, tm_wday=3, tm_yday=57, tm_isdst=-1) <= time.struct_time(tm_year=2026, tm_mon=2, tm_mday=26, tm_hour=3, tm_min=3, tm_sec=11, tm_wday=3, tm_yday=57, tm_isdst=0) 2026-02-26T03:03:13.721 INFO:tasks.ceph:pgid 1.3 last_scrub_stamp 2026-02-26T03:01:18.659251+0000 time.struct_time(tm_year=2026, tm_mon=2, tm_mday=26, tm_hour=3, tm_min=1, tm_sec=18, tm_wday=3, tm_yday=57, tm_isdst=-1) <= time.struct_time(tm_year=2026, tm_mon=2, tm_mday=26, tm_hour=3, tm_min=3, tm_sec=11, tm_wday=3, tm_yday=57, tm_isdst=0) 2026-02-26T03:03:13.721 INFO:tasks.ceph:pgid 1.2 last_scrub_stamp 2026-02-26T03:01:18.659251+0000 time.struct_time(tm_year=2026, tm_mon=2, tm_mday=26, tm_hour=3, tm_min=1, tm_sec=18, tm_wday=3, tm_yday=57, tm_isdst=-1) <= time.struct_time(tm_year=2026, tm_mon=2, tm_mday=26, tm_hour=3, tm_min=3, tm_sec=11, tm_wday=3, tm_yday=57, tm_isdst=0) 2026-02-26T03:03:13.721 INFO:tasks.ceph:pgid 1.1 last_scrub_stamp 2026-02-26T03:01:18.659251+0000 time.struct_time(tm_year=2026, tm_mon=2, tm_mday=26, tm_hour=3, tm_min=1, tm_sec=18, tm_wday=3, tm_yday=57, tm_isdst=-1) <= time.struct_time(tm_year=2026, tm_mon=2, tm_mday=26, tm_hour=3, tm_min=3, tm_sec=11, tm_wday=3, tm_yday=57, tm_isdst=0) 2026-02-26T03:03:13.721 INFO:tasks.ceph:pgid 1.0 last_scrub_stamp 2026-02-26T03:01:18.659251+0000 time.struct_time(tm_year=2026, tm_mon=2, tm_mday=26, tm_hour=3, tm_min=1, tm_sec=18, tm_wday=3, tm_yday=57, tm_isdst=-1) <= time.struct_time(tm_year=2026, tm_mon=2, tm_mday=26, tm_hour=3, tm_min=3, tm_sec=11, tm_wday=3, tm_yday=57, tm_isdst=0) 2026-02-26T03:03:13.722 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2026-02-26T03:03:33.722 DEBUG:teuthology.orchestra.run.trial197:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-26T03:03:33.881 INFO:teuthology.orchestra.run.trial197.stdout: 2026-02-26T03:03:33.881 INFO:teuthology.orchestra.run.trial197.stderr:dumped all 2026-02-26T03:03:33.890 INFO:teuthology.orchestra.run.trial197.stdout:{"pg_ready":true,"pg_map":{"version":110,"stamp":"2026-02-26T03:03:32.393538+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":65229,"num_objects":103,"num_object_clones":0,"num_object_copies":206,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":103,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":104,"num_write_kb":104,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":6620,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":113,"ondisk_log_size":113,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":93800,"kb_used_data":1848,"kb_used_omap":20,"kb_used_meta":91883,"kb_avail":283021720,"statfs":{"total":289910292480,"available":289814241280,"internally_reserved":0,"allocated":1892352,"data_stored":505641,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":21021,"internal_metadata":94088675},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.000891"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":69,"reported_epoch":42,"state":"active+clean","last_fresh":"2026-02-26T03:03:18.052503+0000","last_change":"2026-02-26T03:03:18.052518+0000","last_active":"2026-02-26T03:03:18.052503+0000","last_peered":"2026-02-26T03:03:18.052503+0000","last_clean":"2026-02-26T03:03:18.052503+0000","last_became_active":"2026-02-26T03:02:48.232368+0000","last_became_peered":"2026-02-26T03:02:48.232368+0000","last_unstale":"2026-02-26T03:03:18.052503+0000","last_undegraded":"2026-02-26T03:03:18.052503+0000","last_fullsized":"2026-02-26T03:03:18.052503+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-26T03:03:18.052457+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-26T03:03:18.052457+0000","last_clean_scrub_stamp":"2026-02-26T03:03:18.052457+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-27T06:14:01.822920+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00060011299999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":61,"reported_epoch":42,"state":"active+clean","last_fresh":"2026-02-26T03:03:14.072813+0000","last_change":"2026-02-26T03:03:14.072827+0000","last_active":"2026-02-26T03:03:14.072813+0000","last_peered":"2026-02-26T03:03:14.072813+0000","last_clean":"2026-02-26T03:03:14.072813+0000","last_became_active":"2026-02-26T03:02:16.405505+0000","last_became_peered":"2026-02-26T03:02:16.405505+0000","last_unstale":"2026-02-26T03:03:14.072813+0000","last_undegraded":"2026-02-26T03:03:14.072813+0000","last_fullsized":"2026-02-26T03:03:14.072813+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-26T03:03:14.072754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-26T03:03:14.072754+0000","last_clean_scrub_stamp":"2026-02-26T03:03:14.072754+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-27T05:27:43.725432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00068042299999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":54,"reported_epoch":42,"state":"active+clean","last_fresh":"2026-02-26T03:03:15.615358+0000","last_change":"2026-02-26T03:03:15.615380+0000","last_active":"2026-02-26T03:03:15.615358+0000","last_peered":"2026-02-26T03:03:15.615358+0000","last_clean":"2026-02-26T03:03:15.615358+0000","last_became_active":"2026-02-26T03:02:48.762400+0000","last_became_peered":"2026-02-26T03:02:48.762400+0000","last_unstale":"2026-02-26T03:03:15.615358+0000","last_undegraded":"2026-02-26T03:03:15.615358+0000","last_fullsized":"2026-02-26T03:03:15.615358+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-26T03:03:15.615325+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-26T03:03:15.615325+0000","last_clean_scrub_stamp":"2026-02-26T03:03:15.615325+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-27T11:27:57.196896+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":61,"reported_epoch":42,"state":"active+clean","last_fresh":"2026-02-26T03:03:16.014588+0000","last_change":"2026-02-26T03:03:16.014618+0000","last_active":"2026-02-26T03:03:16.014588+0000","last_peered":"2026-02-26T03:03:16.014588+0000","last_clean":"2026-02-26T03:03:16.014588+0000","last_became_active":"2026-02-26T03:02:16.405921+0000","last_became_peered":"2026-02-26T03:02:16.405921+0000","last_unstale":"2026-02-26T03:03:16.014588+0000","last_undegraded":"2026-02-26T03:03:16.014588+0000","last_fullsized":"2026-02-26T03:03:16.014588+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-26T03:03:16.014515+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-26T03:03:16.014515+0000","last_clean_scrub_stamp":"2026-02-26T03:03:16.014515+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-27T04:22:26.982290+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000482733,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"42'203","reported_seq":296,"reported_epoch":42,"state":"active+clean","last_fresh":"2026-02-26T03:03:13.634956+0000","last_change":"2026-02-26T03:03:13.634973+0000","last_active":"2026-02-26T03:03:13.634956+0000","last_peered":"2026-02-26T03:03:13.634956+0000","last_clean":"2026-02-26T03:03:13.634956+0000","last_became_active":"2026-02-26T03:02:48.232430+0000","last_became_peered":"2026-02-26T03:02:48.232430+0000","last_unstale":"2026-02-26T03:03:13.634956+0000","last_undegraded":"2026-02-26T03:03:13.634956+0000","last_fullsized":"2026-02-26T03:03:13.634956+0000","mapping_epoch":41,"log_start":"20'95","ondisk_log_start":"20'95","created":18,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"42'203","last_scrub_stamp":"2026-02-26T03:03:13.634926+0000","last_deep_scrub":"42'203","last_deep_scrub_stamp":"2026-02-26T03:03:13.634926+0000","last_clean_scrub_stamp":"2026-02-26T03:03:13.634926+0000","objects_scrubbed":101,"log_size":108,"log_dups_size":95,"ondisk_log_size":108,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2026-02-27T09:18:49.152111+0000","scrub_duration":31,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":65210,"num_objects":101,"num_object_clones":0,"num_object_copies":202,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":101,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":102,"num_write_kb":102,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":6620,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"42'2","reported_seq":75,"reported_epoch":42,"state":"active+clean","last_fresh":"2026-02-26T03:03:17.039686+0000","last_change":"2026-02-26T03:03:17.039700+0000","last_active":"2026-02-26T03:03:17.039686+0000","last_peered":"2026-02-26T03:03:17.039686+0000","last_clean":"2026-02-26T03:03:17.039686+0000","last_became_active":"2026-02-26T03:02:48.232286+0000","last_became_peered":"2026-02-26T03:02:48.232286+0000","last_unstale":"2026-02-26T03:03:17.039686+0000","last_undegraded":"2026-02-26T03:03:17.039686+0000","last_fullsized":"2026-02-26T03:03:17.039686+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"42'2","last_scrub_stamp":"2026-02-26T03:03:17.039659+0000","last_deep_scrub":"42'2","last_deep_scrub_stamp":"2026-02-26T03:03:17.039659+0000","last_clean_scrub_stamp":"2026-02-26T03:03:17.039659+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2026-02-27T06:00:02.557818+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00034461499999999999,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"42'3","reported_seq":66,"reported_epoch":42,"state":"active+clean","last_fresh":"2026-02-26T03:03:13.865788+0000","last_change":"2026-02-26T03:03:13.865818+0000","last_active":"2026-02-26T03:03:13.865788+0000","last_peered":"2026-02-26T03:03:13.865788+0000","last_clean":"2026-02-26T03:03:13.865788+0000","last_became_active":"2026-02-26T03:02:15.149144+0000","last_became_peered":"2026-02-26T03:02:15.149144+0000","last_unstale":"2026-02-26T03:03:13.865788+0000","last_undegraded":"2026-02-26T03:03:13.865788+0000","last_fullsized":"2026-02-26T03:03:13.865788+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"42'3","last_scrub_stamp":"2026-02-26T03:03:13.865672+0000","last_deep_scrub":"42'3","last_deep_scrub_stamp":"2026-02-26T03:03:13.865672+0000","last_clean_scrub_stamp":"2026-02-26T03:03:13.865672+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2026-02-27T03:35:25.086580+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00054467999999999997,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":54,"reported_epoch":42,"state":"active+clean","last_fresh":"2026-02-26T03:03:14.649352+0000","last_change":"2026-02-26T03:03:14.649373+0000","last_active":"2026-02-26T03:03:14.649352+0000","last_peered":"2026-02-26T03:03:14.649352+0000","last_clean":"2026-02-26T03:03:14.649352+0000","last_became_active":"2026-02-26T03:02:48.762315+0000","last_became_peered":"2026-02-26T03:02:48.762315+0000","last_unstale":"2026-02-26T03:03:14.649352+0000","last_undegraded":"2026-02-26T03:03:14.649352+0000","last_fullsized":"2026-02-26T03:03:14.649352+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-26T03:03:14.649312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-26T03:03:14.649312+0000","last_clean_scrub_stamp":"2026-02-26T03:03:14.649312+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-27T14:06:52.970932+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":61,"reported_epoch":42,"state":"active+clean","last_fresh":"2026-02-26T03:03:15.053029+0000","last_change":"2026-02-26T03:03:15.053043+0000","last_active":"2026-02-26T03:03:15.053029+0000","last_peered":"2026-02-26T03:03:15.053029+0000","last_clean":"2026-02-26T03:03:15.053029+0000","last_became_active":"2026-02-26T03:02:16.405837+0000","last_became_peered":"2026-02-26T03:02:16.405837+0000","last_unstale":"2026-02-26T03:03:15.053029+0000","last_undegraded":"2026-02-26T03:03:15.053029+0000","last_fullsized":"2026-02-26T03:03:15.053029+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-26T03:03:15.052981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-26T03:03:15.052981+0000","last_clean_scrub_stamp":"2026-02-26T03:03:15.052981+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-27T03:32:06.061597+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00037062400000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":65210,"num_objects":101,"num_object_clones":0,"num_object_copies":202,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":101,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":102,"num_write_kb":102,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":6620,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":827392,"data_stored":130420,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6077,"internal_metadata":0},"log_size":108,"ondisk_log_size":108,"up":2,"acting":2,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":5,"ondisk_log_size":5,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":1,"up_from":29,"seq":124554051601,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32320,"kb_used_data":752,"kb_used_omap":4,"kb_used_meta":31547,"kb_avail":94339520,"statfs":{"total":96636764160,"available":96603668480,"internally_reserved":0,"allocated":770048,"data_stored":190290,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4434,"internal_metadata":32304814},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":41,"seq":176093659146,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":29500,"kb_used_data":748,"kb_used_omap":11,"kb_used_meta":28724,"kb_avail":94342340,"statfs":{"total":96636764160,"available":96606556160,"internally_reserved":0,"allocated":765952,"data_stored":190271,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12155,"internal_metadata":29413509},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":27,"seq":115964117011,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31980,"kb_used_data":348,"kb_used_omap":4,"kb_used_meta":31611,"kb_avail":94339860,"statfs":{"total":96636764160,"available":96604016640,"internally_reserved":0,"allocated":356352,"data_stored":125080,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4432,"internal_metadata":32370352},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":413696,"data_stored":65210,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":413696,"data_stored":65210,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6077,"internal_metadata":0}]}} 2026-02-26T03:03:33.891 DEBUG:teuthology.orchestra.run.trial197:> sudo ceph --cluster ceph config set global mon_health_to_clog false 2026-02-26T03:03:34.059 INFO:teuthology.misc:Shutting down mds daemons... 2026-02-26T03:03:34.060 INFO:teuthology.misc:Shutting down osd daemons... 2026-02-26T03:03:34.060 DEBUG:tasks.ceph.osd.0:waiting for process to exit 2026-02-26T03:03:34.060 INFO:teuthology.orchestra.run:waiting for 300 2026-02-26T03:03:34.109 INFO:tasks.ceph.osd.0:Stopped 2026-02-26T03:03:34.109 DEBUG:tasks.ceph.osd.1:waiting for process to exit 2026-02-26T03:03:34.109 INFO:teuthology.orchestra.run:waiting for 300 2026-02-26T03:03:34.143 INFO:tasks.ceph.osd.1:Stopped 2026-02-26T03:03:34.143 DEBUG:tasks.ceph.osd.2:waiting for process to exit 2026-02-26T03:03:34.144 INFO:teuthology.orchestra.run:waiting for 300 2026-02-26T03:03:34.215 INFO:tasks.ceph.osd.2:Stopped 2026-02-26T03:03:34.215 INFO:teuthology.misc:Shutting down mgr daemons... 2026-02-26T03:03:34.215 DEBUG:tasks.ceph.mgr.x:waiting for process to exit 2026-02-26T03:03:34.215 INFO:teuthology.orchestra.run:waiting for 300 2026-02-26T03:03:34.285 INFO:tasks.ceph.mgr.x:Stopped 2026-02-26T03:03:34.286 INFO:teuthology.misc:Shutting down mon daemons... 2026-02-26T03:03:34.286 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2026-02-26T03:03:34.286 INFO:teuthology.orchestra.run:waiting for 300 2026-02-26T03:03:34.334 INFO:tasks.ceph.mon.a:Stopped 2026-02-26T03:03:34.334 INFO:tasks.ceph:Checking cluster log for badness... 2026-02-26T03:03:34.334 DEBUG:teuthology.orchestra.run.trial197:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v 'overall HEALTH_' | egrep -v '\(OSDMAP_FLAGS\)' | egrep -v '\(OSD_' | egrep -v '\(PG_' | egrep -v '\(OBJECT_' | egrep -v '\(POOL_APP_NOT_ENABLED\)' | egrep -v '\(OSD_SLOW_PING_TIME' | egrep -v '\(MON_DOWN\)' | head -n 1 2026-02-26T03:03:34.401 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-0 on ubuntu@trial197.front.sepia.ceph.com 2026-02-26T03:03:34.401 DEBUG:teuthology.orchestra.run.trial197:> sync && sudo umount -f /var/lib/ceph/osd/ceph-0 2026-02-26T03:03:34.476 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-1 on ubuntu@trial197.front.sepia.ceph.com 2026-02-26T03:03:34.477 DEBUG:teuthology.orchestra.run.trial197:> sync && sudo umount -f /var/lib/ceph/osd/ceph-1 2026-02-26T03:03:34.549 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-2 on ubuntu@trial197.front.sepia.ceph.com 2026-02-26T03:03:34.549 DEBUG:teuthology.orchestra.run.trial197:> sync && sudo umount -f /var/lib/ceph/osd/ceph-2 2026-02-26T03:03:34.617 INFO:tasks.ceph:Archiving mon data... 2026-02-26T03:03:34.619 DEBUG:teuthology.misc:Transferring archived files from trial197:/var/lib/ceph/mon/ceph-a to /home/teuthworker/mnt/teuthology/yuriw-2026-02-26_01:51:16-rados-wip-yuri9-testing-2026-02-25-1600-squid-distro-default-trial/71117/data/mon.a.tgz 2026-02-26T03:03:34.619 DEBUG:teuthology.orchestra.run.trial197:> mktemp 2026-02-26T03:03:34.673 INFO:teuthology.orchestra.run.trial197.stdout:/tmp/tmp.CABgfl5CgE 2026-02-26T03:03:34.673 DEBUG:teuthology.orchestra.run.trial197:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-a -- . > /tmp/tmp.CABgfl5CgE 2026-02-26T03:03:34.758 DEBUG:teuthology.orchestra.run.trial197:> sudo chmod 0666 /tmp/tmp.CABgfl5CgE 2026-02-26T03:03:34.880 DEBUG:teuthology.orchestra.remote:trial197:/tmp/tmp.CABgfl5CgE is 173KB 2026-02-26T03:03:34.934 DEBUG:teuthology.orchestra.run.trial197:> rm -fr /tmp/tmp.CABgfl5CgE 2026-02-26T03:03:34.949 INFO:tasks.ceph:Cleaning ceph cluster... 2026-02-26T03:03:34.949 DEBUG:teuthology.orchestra.run.trial197:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2026-02-26T03:03:35.070 INFO:teuthology.util.scanner:summary_data or yaml_file is empty! 2026-02-26T03:03:35.070 INFO:tasks.ceph:Archiving crash dumps... 2026-02-26T03:03:35.073 DEBUG:teuthology.misc:Transferring archived files from trial197:/var/lib/ceph/crash to /home/teuthworker/mnt/teuthology/yuriw-2026-02-26_01:51:16-rados-wip-yuri9-testing-2026-02-25-1600-squid-distro-default-trial/71117/remote/trial197/crash 2026-02-26T03:03:35.074 DEBUG:teuthology.orchestra.run.trial197:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2026-02-26T03:03:35.134 INFO:tasks.ceph:Compressing logs... 2026-02-26T03:03:35.134 DEBUG:teuthology.orchestra.run.trial197:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2026-02-26T03:03:35.198 INFO:teuthology.orchestra.run.trial197.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48013.log 2026-02-26T03:03:35.198 INFO:teuthology.orchestra.run.trial197.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.2.log 2026-02-26T03:03:35.198 INFO:teuthology.orchestra.run.trial197.stderr:/var/log/ceph/ceph-client.admin.48013.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47554.log 2026-02-26T03:03:35.198 INFO:teuthology.orchestra.run.trial197.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48013.log.gz 2026-02-26T03:03:35.198 INFO:teuthology.orchestra.run.trial197.stderr:/var/log/ceph/ceph-osd.2.log: gzip -5 --verbose -- /var/log/ceph/ceph-mon.a.log 2026-02-26T03:03:35.199 INFO:teuthology.orchestra.run.trial197.stderr:gzip/var/log/ceph/ceph-client.admin.47554.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.48365.log 2026-02-26T03:03:35.199 INFO:teuthology.orchestra.run.trial197.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47554.log.gz 2026-02-26T03:03:35.199 INFO:teuthology.orchestra.run.trial197.stderr:/var/log/ceph/ceph-mon.a.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59675.log 2026-02-26T03:03:35.199 INFO:teuthology.orchestra.run.trial197.stderr:/var/log/ceph/ceph-client.admin.48365.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49949.log 2026-02-26T03:03:35.199 INFO:teuthology.orchestra.run.trial197.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48365.log.gz 2026-02-26T03:03:35.199 INFO:teuthology.orchestra.run.trial197.stderr:/var/log/ceph/ceph-client.admin.59675.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.59675.log.gz 2026-02-26T03:03:35.199 INFO:teuthology.orchestra.run.trial197.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49773.log 2026-02-26T03:03:35.200 INFO:teuthology.orchestra.run.trial197.stderr:/var/log/ceph/ceph-client.admin.49949.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60013.log 2026-02-26T03:03:35.200 INFO:teuthology.orchestra.run.trial197.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49949.log.gz 2026-02-26T03:03:35.200 INFO:teuthology.orchestra.run.trial197.stderr:/var/log/ceph/ceph-client.admin.49773.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49773.log.gz 2026-02-26T03:03:35.200 INFO:teuthology.orchestra.run.trial197.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56516.log 2026-02-26T03:03:35.200 INFO:teuthology.orchestra.run.trial197.stderr:/var/log/ceph/ceph-client.admin.60013.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59631.log 2026-02-26T03:03:35.200 INFO:teuthology.orchestra.run.trial197.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.60013.log.gz 2026-02-26T03:03:35.201 INFO:teuthology.orchestra.run.trial197.stderr:/var/log/ceph/ceph-client.admin.56516.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.56516.log.gz 2026-02-26T03:03:35.201 INFO:teuthology.orchestra.run.trial197.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.0.log 2026-02-26T03:03:35.201 INFO:teuthology.orchestra.run.trial197.stderr:/var/log/ceph/ceph-client.admin.59631.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50125.log 2026-02-26T03:03:35.201 INFO:teuthology.orchestra.run.trial197.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.59631.log.gz 2026-02-26T03:03:35.201 INFO:teuthology.orchestra.run.trial197.stderr:/var/log/ceph/ceph-osd.0.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52237.log 2026-02-26T03:03:35.202 INFO:teuthology.orchestra.run.trial197.stderr:/var/log/ceph/ceph-client.admin.50125.log: gzip 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50125.log.gz -5 2026-02-26T03:03:35.202 INFO:teuthology.orchestra.run.trial197.stderr: --verbose -- /var/log/ceph/ceph-client.admin.47276.log 2026-02-26T03:03:35.202 INFO:teuthology.orchestra.run.trial197.stderr:/var/log/ceph/ceph-client.admin.52237.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52237.log.gz 2026-02-26T03:03:35.202 INFO:teuthology.orchestra.run.trial197.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50521.log 2026-02-26T03:03:35.202 INFO:teuthology.orchestra.run.trial197.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46979.log 2026-02-26T03:03:35.202 INFO:teuthology.orchestra.run.trial197.stderr:/var/log/ceph/ceph-client.admin.47276.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47276.log.gz 2026-02-26T03:03:35.202 INFO:teuthology.orchestra.run.trial197.stderr:/var/log/ceph/ceph-client.admin.50521.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50521.log.gz 2026-02-26T03:03:35.202 INFO:teuthology.orchestra.run.trial197.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51225.log 2026-02-26T03:03:35.203 INFO:teuthology.orchestra.run.trial197.stderr:/var/log/ceph/ceph-client.admin.46979.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46979.log.gz 2026-02-26T03:03:35.203 INFO:teuthology.orchestra.run.trial197.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59455.log 2026-02-26T03:03:35.203 INFO:teuthology.orchestra.run.trial197.stderr:gzip/var/log/ceph/ceph-client.admin.51225.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.50257.log 2026-02-26T03:03:35.203 INFO:teuthology.orchestra.run.trial197.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51225.log.gz 2026-02-26T03:03:35.203 INFO:teuthology.orchestra.run.trial197.stderr:/var/log/ceph/ceph-client.admin.59455.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.59455.log.gz 2026-02-26T03:03:35.203 INFO:teuthology.orchestra.run.trial197.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55342.log 2026-02-26T03:03:35.204 INFO:teuthology.orchestra.run.trial197.stderr:/var/log/ceph/ceph-client.admin.50257.log: gzip -5 --verbose -- /var/log/ceph/objectstore_tool.56632.log 2026-02-26T03:03:35.204 INFO:teuthology.orchestra.run.trial197.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50257.log.gz 2026-02-26T03:03:35.204 INFO:teuthology.orchestra.run.trial197.stderr:/var/log/ceph/ceph-client.admin.55342.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49069.log 2026-02-26T03:03:35.204 INFO:teuthology.orchestra.run.trial197.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.55342.log.gz 2026-02-26T03:03:35.204 INFO:teuthology.orchestra.run.trial197.stderr:/var/log/ceph/objectstore_tool.56632.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53157.log 2026-02-26T03:03:35.204 INFO:teuthology.orchestra.run.trial197.stderr:/var/log/ceph/ceph-client.admin.49069.log: 0.0%gzip -- replaced with /var/log/ceph/ceph-client.admin.49069.log.gz 2026-02-26T03:03:35.204 INFO:teuthology.orchestra.run.trial197.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.42630.log 2026-02-26T03:03:35.205 INFO:teuthology.orchestra.run.trial197.stderr:/var/log/ceph/ceph-client.admin.53157.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47603.log 2026-02-26T03:03:35.205 INFO:teuthology.orchestra.run.trial197.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53157.log.gz 2026-02-26T03:03:35.205 INFO:teuthology.orchestra.run.trial197.stderr:/var/log/ceph/ceph-client.admin.42630.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50829.log 2026-02-26T03:03:35.205 INFO:teuthology.orchestra.run.trial197.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42630.log.gz 2026-02-26T03:03:35.205 INFO:teuthology.orchestra.run.trial197.stderr:/var/log/ceph/ceph-client.admin.47603.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56565.log 2026-02-26T03:03:35.205 INFO:teuthology.orchestra.run.trial197.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47603.log.gz 2026-02-26T03:03:35.206 INFO:teuthology.orchestra.run.trial197.stderr:/var/log/ceph/ceph-client.admin.50829.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50829.log.gz 2026-02-26T03:03:35.206 INFO:teuthology.orchestra.run.trial197.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60354.log 2026-02-26T03:03:35.215 INFO:teuthology.orchestra.run.trial197.stderr:/var/log/ceph/ceph-client.admin.56565.log: gzip 0.0% -5 --verbose -- replaced with /var/log/ceph/ceph-client.admin.56565.log.gz -- 2026-02-26T03:03:35.215 INFO:teuthology.orchestra.run.trial197.stderr: /var/log/ceph/ceph-client.admin.48805.log 2026-02-26T03:03:35.215 INFO:teuthology.orchestra.run.trial197.stderr:/var/log/ceph/ceph-client.admin.60354.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.60354.log.gz 2026-02-26T03:03:35.215 INFO:teuthology.orchestra.run.trial197.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42773.log 2026-02-26T03:03:35.215 INFO:teuthology.orchestra.run.trial197.stderr:/var/log/ceph/ceph-client.admin.48805.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48805.log.gz 2026-02-26T03:03:35.215 INFO:teuthology.orchestra.run.trial197.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46333.log 2026-02-26T03:03:35.215 INFO:teuthology.orchestra.run.trial197.stderr:/var/log/ceph/ceph-client.admin.42773.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42773.log.gz 2026-02-26T03:03:35.215 INFO:teuthology.orchestra.run.trial197.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49993.log 2026-02-26T03:03:35.215 INFO:teuthology.orchestra.run.trial197.stderr:/var/log/ceph/ceph-client.admin.46333.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46333.log.gz 2026-02-26T03:03:35.215 INFO:teuthology.orchestra.run.trial197.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50081.log 2026-02-26T03:03:35.216 INFO:teuthology.orchestra.run.trial197.stderr:/var/log/ceph/ceph-client.admin.49993.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49993.log.gz 2026-02-26T03:03:35.216 INFO:teuthology.orchestra.run.trial197.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47456.log 2026-02-26T03:03:35.216 INFO:teuthology.orchestra.run.trial197.stderr:/var/log/ceph/ceph-client.admin.50081.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50081.log.gz 2026-02-26T03:03:35.216 INFO:teuthology.orchestra.run.trial197.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55101.log 2026-02-26T03:03:35.216 INFO:teuthology.orchestra.run.trial197.stderr:/var/log/ceph/ceph-client.admin.47456.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47456.log.gz 2026-02-26T03:03:35.216 INFO:teuthology.orchestra.run.trial197.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48761.log 2026-02-26T03:03:35.216 INFO:teuthology.orchestra.run.trial197.stderr:/var/log/ceph/ceph-client.admin.55101.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.55101.log.gz 2026-02-26T03:03:35.216 INFO:teuthology.orchestra.run.trial197.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49641.log 2026-02-26T03:03:35.216 INFO:teuthology.orchestra.run.trial197.stderr:/var/log/ceph/ceph-client.admin.48761.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48761.log.gz 2026-02-26T03:03:35.216 INFO:teuthology.orchestra.run.trial197.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49905.log 2026-02-26T03:03:35.216 INFO:teuthology.orchestra.run.trial197.stderr:/var/log/ceph/ceph-client.admin.49641.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49641.log.gz 2026-02-26T03:03:35.216 INFO:teuthology.orchestra.run.trial197.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47358.log 2026-02-26T03:03:35.216 INFO:teuthology.orchestra.run.trial197.stderr:/var/log/ceph/ceph-client.admin.49905.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49905.log.gz 2026-02-26T03:03:35.216 INFO:teuthology.orchestra.run.trial197.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49421.log 2026-02-26T03:03:35.216 INFO:teuthology.orchestra.run.trial197.stderr:/var/log/ceph/ceph-client.admin.47358.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47358.log.gz 2026-02-26T03:03:35.216 INFO:teuthology.orchestra.run.trial197.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47505.log 2026-02-26T03:03:35.217 INFO:teuthology.orchestra.run.trial197.stderr:/var/log/ceph/ceph-client.admin.49421.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49421.log.gz 2026-02-26T03:03:35.217 INFO:teuthology.orchestra.run.trial197.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59323.log 2026-02-26T03:03:35.217 INFO:teuthology.orchestra.run.trial197.stderr:/var/log/ceph/ceph-client.admin.47505.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47505.log.gz 2026-02-26T03:03:35.217 INFO:teuthology.orchestra.run.trial197.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49817.log 2026-02-26T03:03:35.217 INFO:teuthology.orchestra.run.trial197.stderr:/var/log/ceph/ceph-client.admin.59323.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.59323.log.gz 2026-02-26T03:03:35.217 INFO:teuthology.orchestra.run.trial197.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52281.log 2026-02-26T03:03:35.217 INFO:teuthology.orchestra.run.trial197.stderr:/var/log/ceph/ceph-client.admin.49817.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49817.log.gz 2026-02-26T03:03:35.217 INFO:teuthology.orchestra.run.trial197.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50961.log 2026-02-26T03:03:35.217 INFO:teuthology.orchestra.run.trial197.stderr:/var/log/ceph/ceph-client.admin.52281.log: gzip -5 28.9% --verbose -- replaced with /var/log/ceph/ceph-client.admin.52281.log.gz -- 2026-02-26T03:03:35.217 INFO:teuthology.orchestra.run.trial197.stderr: /var/log/ceph/ceph-client.admin.46332.log 2026-02-26T03:03:35.217 INFO:teuthology.orchestra.run.trial197.stderr:/var/log/ceph/ceph-client.admin.50961.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50961.log.gz 2026-02-26T03:03:35.217 INFO:teuthology.orchestra.run.trial197.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55150.log 2026-02-26T03:03:35.217 INFO:teuthology.orchestra.run.trial197.stderr:/var/log/ceph/ceph-client.admin.46332.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46332.log.gz 2026-02-26T03:03:35.217 INFO:teuthology.orchestra.run.trial197.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46822.log 2026-02-26T03:03:35.217 INFO:teuthology.orchestra.run.trial197.stderr:/var/log/ceph/ceph-client.admin.55150.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.55150.log.gz 2026-02-26T03:03:35.217 INFO:teuthology.orchestra.run.trial197.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59207.log 2026-02-26T03:03:35.217 INFO:teuthology.orchestra.run.trial197.stderr:/var/log/ceph/ceph-client.admin.46822.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46822.log.gz 2026-02-26T03:03:35.218 INFO:teuthology.orchestra.run.trial197.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59817.log 2026-02-26T03:03:35.218 INFO:teuthology.orchestra.run.trial197.stderr:/var/log/ceph/ceph-client.admin.59207.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.59207.log.gz 2026-02-26T03:03:35.218 INFO:teuthology.orchestra.run.trial197.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46574.log 2026-02-26T03:03:35.218 INFO:teuthology.orchestra.run.trial197.stderr:/var/log/ceph/ceph-client.admin.59817.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.59817.log.gz 2026-02-26T03:03:35.218 INFO:teuthology.orchestra.run.trial197.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60111.log 2026-02-26T03:03:35.218 INFO:teuthology.orchestra.run.trial197.stderr:/var/log/ceph/ceph-client.admin.46574.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46574.log.gz 2026-02-26T03:03:35.218 INFO:teuthology.orchestra.run.trial197.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50477.log 2026-02-26T03:03:35.218 INFO:teuthology.orchestra.run.trial197.stderr:/var/log/ceph/ceph-client.admin.60111.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.60111.log.gz 2026-02-26T03:03:35.218 INFO:teuthology.orchestra.run.trial197.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48849.log 2026-02-26T03:03:35.218 INFO:teuthology.orchestra.run.trial197.stderr:/var/log/ceph/ceph-client.admin.50477.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50477.log.gz 2026-02-26T03:03:35.218 INFO:teuthology.orchestra.run.trial197.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46675.log 2026-02-26T03:03:35.218 INFO:teuthology.orchestra.run.trial197.stderr:/var/log/ceph/ceph-client.admin.48849.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48849.log.gz 2026-02-26T03:03:35.218 INFO:teuthology.orchestra.run.trial197.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46334.log 2026-02-26T03:03:35.218 INFO:teuthology.orchestra.run.trial197.stderr:/var/log/ceph/ceph-client.admin.46675.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46675.log.gz 2026-02-26T03:03:35.218 INFO:teuthology.orchestra.run.trial197.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60160.log 2026-02-26T03:03:35.218 INFO:teuthology.orchestra.run.trial197.stderr:/var/log/ceph/ceph-client.admin.46334.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46334.log.gz 2026-02-26T03:03:35.218 INFO:teuthology.orchestra.run.trial197.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51181.log 2026-02-26T03:03:35.218 INFO:teuthology.orchestra.run.trial197.stderr:/var/log/ceph/ceph-client.admin.60160.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55293.log 2026-02-26T03:03:35.219 INFO:teuthology.orchestra.run.trial197.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.60160.log.gz 2026-02-26T03:03:35.219 INFO:teuthology.orchestra.run.trial197.stderr:/var/log/ceph/ceph-client.admin.51181.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51181.log.gz 2026-02-26T03:03:35.219 INFO:teuthology.orchestra.run.trial197.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49861.log 2026-02-26T03:03:35.219 INFO:teuthology.orchestra.run.trial197.stderr:/var/log/ceph/ceph-client.admin.55293.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48541.log 2026-02-26T03:03:35.219 INFO:teuthology.orchestra.run.trial197.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.55293.log.gz 2026-02-26T03:03:35.219 INFO:teuthology.orchestra.run.trial197.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52653.log 2026-02-26T03:03:35.219 INFO:teuthology.orchestra.run.trial197.stderr:/var/log/ceph/ceph-client.admin.49861.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49861.log.gz 2026-02-26T03:03:35.219 INFO:teuthology.orchestra.run.trial197.stderr:/var/log/ceph/ceph-client.admin.48541.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48541.log.gz 2026-02-26T03:03:35.219 INFO:teuthology.orchestra.run.trial197.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59543.log 2026-02-26T03:03:35.232 INFO:teuthology.orchestra.run.trial197.stderr:/var/log/ceph/ceph-client.admin.52653.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52653.log.gz 2026-02-26T03:03:35.232 INFO:teuthology.orchestra.run.trial197.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51137.log 2026-02-26T03:03:35.232 INFO:teuthology.orchestra.run.trial197.stderr:/var/log/ceph/ceph-client.admin.59543.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.59543.log.gz 2026-02-26T03:03:35.232 INFO:teuthology.orchestra.run.trial197.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46824.log 2026-02-26T03:03:35.232 INFO:teuthology.orchestra.run.trial197.stderr:/var/log/ceph/ceph-client.admin.51137.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51137.log.gz 2026-02-26T03:03:35.232 INFO:teuthology.orchestra.run.trial197.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48189.log 2026-02-26T03:03:35.232 INFO:teuthology.orchestra.run.trial197.stderr:/var/log/ceph/ceph-client.admin.46824.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46824.log.gz 2026-02-26T03:03:35.232 INFO:teuthology.orchestra.run.trial197.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51753.log 2026-02-26T03:03:35.232 INFO:teuthology.orchestra.run.trial197.stderr:/var/log/ceph/ceph-client.admin.48189.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48189.log.gz 2026-02-26T03:03:35.232 INFO:teuthology.orchestra.run.trial197.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51885.log 2026-02-26T03:03:35.232 INFO:teuthology.orchestra.run.trial197.stderr:/var/log/ceph/ceph-client.admin.51753.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51753.log.gz 2026-02-26T03:03:35.232 INFO:teuthology.orchestra.run.trial197.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59964.log 2026-02-26T03:03:35.232 INFO:teuthology.orchestra.run.trial197.stderr:/var/log/ceph/ceph-client.admin.51885.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51885.log.gz 2026-02-26T03:03:35.233 INFO:teuthology.orchestra.run.trial197.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46981.log 2026-02-26T03:03:35.233 INFO:teuthology.orchestra.run.trial197.stderr:/var/log/ceph/ceph-client.admin.59964.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.59964.log.gz 2026-02-26T03:03:35.233 INFO:teuthology.orchestra.run.trial197.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51489.log 2026-02-26T03:03:35.233 INFO:teuthology.orchestra.run.trial197.stderr:/var/log/ceph/ceph-client.admin.46981.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46981.log.gz 2026-02-26T03:03:35.233 INFO:teuthology.orchestra.run.trial197.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46039.log 2026-02-26T03:03:35.233 INFO:teuthology.orchestra.run.trial197.stderr:/var/log/ceph/ceph-client.admin.51489.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51489.log.gz 2026-02-26T03:03:35.233 INFO:teuthology.orchestra.run.trial197.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45888.log 2026-02-26T03:03:35.233 INFO:teuthology.orchestra.run.trial197.stderr:/var/log/ceph/ceph-client.admin.46039.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46039.log.gz 2026-02-26T03:03:35.233 INFO:teuthology.orchestra.run.trial197.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59367.log 2026-02-26T03:03:35.233 INFO:teuthology.orchestra.run.trial197.stderr:/var/log/ceph/ceph-client.admin.45888.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45888.log.gz 2026-02-26T03:03:35.233 INFO:teuthology.orchestra.run.trial197.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52609.log 2026-02-26T03:03:35.233 INFO:teuthology.orchestra.run.trial197.stderr:/var/log/ceph/ceph-client.admin.59367.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.59367.log.gz 2026-02-26T03:03:35.233 INFO:teuthology.orchestra.run.trial197.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51533.log 2026-02-26T03:03:35.233 INFO:teuthology.orchestra.run.trial197.stderr:/var/log/ceph/ceph-client.admin.52609.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52609.log.gz 2026-02-26T03:03:35.233 INFO:teuthology.orchestra.run.trial197.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42820.log 2026-02-26T03:03:35.233 INFO:teuthology.orchestra.run.trial197.stderr:/var/log/ceph/ceph-client.admin.51533.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51533.log.gz 2026-02-26T03:03:35.233 INFO:teuthology.orchestra.run.trial197.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45937.log 2026-02-26T03:03:35.233 INFO:teuthology.orchestra.run.trial197.stderr:/var/log/ceph/ceph-client.admin.42820.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42820.log.gz 2026-02-26T03:03:35.233 INFO:teuthology.orchestra.run.trial197.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51005.log 2026-02-26T03:03:35.233 INFO:teuthology.orchestra.run.trial197.stderr:/var/log/ceph/ceph-client.admin.45937.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45937.log.gz 2026-02-26T03:03:35.233 INFO:teuthology.orchestra.run.trial197.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51973.log 2026-02-26T03:03:35.234 INFO:teuthology.orchestra.run.trial197.stderr:/var/log/ceph/ceph-client.admin.51005.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51005.log.gz 2026-02-26T03:03:35.234 INFO:teuthology.orchestra.run.trial197.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48277.log 2026-02-26T03:03:35.234 INFO:teuthology.orchestra.run.trial197.stderr: 93.8% -- replaced with /var/log/ceph/objectstore_tool.56632.log.gz 2026-02-26T03:03:35.234 INFO:teuthology.orchestra.run.trial197.stderr:/var/log/ceph/ceph-client.admin.51973.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51973.log.gz 2026-02-26T03:03:35.234 INFO:teuthology.orchestra.run.trial197.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50785.log 2026-02-26T03:03:35.234 INFO:teuthology.orchestra.run.trial197.stderr:/var/log/ceph/ceph-client.admin.48277.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48277.log.gz 2026-02-26T03:03:35.234 INFO:teuthology.orchestra.run.trial197.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59158.log 2026-02-26T03:03:35.234 INFO:teuthology.orchestra.run.trial197.stderr:/var/log/ceph/ceph-client.admin.50785.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50785.log.gz 2026-02-26T03:03:35.234 INFO:teuthology.orchestra.run.trial197.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49201.log 2026-02-26T03:03:35.234 INFO:teuthology.orchestra.run.trial197.stderr:/var/log/ceph/ceph-client.admin.59158.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.59158.log.gz 2026-02-26T03:03:35.234 INFO:teuthology.orchestra.run.trial197.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53059.log 2026-02-26T03:03:35.234 INFO:teuthology.orchestra.run.trial197.stderr:/var/log/ceph/ceph-client.admin.49201.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49201.log.gz 2026-02-26T03:03:35.234 INFO:teuthology.orchestra.run.trial197.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51357.log 2026-02-26T03:03:35.234 INFO:teuthology.orchestra.run.trial197.stderr:/var/log/ceph/ceph-client.admin.53059.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53059.log.gz 2026-02-26T03:03:35.234 INFO:teuthology.orchestra.run.trial197.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53010.log 2026-02-26T03:03:35.234 INFO:teuthology.orchestra.run.trial197.stderr:/var/log/ceph/ceph-client.admin.51357.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51357.log.gz 2026-02-26T03:03:35.234 INFO:teuthology.orchestra.run.trial197.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46917.log 2026-02-26T03:03:35.234 INFO:teuthology.orchestra.run.trial197.stderr:/var/log/ceph/ceph-client.admin.53010.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53010.log.gz 2026-02-26T03:03:35.234 INFO:teuthology.orchestra.run.trial197.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49289.log 2026-02-26T03:03:35.234 INFO:teuthology.orchestra.run.trial197.stderr:/var/log/ceph/ceph-client.admin.46917.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46917.log.gz 2026-02-26T03:03:35.234 INFO:teuthology.orchestra.run.trial197.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59915.log 2026-02-26T03:03:35.234 INFO:teuthology.orchestra.run.trial197.stderr:/var/log/ceph/ceph-client.admin.49289.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49289.log.gz 2026-02-26T03:03:35.235 INFO:teuthology.orchestra.run.trial197.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59279.log 2026-02-26T03:03:35.235 INFO:teuthology.orchestra.run.trial197.stderr:/var/log/ceph/ceph-client.admin.59915.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.59915.log.gz 2026-02-26T03:03:35.235 INFO:teuthology.orchestra.run.trial197.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50169.log 2026-02-26T03:03:35.235 INFO:teuthology.orchestra.run.trial197.stderr:/var/log/ceph/ceph-client.admin.59279.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.59279.log.gz 2026-02-26T03:03:35.235 INFO:teuthology.orchestra.run.trial197.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48409.log 2026-02-26T03:03:35.235 INFO:teuthology.orchestra.run.trial197.stderr:/var/log/ceph/ceph-client.admin.50169.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50169.log.gz 2026-02-26T03:03:35.235 INFO:teuthology.orchestra.run.trial197.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49245.log 2026-02-26T03:03:35.235 INFO:teuthology.orchestra.run.trial197.stderr:/var/log/ceph/ceph-client.admin.48409.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48409.log.gz 2026-02-26T03:03:35.235 INFO:teuthology.orchestra.run.trial197.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47407.log 2026-02-26T03:03:35.235 INFO:teuthology.orchestra.run.trial197.stderr:/var/log/ceph/ceph-client.admin.49245.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49245.log.gz 2026-02-26T03:03:35.235 INFO:teuthology.orchestra.run.trial197.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50433.log 2026-02-26T03:03:35.235 INFO:teuthology.orchestra.run.trial197.stderr:/var/log/ceph/ceph-client.admin.47407.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47407.log.gz 2026-02-26T03:03:35.235 INFO:teuthology.orchestra.run.trial197.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42517.log 2026-02-26T03:03:35.235 INFO:teuthology.orchestra.run.trial197.stderr:/var/log/ceph/ceph-client.admin.50433.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50433.log.gz 2026-02-26T03:03:35.235 INFO:teuthology.orchestra.run.trial197.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51797.log 2026-02-26T03:03:35.235 INFO:teuthology.orchestra.run.trial197.stderr:/var/log/ceph/ceph-client.admin.42517.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42517.log.gz 2026-02-26T03:03:35.235 INFO:teuthology.orchestra.run.trial197.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46185.log 2026-02-26T03:03:35.235 INFO:teuthology.orchestra.run.trial197.stderr:/var/log/ceph/ceph-client.admin.51797.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51797.log.gz 2026-02-26T03:03:35.235 INFO:teuthology.orchestra.run.trial197.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49025.log 2026-02-26T03:03:35.236 INFO:teuthology.orchestra.run.trial197.stderr:/var/log/ceph/ceph-client.admin.46185.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46185.log.gz 2026-02-26T03:03:35.236 INFO:teuthology.orchestra.run.trial197.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49685.log 2026-02-26T03:03:35.236 INFO:teuthology.orchestra.run.trial197.stderr:/var/log/ceph/ceph-client.admin.49025.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49025.log.gz 2026-02-26T03:03:35.236 INFO:teuthology.orchestra.run.trial197.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52697.log 2026-02-26T03:03:35.236 INFO:teuthology.orchestra.run.trial197.stderr:/var/log/ceph/ceph-client.admin.49685.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52325.log 2026-02-26T03:03:35.236 INFO:teuthology.orchestra.run.trial197.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49685.log.gz 2026-02-26T03:03:35.236 INFO:teuthology.orchestra.run.trial197.stderr:/var/log/ceph/ceph-client.admin.52697.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52697.log.gz 2026-02-26T03:03:35.236 INFO:teuthology.orchestra.run.trial197.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47274.log 2026-02-26T03:03:35.236 INFO:teuthology.orchestra.run.trial197.stderr:/var/log/ceph/ceph-client.admin.52325.log: 27.3% -- replaced with /var/log/ceph/ceph-client.admin.52325.log.gz 2026-02-26T03:03:35.236 INFO:teuthology.orchestra.run.trial197.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50213.log 2026-02-26T03:03:35.236 INFO:teuthology.orchestra.run.trial197.stderr:/var/log/ceph/ceph-client.admin.47274.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47274.log.gz 2026-02-26T03:03:35.236 INFO:teuthology.orchestra.run.trial197.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52785.log 2026-02-26T03:03:35.236 INFO:teuthology.orchestra.run.trial197.stderr:/var/log/ceph/ceph-client.admin.50213.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50213.log.gz 2026-02-26T03:03:35.236 INFO:teuthology.orchestra.run.trial197.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.tmp-client.admin.37896.log 2026-02-26T03:03:35.236 INFO:teuthology.orchestra.run.trial197.stderr:/var/log/ceph/ceph-client.admin.52785.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52785.log.gz 2026-02-26T03:03:35.236 INFO:teuthology.orchestra.run.trial197.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53206.log 2026-02-26T03:03:35.236 INFO:teuthology.orchestra.run.trial197.stderr:/var/log/ceph/ceph.tmp-client.admin.37896.log: 0.0% -- replaced with /var/log/ceph/ceph.tmp-client.admin.37896.log.gz 2026-02-26T03:03:35.236 INFO:teuthology.orchestra.run.trial197.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56467.log 2026-02-26T03:03:35.236 INFO:teuthology.orchestra.run.trial197.stderr:/var/log/ceph/ceph-client.admin.53206.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53206.log.gz 2026-02-26T03:03:35.236 INFO:teuthology.orchestra.run.trial197.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48233.log 2026-02-26T03:03:35.236 INFO:teuthology.orchestra.run.trial197.stderr:/var/log/ceph/ceph-client.admin.56467.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.56467.log.gz 2026-02-26T03:03:35.236 INFO:teuthology.orchestra.run.trial197.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52017.log 2026-02-26T03:03:35.237 INFO:teuthology.orchestra.run.trial197.stderr:/var/log/ceph/ceph-client.admin.48233.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48233.log.gz 2026-02-26T03:03:35.237 INFO:teuthology.orchestra.run.trial197.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42994.log 2026-02-26T03:03:35.237 INFO:teuthology.orchestra.run.trial197.stderr:/var/log/ceph/ceph-client.admin.52017.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52017.log.gz 2026-02-26T03:03:35.237 INFO:teuthology.orchestra.run.trial197.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46677.log 2026-02-26T03:03:35.237 INFO:teuthology.orchestra.run.trial197.stderr:/var/log/ceph/ceph-client.admin.42994.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42994.log.gz 2026-02-26T03:03:35.237 INFO:teuthology.orchestra.run.trial197.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46187.log 2026-02-26T03:03:35.237 INFO:teuthology.orchestra.run.trial197.stderr:/var/log/ceph/ceph-client.admin.46677.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46677.log.gz 2026-02-26T03:03:35.237 INFO:teuthology.orchestra.run.trial197.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45679.log 2026-02-26T03:03:35.237 INFO:teuthology.orchestra.run.trial197.stderr:/var/log/ceph/ceph-client.admin.46187.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46187.log.gz 2026-02-26T03:03:35.237 INFO:teuthology.orchestra.run.trial197.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51709.log 2026-02-26T03:03:35.237 INFO:teuthology.orchestra.run.trial197.stderr:/var/log/ceph/ceph-client.admin.45679.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45679.log.gz 2026-02-26T03:03:35.237 INFO:teuthology.orchestra.run.trial197.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47127.log 2026-02-26T03:03:35.237 INFO:teuthology.orchestra.run.trial197.stderr:/var/log/ceph/ceph-client.admin.51709.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51709.log.gz 2026-02-26T03:03:35.237 INFO:teuthology.orchestra.run.trial197.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53108.log 2026-02-26T03:03:35.237 INFO:teuthology.orchestra.run.trial197.stderr:/var/log/ceph/ceph-client.admin.47127.log: gzip 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47127.log.gz -5 2026-02-26T03:03:35.237 INFO:teuthology.orchestra.run.trial197.stderr: --verbose -- /var/log/ceph/ceph-client.admin.51665.log 2026-02-26T03:03:35.237 INFO:teuthology.orchestra.run.trial197.stderr:/var/log/ceph/ceph-client.admin.53108.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53108.log.gz 2026-02-26T03:03:35.237 INFO:teuthology.orchestra.run.trial197.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52105.log 2026-02-26T03:03:35.237 INFO:teuthology.orchestra.run.trial197.stderr:/var/log/ceph/ceph-client.admin.51665.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51665.log.gz 2026-02-26T03:03:35.237 INFO:teuthology.orchestra.run.trial197.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55494.log 2026-02-26T03:03:35.237 INFO:teuthology.orchestra.run.trial197.stderr:/var/log/ceph/ceph-client.admin.52105.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52105.log.gz 2026-02-26T03:03:35.237 INFO:teuthology.orchestra.run.trial197.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50389.log 2026-02-26T03:03:35.237 INFO:teuthology.orchestra.run.trial197.stderr:/var/log/ceph/ceph-client.admin.55494.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.55494.log.gz 2026-02-26T03:03:35.238 INFO:teuthology.orchestra.run.trial197.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42867.log 2026-02-26T03:03:35.238 INFO:teuthology.orchestra.run.trial197.stderr:/var/log/ceph/ceph-client.admin.50389.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50389.log.gz 2026-02-26T03:03:35.238 INFO:teuthology.orchestra.run.trial197.stderr:gzip -5 --verbose -- /var/log/ceph/objectstore_tool.57406.log 2026-02-26T03:03:35.238 INFO:teuthology.orchestra.run.trial197.stderr:/var/log/ceph/ceph-client.admin.42867.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42867.log.gz 2026-02-26T03:03:35.238 INFO:teuthology.orchestra.run.trial197.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48981.log 2026-02-26T03:03:35.238 INFO:teuthology.orchestra.run.trial197.stderr:/var/log/ceph/objectstore_tool.57406.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46038.log 2026-02-26T03:03:35.238 INFO:teuthology.orchestra.run.trial197.stderr:/var/log/ceph/ceph-client.admin.48981.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48981.log.gz 2026-02-26T03:03:35.238 INFO:teuthology.orchestra.run.trial197.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52897.log 2026-02-26T03:03:35.238 INFO:teuthology.orchestra.run.trial197.stderr:/var/log/ceph/ceph-client.admin.46038.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46038.log.gz 2026-02-26T03:03:35.238 INFO:teuthology.orchestra.run.trial197.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51401.log 2026-02-26T03:03:35.238 INFO:teuthology.orchestra.run.trial197.stderr:/var/log/ceph/ceph-client.admin.52897.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52897.log.gz 2026-02-26T03:03:35.238 INFO:teuthology.orchestra.run.trial197.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51445.log 2026-02-26T03:03:35.238 INFO:teuthology.orchestra.run.trial197.stderr:/var/log/ceph/ceph-client.admin.51401.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51401.log.gz 2026-02-26T03:03:35.238 INFO:teuthology.orchestra.run.trial197.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47897.log 2026-02-26T03:03:35.238 INFO:teuthology.orchestra.run.trial197.stderr:/var/log/ceph/ceph-client.admin.51445.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51445.log.gz 2026-02-26T03:03:35.238 INFO:teuthology.orchestra.run.trial197.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52829.log 2026-02-26T03:03:35.238 INFO:teuthology.orchestra.run.trial197.stderr:/var/log/ceph/ceph-client.admin.47897.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47897.log.gz 2026-02-26T03:03:35.238 INFO:teuthology.orchestra.run.trial197.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43058.log 2026-02-26T03:03:35.238 INFO:teuthology.orchestra.run.trial197.stderr:/var/log/ceph/ceph-client.admin.52829.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52829.log.gz 2026-02-26T03:03:35.238 INFO:teuthology.orchestra.run.trial197.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50697.log 2026-02-26T03:03:35.246 INFO:teuthology.orchestra.run.trial197.stderr:/var/log/ceph/ceph-client.admin.43058.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43058.log.gz 2026-02-26T03:03:35.246 INFO:teuthology.orchestra.run.trial197.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2026-02-26T03:03:35.246 INFO:teuthology.orchestra.run.trial197.stderr:/var/log/ceph/ceph-client.admin.50697.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55391.log 2026-02-26T03:03:35.246 INFO:teuthology.orchestra.run.trial197.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50697.log.gz 2026-02-26T03:03:35.246 INFO:teuthology.orchestra.run.trial197.stderr:/var/log/ceph/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45613.log 2026-02-26T03:03:35.246 INFO:teuthology.orchestra.run.trial197.stderr:/var/log/ceph/ceph-client.admin.55391.log: 88.3% -- replaced with /var/log/ceph/ceph.audit.log.gz 2026-02-26T03:03:35.246 INFO:teuthology.orchestra.run.trial197.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.55391.log.gz 2026-02-26T03:03:35.246 INFO:teuthology.orchestra.run.trial197.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59411.log 2026-02-26T03:03:35.246 INFO:teuthology.orchestra.run.trial197.stderr:/var/log/ceph/ceph-client.admin.45613.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45613.log.gz 2026-02-26T03:03:35.246 INFO:teuthology.orchestra.run.trial197.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50741.log 2026-02-26T03:03:35.246 INFO:teuthology.orchestra.run.trial197.stderr:/var/log/ceph/ceph-client.admin.59411.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.59411.log.gz 2026-02-26T03:03:35.246 INFO:teuthology.orchestra.run.trial197.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59256.log 2026-02-26T03:03:35.246 INFO:teuthology.orchestra.run.trial197.stderr:/var/log/ceph/ceph-client.admin.50741.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50741.log.gz 2026-02-26T03:03:35.246 INFO:teuthology.orchestra.run.trial197.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50873.log 2026-02-26T03:03:35.246 INFO:teuthology.orchestra.run.trial197.stderr:/var/log/ceph/ceph-client.admin.59256.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.59256.log.gz 2026-02-26T03:03:35.246 INFO:teuthology.orchestra.run.trial197.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48101.log 2026-02-26T03:03:35.246 INFO:teuthology.orchestra.run.trial197.stderr:/var/log/ceph/ceph-client.admin.50873.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50873.log.gz 2026-02-26T03:03:35.246 INFO:teuthology.orchestra.run.trial197.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49509.log 2026-02-26T03:03:35.246 INFO:teuthology.orchestra.run.trial197.stderr:/var/log/ceph/ceph-client.admin.48101.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48101.log.gz 2026-02-26T03:03:35.246 INFO:teuthology.orchestra.run.trial197.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.1.log 2026-02-26T03:03:35.246 INFO:teuthology.orchestra.run.trial197.stderr:/var/log/ceph/ceph-client.admin.49509.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49509.log.gz 2026-02-26T03:03:35.247 INFO:teuthology.orchestra.run.trial197.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49377.log 2026-02-26T03:03:35.247 INFO:teuthology.orchestra.run.trial197.stderr:/var/log/ceph/ceph-osd.1.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52521.log 2026-02-26T03:03:35.247 INFO:teuthology.orchestra.run.trial197.stderr:/var/log/ceph/ceph-client.admin.49377.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49377.log.gz 2026-02-26T03:03:35.247 INFO:teuthology.orchestra.run.trial197.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50345.log 2026-02-26T03:03:35.247 INFO:teuthology.orchestra.run.trial197.stderr:/var/log/ceph/ceph-client.admin.52521.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52521.log.gz 2026-02-26T03:03:35.247 INFO:teuthology.orchestra.run.trial197.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52565.log 2026-02-26T03:03:35.247 INFO:teuthology.orchestra.run.trial197.stderr:/var/log/ceph/ceph-client.admin.50345.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50345.log.gz 2026-02-26T03:03:35.247 INFO:teuthology.orchestra.run.trial197.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49465.log 2026-02-26T03:03:35.247 INFO:teuthology.orchestra.run.trial197.stderr:/var/log/ceph/ceph-client.admin.52565.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52565.log.gz 2026-02-26T03:03:35.247 INFO:teuthology.orchestra.run.trial197.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47969.log 2026-02-26T03:03:35.247 INFO:teuthology.orchestra.run.trial197.stderr:/var/log/ceph/ceph-client.admin.49465.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49465.log.gz 2026-02-26T03:03:35.247 INFO:teuthology.orchestra.run.trial197.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55244.log 2026-02-26T03:03:35.247 INFO:teuthology.orchestra.run.trial197.stderr:/var/log/ceph/ceph-client.admin.47969.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47969.log.gz 2026-02-26T03:03:35.247 INFO:teuthology.orchestra.run.trial197.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55173.log 2026-02-26T03:03:35.247 INFO:teuthology.orchestra.run.trial197.stderr:/var/log/ceph/ceph-client.admin.55244.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.55244.log.gz 2026-02-26T03:03:35.247 INFO:teuthology.orchestra.run.trial197.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47064.log 2026-02-26T03:03:35.247 INFO:teuthology.orchestra.run.trial197.stderr:/var/log/ceph/ceph-client.admin.55173.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.55173.log.gz 2026-02-26T03:03:35.247 INFO:teuthology.orchestra.run.trial197.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60209.log 2026-02-26T03:03:35.247 INFO:teuthology.orchestra.run.trial197.stderr:/var/log/ceph/ceph-client.admin.47064.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47064.log.gz 2026-02-26T03:03:35.247 INFO:teuthology.orchestra.run.trial197.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50609.log 2026-02-26T03:03:35.247 INFO:teuthology.orchestra.run.trial197.stderr:/var/log/ceph/ceph-client.admin.60209.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.60209.log.gz 2026-02-26T03:03:35.247 INFO:teuthology.orchestra.run.trial197.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50301.log 2026-02-26T03:03:35.247 INFO:teuthology.orchestra.run.trial197.stderr:/var/log/ceph/ceph-client.admin.50609.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50609.log.gz 2026-02-26T03:03:35.247 INFO:teuthology.orchestra.run.trial197.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48893.log 2026-02-26T03:03:35.247 INFO:teuthology.orchestra.run.trial197.stderr:/var/log/ceph/ceph-client.admin.50301.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48057.log 2026-02-26T03:03:35.247 INFO:teuthology.orchestra.run.trial197.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50301.log.gz 2026-02-26T03:03:35.247 INFO:teuthology.orchestra.run.trial197.stderr:/var/log/ceph/ceph-client.admin.48893.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48893.log.gz 2026-02-26T03:03:35.247 INFO:teuthology.orchestra.run.trial197.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46823.log 2026-02-26T03:03:35.248 INFO:teuthology.orchestra.run.trial197.stderr:/var/log/ceph/ceph-client.admin.48057.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48057.log.gz 2026-02-26T03:03:35.248 INFO:teuthology.orchestra.run.trial197.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50653.log 2026-02-26T03:03:35.248 INFO:teuthology.orchestra.run.trial197.stderr:/var/log/ceph/ceph-client.admin.46823.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46823.log.gz 2026-02-26T03:03:35.248 INFO:teuthology.orchestra.run.trial197.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48497.log 2026-02-26T03:03:35.248 INFO:teuthology.orchestra.run.trial197.stderr:/var/log/ceph/ceph-client.admin.50653.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50653.log.gz 2026-02-26T03:03:35.248 INFO:teuthology.orchestra.run.trial197.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49553.log 2026-02-26T03:03:35.248 INFO:teuthology.orchestra.run.trial197.stderr:/var/log/ceph/ceph-client.admin.48497.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48497.log.gz 2026-02-26T03:03:35.248 INFO:teuthology.orchestra.run.trial197.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46186.log 2026-02-26T03:03:35.248 INFO:teuthology.orchestra.run.trial197.stderr:/var/log/ceph/ceph-client.admin.49553.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49553.log.gz 2026-02-26T03:03:35.248 INFO:teuthology.orchestra.run.trial197.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50037.log 2026-02-26T03:03:35.248 INFO:teuthology.orchestra.run.trial197.stderr:/var/log/ceph/ceph-client.admin.46186.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46186.log.gz 2026-02-26T03:03:35.248 INFO:teuthology.orchestra.run.trial197.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46040.log 2026-02-26T03:03:35.248 INFO:teuthology.orchestra.run.trial197.stderr:/var/log/ceph/ceph-client.admin.50037.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50037.log.gz 2026-02-26T03:03:35.248 INFO:teuthology.orchestra.run.trial197.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52061.log 2026-02-26T03:03:35.248 INFO:teuthology.orchestra.run.trial197.stderr:/var/log/ceph/ceph-client.admin.46040.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46040.log.gz 2026-02-26T03:03:35.248 INFO:teuthology.orchestra.run.trial197.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47128.log 2026-02-26T03:03:35.248 INFO:teuthology.orchestra.run.trial197.stderr:/var/log/ceph/ceph-client.admin.52061.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52061.log.gz 2026-02-26T03:03:35.248 INFO:teuthology.orchestra.run.trial197.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59866.log 2026-02-26T03:03:35.248 INFO:teuthology.orchestra.run.trial197.stderr:/var/log/ceph/ceph-client.admin.47128.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47128.log.gz 2026-02-26T03:03:35.248 INFO:teuthology.orchestra.run.trial197.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49113.log 2026-02-26T03:03:35.248 INFO:teuthology.orchestra.run.trial197.stderr:/var/log/ceph/ceph-client.admin.59866.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.59866.log.gz 2026-02-26T03:03:35.248 INFO:teuthology.orchestra.run.trial197.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48673.log 2026-02-26T03:03:35.248 INFO:teuthology.orchestra.run.trial197.stderr:/var/log/ceph/ceph-client.admin.49113.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49113.log.gz 2026-02-26T03:03:35.248 INFO:teuthology.orchestra.run.trial197.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45749.log 2026-02-26T03:03:35.248 INFO:teuthology.orchestra.run.trial197.stderr:/var/log/ceph/ceph-client.admin.48673.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48673.log.gz 2026-02-26T03:03:35.248 INFO:teuthology.orchestra.run.trial197.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51049.log 2026-02-26T03:03:35.248 INFO:teuthology.orchestra.run.trial197.stderr:/var/log/ceph/ceph-client.admin.45749.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45749.log.gz 2026-02-26T03:03:35.248 INFO:teuthology.orchestra.run.trial197.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52395.log 2026-02-26T03:03:35.248 INFO:teuthology.orchestra.run.trial197.stderr:/var/log/ceph/ceph-client.admin.51049.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51049.log.gz 2026-02-26T03:03:35.248 INFO:teuthology.orchestra.run.trial197.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59499.log 2026-02-26T03:03:35.249 INFO:teuthology.orchestra.run.trial197.stderr:/var/log/ceph/ceph-client.admin.52395.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52395.log.gz 2026-02-26T03:03:35.249 INFO:teuthology.orchestra.run.trial197.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48937.log 2026-02-26T03:03:35.249 INFO:teuthology.orchestra.run.trial197.stderr:/var/log/ceph/ceph-client.admin.59499.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.59499.log.gz 2026-02-26T03:03:35.249 INFO:teuthology.orchestra.run.trial197.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49597.log 2026-02-26T03:03:35.249 INFO:teuthology.orchestra.run.trial197.stderr:/var/log/ceph/ceph-client.admin.48937.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48937.log.gz 2026-02-26T03:03:35.249 INFO:teuthology.orchestra.run.trial197.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46476.log 2026-02-26T03:03:35.249 INFO:teuthology.orchestra.run.trial197.stderr:/var/log/ceph/ceph-client.admin.49597.log: 93.9% -- replaced with /var/log/ceph/objectstore_tool.57406.log.gz 2026-02-26T03:03:35.249 INFO:teuthology.orchestra.run.trial197.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49597.log.gz 2026-02-26T03:03:35.249 INFO:teuthology.orchestra.run.trial197.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48585.log 2026-02-26T03:03:35.249 INFO:teuthology.orchestra.run.trial197.stderr:/var/log/ceph/ceph-client.admin.46476.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46476.log.gz 2026-02-26T03:03:35.249 INFO:teuthology.orchestra.run.trial197.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46525.log 2026-02-26T03:03:35.249 INFO:teuthology.orchestra.run.trial197.stderr:/var/log/ceph/ceph-client.admin.48585.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48585.log.gz 2026-02-26T03:03:35.249 INFO:teuthology.orchestra.run.trial197.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49333.log 2026-02-26T03:03:35.249 INFO:teuthology.orchestra.run.trial197.stderr:/var/log/ceph/ceph-client.admin.46525.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46525.log.gz 2026-02-26T03:03:35.249 INFO:teuthology.orchestra.run.trial197.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47946.log 2026-02-26T03:03:35.249 INFO:teuthology.orchestra.run.trial197.stderr:/var/log/ceph/ceph-client.admin.49333.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49333.log.gz 2026-02-26T03:03:35.249 INFO:teuthology.orchestra.run.trial197.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51841.log 2026-02-26T03:03:35.249 INFO:teuthology.orchestra.run.trial197.stderr:/var/log/ceph/ceph-client.admin.47946.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47946.log.gz 2026-02-26T03:03:35.249 INFO:teuthology.orchestra.run.trial197.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mgr.x.log 2026-02-26T03:03:35.249 INFO:teuthology.orchestra.run.trial197.stderr:/var/log/ceph/ceph-client.admin.51841.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51841.log.gz 2026-02-26T03:03:35.249 INFO:teuthology.orchestra.run.trial197.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51929.log 2026-02-26T03:03:35.249 INFO:teuthology.orchestra.run.trial197.stderr:/var/log/ceph/ceph-mgr.x.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51093.log 2026-02-26T03:03:35.249 INFO:teuthology.orchestra.run.trial197.stderr:/var/log/ceph/ceph-client.admin.51929.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51929.log.gz 2026-02-26T03:03:35.249 INFO:teuthology.orchestra.run.trial197.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49729.log 2026-02-26T03:03:35.249 INFO:teuthology.orchestra.run.trial197.stderr:/var/log/ceph/ceph-client.admin.51093.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51093.log.gz 2026-02-26T03:03:35.249 INFO:teuthology.orchestra.run.trial197.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46676.log 2026-02-26T03:03:35.253 INFO:teuthology.orchestra.run.trial197.stderr:/var/log/ceph/ceph-client.admin.49729.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49729.log.gz 2026-02-26T03:03:35.253 INFO:teuthology.orchestra.run.trial197.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52149.log 2026-02-26T03:03:35.253 INFO:teuthology.orchestra.run.trial197.stderr:/var/log/ceph/ceph-client.admin.46676.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46676.log.gz 2026-02-26T03:03:35.253 INFO:teuthology.orchestra.run.trial197.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52444.log 2026-02-26T03:03:35.253 INFO:teuthology.orchestra.run.trial197.stderr:/var/log/ceph/ceph-client.admin.52149.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52149.log.gz 2026-02-26T03:03:35.253 INFO:teuthology.orchestra.run.trial197.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45774.log 2026-02-26T03:03:35.253 INFO:teuthology.orchestra.run.trial197.stderr:/var/log/ceph/ceph-client.admin.52444.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52444.log.gz 2026-02-26T03:03:35.253 INFO:teuthology.orchestra.run.trial197.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48717.log 2026-02-26T03:03:35.253 INFO:teuthology.orchestra.run.trial197.stderr:/var/log/ceph/ceph-client.admin.45774.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45774.log.gz 2026-02-26T03:03:35.253 INFO:teuthology.orchestra.run.trial197.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51313.log 2026-02-26T03:03:35.253 INFO:teuthology.orchestra.run.trial197.stderr:/var/log/ceph/ceph-client.admin.48717.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48717.log.gz 2026-02-26T03:03:35.253 INFO:teuthology.orchestra.run.trial197.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52741.log 2026-02-26T03:03:35.253 INFO:teuthology.orchestra.run.trial197.stderr:/var/log/ceph/ceph-client.admin.51313.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51313.log.gz 2026-02-26T03:03:35.253 INFO:teuthology.orchestra.run.trial197.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56588.log 2026-02-26T03:03:35.253 INFO:teuthology.orchestra.run.trial197.stderr:/var/log/ceph/ceph-client.admin.52741.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52741.log.gz 2026-02-26T03:03:35.253 INFO:teuthology.orchestra.run.trial197.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47211.log 2026-02-26T03:03:35.253 INFO:teuthology.orchestra.run.trial197.stderr:/var/log/ceph/ceph-client.admin.56588.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.56588.log.gz 2026-02-26T03:03:35.253 INFO:teuthology.orchestra.run.trial197.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48629.log 2026-02-26T03:03:35.253 INFO:teuthology.orchestra.run.trial197.stderr:/var/log/ceph/ceph-client.admin.47211.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47211.log.gz 2026-02-26T03:03:35.253 INFO:teuthology.orchestra.run.trial197.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50565.log 2026-02-26T03:03:35.253 INFO:teuthology.orchestra.run.trial197.stderr:/var/log/ceph/ceph-client.admin.48629.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48629.log.gz 2026-02-26T03:03:35.253 INFO:teuthology.orchestra.run.trial197.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.log 2026-02-26T03:03:35.253 INFO:teuthology.orchestra.run.trial197.stderr:/var/log/ceph/ceph-client.admin.50565.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50565.log.gz 2026-02-26T03:03:35.253 INFO:teuthology.orchestra.run.trial197.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48145.log 2026-02-26T03:03:35.253 INFO:teuthology.orchestra.run.trial197.stderr:/var/log/ceph/ceph.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60258.log 2026-02-26T03:03:35.253 INFO:teuthology.orchestra.run.trial197.stderr:/var/log/ceph/ceph-client.admin.48145.log: 84.6% -- replaced with /var/log/ceph/ceph.log.gz 2026-02-26T03:03:35.253 INFO:teuthology.orchestra.run.trial197.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48145.log.gz 2026-02-26T03:03:35.254 INFO:teuthology.orchestra.run.trial197.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60062.log 2026-02-26T03:03:35.254 INFO:teuthology.orchestra.run.trial197.stderr:/var/log/ceph/ceph-client.admin.60258.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.60258.log.gz 2026-02-26T03:03:35.254 INFO:teuthology.orchestra.run.trial197.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50917.log 2026-02-26T03:03:35.254 INFO:teuthology.orchestra.run.trial197.stderr:/var/log/ceph/ceph-client.admin.60062.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.60062.log.gz 2026-02-26T03:03:35.254 INFO:teuthology.orchestra.run.trial197.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51269.log 2026-02-26T03:03:35.254 INFO:teuthology.orchestra.run.trial197.stderr:/var/log/ceph/ceph-client.admin.50917.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50917.log.gz 2026-02-26T03:03:35.254 INFO:teuthology.orchestra.run.trial197.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49157.log 2026-02-26T03:03:35.254 INFO:teuthology.orchestra.run.trial197.stderr:/var/log/ceph/ceph-client.admin.51269.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51269.log.gz 2026-02-26T03:03:35.254 INFO:teuthology.orchestra.run.trial197.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48321.log 2026-02-26T03:03:35.254 INFO:teuthology.orchestra.run.trial197.stderr:/var/log/ceph/ceph-client.admin.49157.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49157.log.gz 2026-02-26T03:03:35.254 INFO:teuthology.orchestra.run.trial197.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52193.log 2026-02-26T03:03:35.254 INFO:teuthology.orchestra.run.trial197.stderr:/var/log/ceph/ceph-client.admin.48321.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48321.log.gz 2026-02-26T03:03:35.254 INFO:teuthology.orchestra.run.trial197.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48453.log 2026-02-26T03:03:35.254 INFO:teuthology.orchestra.run.trial197.stderr:/var/log/ceph/ceph-client.admin.52193.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52193.log.gz 2026-02-26T03:03:35.254 INFO:teuthology.orchestra.run.trial197.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52853.log 2026-02-26T03:03:35.254 INFO:teuthology.orchestra.run.trial197.stderr:/var/log/ceph/ceph-client.admin.48453.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48453.log.gz 2026-02-26T03:03:35.254 INFO:teuthology.orchestra.run.trial197.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59768.log 2026-02-26T03:03:35.254 INFO:teuthology.orchestra.run.trial197.stderr:/var/log/ceph/ceph-client.admin.52853.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52853.log.gz 2026-02-26T03:03:35.254 INFO:teuthology.orchestra.run.trial197.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51621.log 2026-02-26T03:03:35.254 INFO:teuthology.orchestra.run.trial197.stderr:/var/log/ceph/ceph-client.admin.59768.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.59768.log.gz 2026-02-26T03:03:35.254 INFO:teuthology.orchestra.run.trial197.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59587.log 2026-02-26T03:03:35.254 INFO:teuthology.orchestra.run.trial197.stderr:/var/log/ceph/ceph-client.admin.51621.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51621.log.gz 2026-02-26T03:03:35.254 INFO:teuthology.orchestra.run.trial197.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51577.log 2026-02-26T03:03:35.254 INFO:teuthology.orchestra.run.trial197.stderr:/var/log/ceph/ceph-client.admin.59587.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.59587.log.gz 2026-02-26T03:03:35.254 INFO:teuthology.orchestra.run.trial197.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60307.log 2026-02-26T03:03:35.254 INFO:teuthology.orchestra.run.trial197.stderr:/var/log/ceph/ceph-client.admin.51577.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51577.log.gz 2026-02-26T03:03:35.254 INFO:teuthology.orchestra.run.trial197.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45839.log 2026-02-26T03:03:35.254 INFO:teuthology.orchestra.run.trial197.stderr:/var/log/ceph/ceph-client.admin.60307.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.60307.log.gz 2026-02-26T03:03:35.254 INFO:teuthology.orchestra.run.trial197.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46427.log 2026-02-26T03:03:35.254 INFO:teuthology.orchestra.run.trial197.stderr:/var/log/ceph/ceph-client.admin.45839.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45839.log.gz 2026-02-26T03:03:35.255 INFO:teuthology.orchestra.run.trial197.stderr:/var/log/ceph/ceph-client.admin.46427.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46427.log.gz 2026-02-26T03:03:35.259 INFO:teuthology.orchestra.run.trial197.stderr: 91.4% -- replaced with /var/log/ceph/ceph-mgr.x.log.gz 2026-02-26T03:03:35.261 INFO:teuthology.orchestra.run.trial197.stderr: 91.4% -- replaced with /var/log/ceph/ceph-mon.a.log.gz 2026-02-26T03:03:35.317 INFO:teuthology.orchestra.run.trial197.stderr: 92.2% -- replaced with /var/log/ceph/ceph-osd.2.log.gz 2026-02-26T03:03:35.502 INFO:teuthology.orchestra.run.trial197.stderr: 93.5% -- replaced with /var/log/ceph/ceph-osd.0.log.gz 2026-02-26T03:03:35.573 INFO:teuthology.orchestra.run.trial197.stderr: 93.5% -- replaced with /var/log/ceph/ceph-osd.1.log.gz 2026-02-26T03:03:35.574 INFO:teuthology.orchestra.run.trial197.stderr: 2026-02-26T03:03:35.574 INFO:teuthology.orchestra.run.trial197.stderr:real 0m0.385s 2026-02-26T03:03:35.574 INFO:teuthology.orchestra.run.trial197.stderr:user 0m0.866s 2026-02-26T03:03:35.574 INFO:teuthology.orchestra.run.trial197.stderr:sys 0m0.132s 2026-02-26T03:03:35.575 INFO:tasks.ceph:Archiving logs... 2026-02-26T03:03:35.575 DEBUG:teuthology.misc:Transferring archived files from trial197:/var/log/ceph to /home/teuthworker/mnt/teuthology/yuriw-2026-02-26_01:51:16-rados-wip-yuri9-testing-2026-02-25-1600-squid-distro-default-trial/71117/remote/trial197/log 2026-02-26T03:03:35.576 DEBUG:teuthology.orchestra.run.trial197:> sudo tar c -f - -C /var/log/ceph -- . 2026-02-26T03:03:35.991 DEBUG:teuthology.run_tasks:Unwinding manager install 2026-02-26T03:03:36.005 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2026-02-26T03:03:36.005 DEBUG:teuthology.orchestra.run.trial197:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2026-02-26T03:03:36.095 DEBUG:teuthology.run_tasks:Unwinding manager clock 2026-02-26T03:03:36.111 INFO:teuthology.task.clock:Checking final clock skew... 2026-02-26T03:03:36.111 DEBUG:teuthology.orchestra.run.trial197:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2026-02-26T03:03:36.127 INFO:teuthology.orchestra.run.trial197.stderr:bash: line 1: ntpq: command not found 2026-02-26T03:03:36.176 INFO:teuthology.orchestra.run.trial197.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2026-02-26T03:03:36.176 INFO:teuthology.orchestra.run.trial197.stdout:=============================================================================== 2026-02-26T03:03:36.176 INFO:teuthology.orchestra.run.trial197.stdout:^- 108.61.73.243 2 6 177 35 +265us[ +78us] +/- 31ms 2026-02-26T03:03:36.176 INFO:teuthology.orchestra.run.trial197.stdout:^- ntp.shastacoe.net 2 6 177 35 +102us[ +102us] +/- 74ms 2026-02-26T03:03:36.176 INFO:teuthology.orchestra.run.trial197.stdout:^* time.cloudflare.com 3 6 177 35 -63us[ -250us] +/- 6775us 2026-02-26T03:03:36.176 INFO:teuthology.orchestra.run.trial197.stdout:^- 149.28.200.179.vultruser> 2 6 177 37 +562us[ +377us] +/- 41ms 2026-02-26T03:03:36.177 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2026-02-26T03:03:36.189 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2026-02-26T03:03:36.189 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2026-02-26T03:03:36.204 DEBUG:teuthology.orchestra.run.trial197:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2026-02-26T03:03:36.272 DEBUG:teuthology.orchestra.run.trial197:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:getty_t:s0 2026-02-26T03:03:36.298 DEBUG:teuthology.orchestra.run:got remote process result: 1 2026-02-26T03:03:36.299 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2026-02-26T03:03:36.315 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2026-02-26T03:03:36.333 INFO:teuthology.task.internal:Duration was 400.628744 seconds 2026-02-26T03:03:36.334 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2026-02-26T03:03:36.350 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2026-02-26T03:03:36.351 DEBUG:teuthology.orchestra.run.trial197:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2026-02-26T03:03:36.387 INFO:teuthology.orchestra.run.trial197.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2026-02-26T03:03:36.726 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2026-02-26T03:03:36.726 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@trial197.front.sepia.ceph.com 2026-02-26T03:03:36.726 DEBUG:teuthology.orchestra.run.trial197:> grep -E --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | grep -E -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | grep -E -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2026-02-26T03:03:36.784 INFO:teuthology.task.internal.syslog:Gathering journactl... 2026-02-26T03:03:36.784 DEBUG:teuthology.orchestra.run.trial197:> sudo journalctl > /home/ubuntu/cephtest/archive/syslog/journalctl.log 2026-02-26T03:03:36.886 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2026-02-26T03:03:36.886 DEBUG:teuthology.orchestra.run.trial197:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --max-args=1 --max-procs=0 --verbose --no-run-if-empty -- gzip -5 --verbose -- 2026-02-26T03:03:36.910 INFO:teuthology.orchestra.run.trial197.stderr:gzip -5 --verbose -- /home/ubuntu/cephtest/archive/syslog/misc.log 2026-02-26T03:03:36.911 INFO:teuthology.orchestra.run.trial197.stderr:gzip -5 --verbose -- /home/ubuntu/cephtest/archive/syslog/kern.log 2026-02-26T03:03:36.911 INFO:teuthology.orchestra.run.trial197.stderr:/home/ubuntu/cephtest/archive/syslog/misc.log: gzip -5 --verbose -- /home/ubuntu/cephtest/archive/syslog/journalctl.log 2026-02-26T03:03:36.911 INFO:teuthology.orchestra.run.trial197.stderr: 0.0% -- replaced with /home/ubuntu/cephtest/archive/syslog/misc.log.gz 2026-02-26T03:03:36.911 INFO:teuthology.orchestra.run.trial197.stderr:/home/ubuntu/cephtest/archive/syslog/kern.log: 0.0% -- replaced with /home/ubuntu/cephtest/archive/syslog/kern.log.gz 2026-02-26T03:03:36.920 INFO:teuthology.orchestra.run.trial197.stderr:/home/ubuntu/cephtest/archive/syslog/journalctl.log: 81.9% -- replaced with /home/ubuntu/cephtest/archive/syslog/journalctl.log.gz 2026-02-26T03:03:36.922 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2026-02-26T03:03:36.978 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2026-02-26T03:03:36.979 DEBUG:teuthology.orchestra.run.trial197:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2026-02-26T03:03:37.002 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2026-02-26T03:03:37.014 DEBUG:teuthology.orchestra.run.trial197:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2026-02-26T03:03:37.065 INFO:teuthology.orchestra.run.trial197.stdout:kernel.core_pattern = core 2026-02-26T03:03:37.084 DEBUG:teuthology.orchestra.run.trial197:> test -e /home/ubuntu/cephtest/archive/coredump 2026-02-26T03:03:37.139 DEBUG:teuthology.orchestra.run:got remote process result: 1 2026-02-26T03:03:37.139 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2026-02-26T03:03:37.157 INFO:teuthology.task.internal:Transferring archived files... 2026-02-26T03:03:37.158 DEBUG:teuthology.misc:Transferring archived files from trial197:/home/ubuntu/cephtest/archive to /home/teuthworker/mnt/teuthology/yuriw-2026-02-26_01:51:16-rados-wip-yuri9-testing-2026-02-25-1600-squid-distro-default-trial/71117/remote/trial197 2026-02-26T03:03:37.158 DEBUG:teuthology.orchestra.run.trial197:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2026-02-26T03:03:37.215 INFO:teuthology.task.internal:Removing archive directory... 2026-02-26T03:03:37.215 DEBUG:teuthology.orchestra.run.trial197:> rm -rf -- /home/ubuntu/cephtest/archive 2026-02-26T03:03:37.264 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2026-02-26T03:03:37.278 INFO:teuthology.task.internal:Not uploading archives. 2026-02-26T03:03:37.278 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2026-02-26T03:03:37.292 INFO:teuthology.task.internal:Tidying up after the test... 2026-02-26T03:03:37.292 DEBUG:teuthology.orchestra.run.trial197:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2026-02-26T03:03:37.319 INFO:teuthology.orchestra.run.trial197.stdout: 920834 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Feb 26 03:03 /home/ubuntu/cephtest 2026-02-26T03:03:37.320 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2026-02-26T03:03:37.332 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2026-02-26T03:03:37.359 INFO:teuthology.run:Summary data: description: rados/singleton/{all/divergent_priors2 mon_election/classic msgr-failures/few msgr/async objectstore/bluestore-comp-snappy rados supported-random-distro$/{centos_latest}} duration: 400.62874364852905 flavor: default owner: scheduled_yuriw@soko04.front.sepia.ceph.com success: true 2026-02-26T03:03:37.359 DEBUG:teuthology.report:Pushing job info to https://paddles-paddles.apps.pok.os.sepia.ceph.com 2026-02-26T03:03:37.458 INFO:teuthology.run:pass