2026-02-22T22:30:34.122 INFO:root:teuthology version: 1.2.4.dev9+gfa17720d0 2026-02-22T22:30:34.137 DEBUG:teuthology.report:Pushing job info to https://paddles-paddles.apps.pok.os.sepia.ceph.com 2026-02-22T22:30:34.230 INFO:teuthology.run:Config: archive_path: /home/teuthworker/mnt/teuthology/yuriw-2026-02-22_19:35:40-rados-wip-yuri6-testing-2026-02-22-1629-squid-distro-default-trial/64252 branch: wip-yuri6-testing-2026-02-22-1629-squid description: rados/singleton/{all/rebuild-mondb mon_election/connectivity msgr-failures/few msgr/async objectstore/bluestore-comp-zstd rados supported-random-distro$/{ubuntu_latest}} email: yweinste@redhat.com first_in_suite: false flavor: default job_id: '64252' kernel: branch: distro kdb: true sha1: distro ktype: distro last_in_suite: false machine_type: trial name: yuriw-2026-02-22_19:35:40-rados-wip-yuri6-testing-2026-02-22-1629-squid-distro-default-trial no_nested_subset: false openstack: - volumes: count: 3 size: 10 os_type: ubuntu os_version: '22.04' overrides: admin_socket: branch: wip-yuri6-testing-2026-02-22-1629-squid ceph: conf: global: mon client directed command retry: 5 mon election default strategy: 3 ms inject socket failures: 5000 ms type: async mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon scrub interval: 300 osd: bluestore block size: 96636764160 bluestore compression algorithm: zstd bluestore compression mode: aggressive bluestore fsck on mount: true bluestore zero block detection: true debug bluefs: 20 debug bluestore: 20 debug ms: 1 debug osd: 20 debug rocksdb: 10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd debug verify cached snaps: true osd debug verify missing on start: true osd failsafe full ratio: 0.95 osd mclock override recovery settings: true osd mclock profile: high_recovery_ops osd mclock skip benchmark: true osd objectstore: bluestore osd op queue: debug_random osd op queue cut off: debug_random flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - \(OSD_SLOW_PING_TIME - \(MON_DOWN\) sha1: f90f4eaebb28b619cf76e80cbbe997656da69579 ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} install: ceph: flavor: default sha1: f90f4eaebb28b619cf76e80cbbe997656da69579 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: wip-yuri6-testing-2026-02-22-1629-squid sha1: f90f4eaebb28b619cf76e80cbbe997656da69579 owner: scheduled_yuriw@soko04.front.sepia.ceph.com priority: 69 repo: https://github.com/ceph/ceph-ci.git roles: - - mon.a - mon.b - mon.c - mgr.x - osd.0 - osd.1 - osd.2 - client.0 seed: 5907 sha1: f90f4eaebb28b619cf76e80cbbe997656da69579 sleep_before_teardown: 0 subset: 111/120000 suite: rados suite_branch: wip-yuri6-testing-2026-02-22-1629-squid suite_path: /home/teuthworker/src/github.com_ceph_ceph-c_f90f4eaebb28b619cf76e80cbbe997656da69579/qa suite_relpath: qa suite_repo: https://github.com/ceph/ceph-ci.git suite_sha1: f90f4eaebb28b619cf76e80cbbe997656da69579 targets: trial023.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBFMzy9ICbBYbIp1CetICBj9of3PshaRAUhq0K0Ih30IB4Xeslfz5yFV6XioYr4lsJZDpn6s07UkGKI6BFd5jdIY= tasks: - install: null - ceph: conf: mon: debug auth: 30 log-ignorelist: - no reply from - overall HEALTH_ - \(MON_DOWN\) - \(MGR_DOWN\) - \(OSDMAP_FLAGS\) - \(OSD_ - \(PG_ - \(POOL_APP_NOT_ENABLED\) pre-mgr-commands: - sudo ceph config set mgr mgr_pool false --force - full_sequential: - radosbench: clients: - client.0 time: 30 - rebuild_mondb: null - radosbench: clients: - client.0 time: 30 teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: fa17720d0088c3ac28e473468bfc79eeaff5cd38 timestamp: 2026-02-22_19:35:40 tube: trial user: yuriw verbose: true worker_log: /home/teuthworker/mnt/teuthology/worker_logs/dispatcher.trial.3318653 2026-02-22T22:30:34.230 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/github.com_ceph_ceph-c_f90f4eaebb28b619cf76e80cbbe997656da69579/qa; will attempt to use it 2026-02-22T22:30:34.231 INFO:teuthology.run:Found tasks at /home/teuthworker/src/github.com_ceph_ceph-c_f90f4eaebb28b619cf76e80cbbe997656da69579/qa/tasks 2026-02-22T22:30:34.231 INFO:teuthology.run_tasks:Running task internal.check_packages... 2026-02-22T22:30:34.232 INFO:teuthology.task.internal:Checking packages... 2026-02-22T22:30:34.232 INFO:teuthology.task.internal:Checking packages for os_type 'ubuntu', flavor 'default' and ceph hash 'f90f4eaebb28b619cf76e80cbbe997656da69579' 2026-02-22T22:30:34.233 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2026-02-22T22:30:34.233 INFO:teuthology.packaging:ref: None 2026-02-22T22:30:34.233 INFO:teuthology.packaging:tag: None 2026-02-22T22:30:34.233 INFO:teuthology.packaging:branch: wip-yuri6-testing-2026-02-22-1629-squid 2026-02-22T22:30:34.233 INFO:teuthology.packaging:sha1: f90f4eaebb28b619cf76e80cbbe997656da69579 2026-02-22T22:30:34.233 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&ref=wip-yuri6-testing-2026-02-22-1629-squid 2026-02-22T22:30:34.352 INFO:teuthology.task.internal:Found packages for ceph version 19.2.3-689-gf90f4eae-1jammy 2026-02-22T22:30:34.354 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2026-02-22T22:30:34.368 INFO:teuthology.task.internal:no buildpackages task found 2026-02-22T22:30:34.368 INFO:teuthology.run_tasks:Running task internal.save_config... 2026-02-22T22:30:34.383 INFO:teuthology.task.internal:Saving configuration 2026-02-22T22:30:34.392 INFO:teuthology.run_tasks:Running task internal.check_lock... 2026-02-22T22:30:34.405 INFO:teuthology.task.internal.check_lock:Checking locks... 2026-02-22T22:30:34.488 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'trial023.front.sepia.ceph.com', 'description': '/home/teuthworker/mnt/teuthology/yuriw-2026-02-22_19:35:40-rados-wip-yuri6-testing-2026-02-22-1629-squid-distro-default-trial/64252', 'up': True, 'machine_type': 'trial', 'is_vm': False, 'vm_host': None, 'os_type': 'ubuntu', 'os_version': '22.04', 'arch': 'x86_64', 'locked': True, 'locked_since': '2026-02-22 22:27:02.235538', 'locked_by': 'scheduled_yuriw@soko04.front.sepia.ceph.com', 'mac_address': None, 'ssh_pub_key': 'ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIPE3eqDicTpBpGWAFQavsEZ9E3QYqwUKEj+ujGJztAyj'} 2026-02-22T22:30:34.488 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2026-02-22T22:30:34.499 INFO:teuthology.task.internal:roles: ubuntu@trial023.front.sepia.ceph.com - ['mon.a', 'mon.b', 'mon.c', 'mgr.x', 'osd.0', 'osd.1', 'osd.2', 'client.0'] 2026-02-22T22:30:34.499 INFO:teuthology.run_tasks:Running task console_log... 2026-02-22T22:30:34.824 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7899305fb560>, signals=[15]) 2026-02-22T22:30:34.824 INFO:teuthology.run_tasks:Running task internal.connect... 2026-02-22T22:30:34.837 INFO:teuthology.task.internal:Opening connections... 2026-02-22T22:30:34.837 DEBUG:teuthology.task.internal:connecting to ubuntu@trial023.front.sepia.ceph.com 2026-02-22T22:30:34.838 DEBUG:teuthology.orchestra.connection:{'hostname': 'trial023.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2026-02-22T22:30:34.910 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2026-02-22T22:30:34.922 DEBUG:teuthology.orchestra.run.trial023:> uname -m 2026-02-22T22:30:34.987 INFO:teuthology.orchestra.run.trial023.stdout:x86_64 2026-02-22T22:30:34.988 DEBUG:teuthology.orchestra.run.trial023:> cat /etc/os-release 2026-02-22T22:30:35.032 INFO:teuthology.orchestra.run.trial023.stdout:PRETTY_NAME="Ubuntu 22.04.5 LTS" 2026-02-22T22:30:35.032 INFO:teuthology.orchestra.run.trial023.stdout:NAME="Ubuntu" 2026-02-22T22:30:35.032 INFO:teuthology.orchestra.run.trial023.stdout:VERSION_ID="22.04" 2026-02-22T22:30:35.032 INFO:teuthology.orchestra.run.trial023.stdout:VERSION="22.04.5 LTS (Jammy Jellyfish)" 2026-02-22T22:30:35.032 INFO:teuthology.orchestra.run.trial023.stdout:VERSION_CODENAME=jammy 2026-02-22T22:30:35.032 INFO:teuthology.orchestra.run.trial023.stdout:ID=ubuntu 2026-02-22T22:30:35.032 INFO:teuthology.orchestra.run.trial023.stdout:ID_LIKE=debian 2026-02-22T22:30:35.032 INFO:teuthology.orchestra.run.trial023.stdout:HOME_URL="https://www.ubuntu.com/" 2026-02-22T22:30:35.032 INFO:teuthology.orchestra.run.trial023.stdout:SUPPORT_URL="https://help.ubuntu.com/" 2026-02-22T22:30:35.032 INFO:teuthology.orchestra.run.trial023.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" 2026-02-22T22:30:35.032 INFO:teuthology.orchestra.run.trial023.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" 2026-02-22T22:30:35.032 INFO:teuthology.orchestra.run.trial023.stdout:UBUNTU_CODENAME=jammy 2026-02-22T22:30:35.033 INFO:teuthology.lock.ops:Updating trial023.front.sepia.ceph.com on lock server 2026-02-22T22:30:35.244 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2026-02-22T22:30:35.257 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2026-02-22T22:30:35.268 INFO:teuthology.task.internal:Checking for old test directory... 2026-02-22T22:30:35.268 DEBUG:teuthology.orchestra.run.trial023:> test '!' -e /home/ubuntu/cephtest 2026-02-22T22:30:35.272 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2026-02-22T22:30:35.282 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2026-02-22T22:30:35.282 DEBUG:teuthology.orchestra.run.trial023:> test -z $(ls -A /var/lib/ceph) 2026-02-22T22:30:35.320 INFO:teuthology.orchestra.run.trial023.stderr:ls: cannot access '/var/lib/ceph': No such file or directory 2026-02-22T22:30:35.320 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2026-02-22T22:30:35.406 INFO:teuthology.run_tasks:Running task kernel... 2026-02-22T22:30:35.426 INFO:teuthology.task.kernel:normalize config orig: {'branch': 'distro', 'kdb': True, 'sha1': 'distro'} 2026-02-22T22:30:35.426 INFO:teuthology.task.kernel:config {'mon.a': {'branch': 'distro', 'kdb': True, 'sha1': 'distro'}}, timeout 300 2026-02-22T22:30:35.426 DEBUG:teuthology.orchestra.run.trial023:> test -f /run/.containerenv -o -f /.dockerenv 2026-02-22T22:30:35.430 DEBUG:teuthology.orchestra.run:got remote process result: 1 2026-02-22T22:30:35.430 DEBUG:teuthology.orchestra.run.trial023:> uname -r 2026-02-22T22:30:35.476 INFO:teuthology.orchestra.run.trial023.stdout:5.15.0-170-generic 2026-02-22T22:30:35.477 INFO:teuthology.task.kernel:Running kernel on trial023: 5.15.0-170-generic 2026-02-22T22:30:35.477 DEBUG:teuthology.orchestra.run.trial023:> sudo apt-get clean 2026-02-22T22:30:35.538 DEBUG:teuthology.orchestra.run.trial023:> sudo apt-get update 2026-02-22T22:30:36.070 INFO:teuthology.orchestra.run.trial023.stdout:Hit:1 https://archive.ubuntu.com/ubuntu jammy InRelease 2026-02-22T22:30:36.163 INFO:teuthology.orchestra.run.trial023.stdout:Get:2 https://archive.ubuntu.com/ubuntu jammy-updates InRelease [128 kB] 2026-02-22T22:30:36.515 INFO:teuthology.orchestra.run.trial023.stdout:Get:3 https://archive.ubuntu.com/ubuntu jammy-security InRelease [129 kB] 2026-02-22T22:30:36.669 INFO:teuthology.orchestra.run.trial023.stdout:Get:4 https://archive.ubuntu.com/ubuntu jammy-backports InRelease [127 kB] 2026-02-22T22:30:36.813 INFO:teuthology.orchestra.run.trial023.stdout:Get:5 https://archive.ubuntu.com/ubuntu jammy-updates/main amd64 Packages [3245 kB] 2026-02-22T22:30:37.227 INFO:teuthology.orchestra.run.trial023.stdout:Get:6 https://archive.ubuntu.com/ubuntu jammy-updates/main Translation-en [495 kB] 2026-02-22T22:30:37.240 INFO:teuthology.orchestra.run.trial023.stdout:Get:7 https://archive.ubuntu.com/ubuntu jammy-updates/main amd64 c-n-f Metadata [19.1 kB] 2026-02-22T22:30:37.241 INFO:teuthology.orchestra.run.trial023.stdout:Get:8 https://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 Packages [1256 kB] 2026-02-22T22:30:37.277 INFO:teuthology.orchestra.run.trial023.stdout:Get:9 https://archive.ubuntu.com/ubuntu jammy-updates/universe Translation-en [314 kB] 2026-02-22T22:30:37.285 INFO:teuthology.orchestra.run.trial023.stdout:Get:10 https://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 c-n-f Metadata [30.3 kB] 2026-02-22T22:30:37.286 INFO:teuthology.orchestra.run.trial023.stdout:Get:11 https://archive.ubuntu.com/ubuntu jammy-updates/restricted amd64 Packages [5310 kB] 2026-02-22T22:30:37.447 INFO:teuthology.orchestra.run.trial023.stdout:Get:12 https://archive.ubuntu.com/ubuntu jammy-updates/restricted Translation-en [1005 kB] 2026-02-22T22:30:37.471 INFO:teuthology.orchestra.run.trial023.stdout:Get:13 https://archive.ubuntu.com/ubuntu jammy-security/main amd64 Packages [2983 kB] 2026-02-22T22:30:37.561 INFO:teuthology.orchestra.run.trial023.stdout:Get:14 https://archive.ubuntu.com/ubuntu jammy-security/main Translation-en [427 kB] 2026-02-22T22:30:37.570 INFO:teuthology.orchestra.run.trial023.stdout:Get:15 https://archive.ubuntu.com/ubuntu jammy-security/main amd64 c-n-f Metadata [14.1 kB] 2026-02-22T22:30:37.571 INFO:teuthology.orchestra.run.trial023.stdout:Get:16 https://archive.ubuntu.com/ubuntu jammy-security/universe amd64 Packages [1018 kB] 2026-02-22T22:30:37.606 INFO:teuthology.orchestra.run.trial023.stdout:Get:17 https://archive.ubuntu.com/ubuntu jammy-security/universe amd64 c-n-f Metadata [22.7 kB] 2026-02-22T22:30:37.607 INFO:teuthology.orchestra.run.trial023.stdout:Get:18 https://archive.ubuntu.com/ubuntu jammy-security/restricted amd64 Packages [5147 kB] 2026-02-22T22:30:37.759 INFO:teuthology.orchestra.run.trial023.stdout:Get:19 https://archive.ubuntu.com/ubuntu jammy-security/restricted Translation-en [979 kB] 2026-02-22T22:30:41.160 INFO:teuthology.orchestra.run.trial023.stdout:Fetched 22.6 MB in 3s (7653 kB/s) 2026-02-22T22:30:41.677 INFO:teuthology.orchestra.run.trial023.stdout:Reading package lists... 2026-02-22T22:30:41.691 DEBUG:teuthology.orchestra.run.trial023:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2026-02-22T22:30:41.723 INFO:teuthology.orchestra.run.trial023.stdout:Reading package lists... 2026-02-22T22:30:41.829 INFO:teuthology.orchestra.run.trial023.stdout:Building dependency tree... 2026-02-22T22:30:41.830 INFO:teuthology.orchestra.run.trial023.stdout:Reading state information... 2026-02-22T22:30:41.915 INFO:teuthology.orchestra.run.trial023.stdout:linux-image-generic is already the newest version (5.15.0.170.159). 2026-02-22T22:30:41.915 INFO:teuthology.orchestra.run.trial023.stdout:linux-image-generic set to manually installed. 2026-02-22T22:30:41.915 INFO:teuthology.orchestra.run.trial023.stdout:The following packages were automatically installed and are no longer required: 2026-02-22T22:30:41.915 INFO:teuthology.orchestra.run.trial023.stdout: kpartx libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 2026-02-22T22:30:41.915 INFO:teuthology.orchestra.run.trial023.stdout: libgfxdr0 libglusterfs0 libiscsi7 libpmemobj1 libsgutils2-2 sg3-utils 2026-02-22T22:30:41.915 INFO:teuthology.orchestra.run.trial023.stdout: sg3-utils-udev 2026-02-22T22:30:41.916 INFO:teuthology.orchestra.run.trial023.stdout:Use 'sudo apt autoremove' to remove them. 2026-02-22T22:30:41.934 INFO:teuthology.orchestra.run.trial023.stdout:0 upgraded, 0 newly installed, 0 to remove and 13 not upgraded. 2026-02-22T22:30:41.936 DEBUG:teuthology.orchestra.run.trial023:> dpkg -s linux-image-generic 2026-02-22T22:30:41.986 INFO:teuthology.orchestra.run.trial023.stdout:Package: linux-image-generic 2026-02-22T22:30:41.986 INFO:teuthology.orchestra.run.trial023.stdout:Status: install ok installed 2026-02-22T22:30:41.986 INFO:teuthology.orchestra.run.trial023.stdout:Priority: optional 2026-02-22T22:30:41.986 INFO:teuthology.orchestra.run.trial023.stdout:Section: kernel 2026-02-22T22:30:41.986 INFO:teuthology.orchestra.run.trial023.stdout:Installed-Size: 21 2026-02-22T22:30:41.986 INFO:teuthology.orchestra.run.trial023.stdout:Maintainer: Ubuntu Kernel Team 2026-02-22T22:30:41.986 INFO:teuthology.orchestra.run.trial023.stdout:Architecture: amd64 2026-02-22T22:30:41.986 INFO:teuthology.orchestra.run.trial023.stdout:Source: linux-meta 2026-02-22T22:30:41.986 INFO:teuthology.orchestra.run.trial023.stdout:Version: 5.15.0.170.159 2026-02-22T22:30:41.986 INFO:teuthology.orchestra.run.trial023.stdout:Provides: spl-modules (= 2.1.5-1ubuntu6~22.04.6), v4l2loopback-modules (= 0.12.7-2ubuntu2~22.04.1), virtualbox-guest-modules (= 5.15.0-170), wireguard-modules (= 1.0.0), zfs-modules (= 2.1.5-1ubuntu6~22.04.6) 2026-02-22T22:30:41.986 INFO:teuthology.orchestra.run.trial023.stdout:Depends: linux-image-5.15.0-170-generic, linux-modules-extra-5.15.0-170-generic, linux-firmware, intel-microcode, amd64-microcode 2026-02-22T22:30:41.986 INFO:teuthology.orchestra.run.trial023.stdout:Recommends: thermald 2026-02-22T22:30:41.986 INFO:teuthology.orchestra.run.trial023.stdout:Description: Generic Linux kernel image 2026-02-22T22:30:41.986 INFO:teuthology.orchestra.run.trial023.stdout: This package will always depend on the latest generic kernel image 2026-02-22T22:30:41.986 INFO:teuthology.orchestra.run.trial023.stdout: available. 2026-02-22T22:30:41.986 INFO:teuthology.task.kernel:Newest distro kernel installed and running 2026-02-22T22:30:41.987 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2026-02-22T22:30:41.987 DEBUG:teuthology.orchestra.run.trial023:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2026-02-22T22:30:42.036 INFO:teuthology.orchestra.run.trial023.stdout:ttyS1 2026-02-22T22:30:42.074 DEBUG:teuthology.parallel:result is None 2026-02-22T22:30:42.074 INFO:teuthology.run_tasks:Running task internal.base... 2026-02-22T22:30:42.086 INFO:teuthology.task.internal:Creating test directory... 2026-02-22T22:30:42.086 DEBUG:teuthology.orchestra.run.trial023:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2026-02-22T22:30:42.119 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2026-02-22T22:30:42.130 INFO:teuthology.run_tasks:Running task internal.archive... 2026-02-22T22:30:42.143 INFO:teuthology.task.internal:Creating archive directory... 2026-02-22T22:30:42.144 DEBUG:teuthology.orchestra.run.trial023:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2026-02-22T22:30:42.167 INFO:teuthology.run_tasks:Running task internal.coredump... 2026-02-22T22:30:42.180 INFO:teuthology.task.internal:Enabling coredump saving... 2026-02-22T22:30:42.180 DEBUG:teuthology.orchestra.run.trial023:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2026-02-22T22:30:42.217 INFO:teuthology.orchestra.run.trial023.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-22T22:30:42.223 INFO:teuthology.orchestra.run.trial023.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-22T22:30:42.224 INFO:teuthology.run_tasks:Running task internal.sudo... 2026-02-22T22:30:42.239 INFO:teuthology.task.internal:Configuring sudo... 2026-02-22T22:30:42.240 DEBUG:teuthology.orchestra.run.trial023:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2026-02-22T22:30:42.272 INFO:teuthology.run_tasks:Running task internal.syslog... 2026-02-22T22:30:42.285 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2026-02-22T22:30:42.285 DEBUG:teuthology.orchestra.run.trial023:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2026-02-22T22:30:42.315 DEBUG:teuthology.orchestra.run.trial023:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2026-02-22T22:30:42.359 DEBUG:teuthology.orchestra.run.trial023:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2026-02-22T22:30:42.404 DEBUG:teuthology.orchestra.run.trial023:> set -ex 2026-02-22T22:30:42.404 DEBUG:teuthology.orchestra.run.trial023:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2026-02-22T22:30:42.453 DEBUG:teuthology.orchestra.run.trial023:> sudo service rsyslog restart 2026-02-22T22:30:42.513 INFO:teuthology.run_tasks:Running task internal.timer... 2026-02-22T22:30:42.526 INFO:teuthology.task.internal:Starting timer... 2026-02-22T22:30:42.526 INFO:teuthology.run_tasks:Running task pcp... 2026-02-22T22:30:42.548 INFO:teuthology.run_tasks:Running task selinux... 2026-02-22T22:30:42.566 DEBUG:teuthology.task.selinux:Excluding trial023: OS 'ubuntu' does not support SELinux 2026-02-22T22:30:42.566 DEBUG:teuthology.task.selinux:Getting current SELinux state 2026-02-22T22:30:42.566 DEBUG:teuthology.task.selinux:Existing SELinux modes: {} 2026-02-22T22:30:42.566 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2026-02-22T22:30:42.567 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2026-02-22T22:30:42.580 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2026-02-22T22:30:42.593 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2026-02-22T22:30:42.594 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit trial023.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2026-02-22T22:33:04.957 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@trial023.front.sepia.ceph.com')] 2026-02-22T22:33:04.958 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@trial023.front.sepia.ceph.com' 2026-02-22T22:33:04.958 DEBUG:teuthology.orchestra.connection:{'hostname': 'trial023.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2026-02-22T22:33:05.031 DEBUG:teuthology.orchestra.run.trial023:> true 2026-02-22T22:33:05.274 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@trial023.front.sepia.ceph.com' 2026-02-22T22:33:05.275 INFO:teuthology.run_tasks:Running task clock... 2026-02-22T22:33:05.294 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2026-02-22T22:33:05.294 INFO:teuthology.orchestra.run:Running command with timeout 360 2026-02-22T22:33:05.295 DEBUG:teuthology.orchestra.run.trial023:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2026-02-22T22:33:05.334 INFO:teuthology.orchestra.run.trial023.stdout:22 Feb 22:33:05 ntpd[9424]: ntpd 4.2.8p15@1.3728-o Wed Feb 16 17:13:02 UTC 2022 (1): Starting 2026-02-22T22:33:05.335 INFO:teuthology.orchestra.run.trial023.stdout:22 Feb 22:33:05 ntpd[9424]: Command line: ntpd -gq 2026-02-22T22:33:05.335 INFO:teuthology.orchestra.run.trial023.stdout:22 Feb 22:33:05 ntpd[9424]: ---------------------------------------------------- 2026-02-22T22:33:05.335 INFO:teuthology.orchestra.run.trial023.stdout:22 Feb 22:33:05 ntpd[9424]: ntp-4 is maintained by Network Time Foundation, 2026-02-22T22:33:05.335 INFO:teuthology.orchestra.run.trial023.stdout:22 Feb 22:33:05 ntpd[9424]: Inc. (NTF), a non-profit 501(c)(3) public-benefit 2026-02-22T22:33:05.336 INFO:teuthology.orchestra.run.trial023.stdout:22 Feb 22:33:05 ntpd[9424]: corporation. Support and training for ntp-4 are 2026-02-22T22:33:05.336 INFO:teuthology.orchestra.run.trial023.stdout:22 Feb 22:33:05 ntpd[9424]: available at https://www.nwtime.org/support 2026-02-22T22:33:05.336 INFO:teuthology.orchestra.run.trial023.stdout:22 Feb 22:33:05 ntpd[9424]: ---------------------------------------------------- 2026-02-22T22:33:05.336 INFO:teuthology.orchestra.run.trial023.stdout:22 Feb 22:33:05 ntpd[9424]: proto: precision = 0.030 usec (-25) 2026-02-22T22:33:05.336 INFO:teuthology.orchestra.run.trial023.stdout:22 Feb 22:33:05 ntpd[9424]: basedate set to 2022-02-04 2026-02-22T22:33:05.336 INFO:teuthology.orchestra.run.trial023.stdout:22 Feb 22:33:05 ntpd[9424]: gps base set to 2022-02-06 (week 2196) 2026-02-22T22:33:05.337 INFO:teuthology.orchestra.run.trial023.stdout:22 Feb 22:33:05 ntpd[9424]: restrict 0.0.0.0: KOD does nothing without LIMITED. 2026-02-22T22:33:05.337 INFO:teuthology.orchestra.run.trial023.stderr:restrict 0.0.0.0: KOD does nothing without LIMITED. 2026-02-22T22:33:05.338 INFO:teuthology.orchestra.run.trial023.stderr:restrict ::: KOD does nothing without LIMITED. 2026-02-22T22:33:05.340 INFO:teuthology.orchestra.run.trial023.stdout:22 Feb 22:33:05 ntpd[9424]: restrict ::: KOD does nothing without LIMITED. 2026-02-22T22:33:05.340 INFO:teuthology.orchestra.run.trial023.stdout:22 Feb 22:33:05 ntpd[9424]: Listen and drop on 0 v6wildcard [::]:123 2026-02-22T22:33:05.340 INFO:teuthology.orchestra.run.trial023.stdout:22 Feb 22:33:05 ntpd[9424]: Listen and drop on 1 v4wildcard 0.0.0.0:123 2026-02-22T22:33:05.340 INFO:teuthology.orchestra.run.trial023.stdout:22 Feb 22:33:05 ntpd[9424]: Listen normally on 2 lo 127.0.0.1:123 2026-02-22T22:33:05.340 INFO:teuthology.orchestra.run.trial023.stdout:22 Feb 22:33:05 ntpd[9424]: Listen normally on 3 enp1s0f0 10.20.193.23:123 2026-02-22T22:33:05.340 INFO:teuthology.orchestra.run.trial023.stdout:22 Feb 22:33:05 ntpd[9424]: Listen normally on 4 lo [::1]:123 2026-02-22T22:33:05.341 INFO:teuthology.orchestra.run.trial023.stdout:22 Feb 22:33:05 ntpd[9424]: Listen normally on 5 enp1s0f0 [fe80::925a:8ff:fe77:648e%2]:123 2026-02-22T22:33:05.341 INFO:teuthology.orchestra.run.trial023.stdout:22 Feb 22:33:05 ntpd[9424]: Listening on routing socket on fd #22 for interface updates 2026-02-22T22:33:06.334 INFO:teuthology.orchestra.run.trial023.stderr:22 Feb 22:33:06 ntpd[9424]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2026-02-22T22:33:06.335 INFO:teuthology.orchestra.run.trial023.stderr:22 Feb 22:33:06 ntpd[9424]: can't open /var/log/ntpstats/rawstats.20260222: Permission denied 2026-02-22T22:33:06.335 INFO:teuthology.orchestra.run.trial023.stderr:22 Feb 22:33:06 ntpd[9424]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2026-02-22T22:33:06.335 INFO:teuthology.orchestra.run.trial023.stderr:22 Feb 22:33:06 ntpd[9424]: can't open /var/log/ntpstats/rawstats.20260222: Permission denied 2026-02-22T22:33:06.371 INFO:teuthology.orchestra.run.trial023.stderr:22 Feb 22:33:06 ntpd[9424]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2026-02-22T22:33:06.371 INFO:teuthology.orchestra.run.trial023.stderr:22 Feb 22:33:06 ntpd[9424]: can't open /var/log/ntpstats/rawstats.20260222: Permission denied 2026-02-22T22:33:06.372 INFO:teuthology.orchestra.run.trial023.stderr:22 Feb 22:33:06 ntpd[9424]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2026-02-22T22:33:06.372 INFO:teuthology.orchestra.run.trial023.stderr:22 Feb 22:33:06 ntpd[9424]: can't open /var/log/ntpstats/peerstats.20260222: Permission denied 2026-02-22T22:33:06.398 INFO:teuthology.orchestra.run.trial023.stderr:22 Feb 22:33:06 ntpd[9424]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2026-02-22T22:33:06.398 INFO:teuthology.orchestra.run.trial023.stderr:22 Feb 22:33:06 ntpd[9424]: can't open /var/log/ntpstats/rawstats.20260222: Permission denied 2026-02-22T22:33:06.398 INFO:teuthology.orchestra.run.trial023.stderr:22 Feb 22:33:06 ntpd[9424]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2026-02-22T22:33:06.398 INFO:teuthology.orchestra.run.trial023.stderr:22 Feb 22:33:06 ntpd[9424]: can't open /var/log/ntpstats/peerstats.20260222: Permission denied 2026-02-22T22:33:07.334 INFO:teuthology.orchestra.run.trial023.stderr:22 Feb 22:33:07 ntpd[9424]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2026-02-22T22:33:07.334 INFO:teuthology.orchestra.run.trial023.stderr:22 Feb 22:33:07 ntpd[9424]: can't open /var/log/ntpstats/rawstats.20260222: Permission denied 2026-02-22T22:33:07.401 INFO:teuthology.orchestra.run.trial023.stderr:22 Feb 22:33:07 ntpd[9424]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2026-02-22T22:33:07.401 INFO:teuthology.orchestra.run.trial023.stderr:22 Feb 22:33:07 ntpd[9424]: can't open /var/log/ntpstats/rawstats.20260222: Permission denied 2026-02-22T22:33:07.401 INFO:teuthology.orchestra.run.trial023.stderr:22 Feb 22:33:07 ntpd[9424]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2026-02-22T22:33:07.401 INFO:teuthology.orchestra.run.trial023.stderr:22 Feb 22:33:07 ntpd[9424]: can't open /var/log/ntpstats/peerstats.20260222: Permission denied 2026-02-22T22:33:08.334 INFO:teuthology.orchestra.run.trial023.stderr:22 Feb 22:33:08 ntpd[9424]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2026-02-22T22:33:08.334 INFO:teuthology.orchestra.run.trial023.stderr:22 Feb 22:33:08 ntpd[9424]: can't open /var/log/ntpstats/rawstats.20260222: Permission denied 2026-02-22T22:33:08.335 INFO:teuthology.orchestra.run.trial023.stderr:22 Feb 22:33:08 ntpd[9424]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2026-02-22T22:33:08.335 INFO:teuthology.orchestra.run.trial023.stderr:22 Feb 22:33:08 ntpd[9424]: can't open /var/log/ntpstats/rawstats.20260222: Permission denied 2026-02-22T22:33:08.371 INFO:teuthology.orchestra.run.trial023.stderr:22 Feb 22:33:08 ntpd[9424]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2026-02-22T22:33:08.371 INFO:teuthology.orchestra.run.trial023.stderr:22 Feb 22:33:08 ntpd[9424]: can't open /var/log/ntpstats/rawstats.20260222: Permission denied 2026-02-22T22:33:08.371 INFO:teuthology.orchestra.run.trial023.stderr:22 Feb 22:33:08 ntpd[9424]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2026-02-22T22:33:08.372 INFO:teuthology.orchestra.run.trial023.stderr:22 Feb 22:33:08 ntpd[9424]: can't open /var/log/ntpstats/peerstats.20260222: Permission denied 2026-02-22T22:33:08.398 INFO:teuthology.orchestra.run.trial023.stderr:22 Feb 22:33:08 ntpd[9424]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2026-02-22T22:33:08.398 INFO:teuthology.orchestra.run.trial023.stderr:22 Feb 22:33:08 ntpd[9424]: can't open /var/log/ntpstats/rawstats.20260222: Permission denied 2026-02-22T22:33:08.399 INFO:teuthology.orchestra.run.trial023.stderr:22 Feb 22:33:08 ntpd[9424]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2026-02-22T22:33:08.399 INFO:teuthology.orchestra.run.trial023.stderr:22 Feb 22:33:08 ntpd[9424]: can't open /var/log/ntpstats/peerstats.20260222: Permission denied 2026-02-22T22:33:09.334 INFO:teuthology.orchestra.run.trial023.stderr:22 Feb 22:33:09 ntpd[9424]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2026-02-22T22:33:09.334 INFO:teuthology.orchestra.run.trial023.stderr:22 Feb 22:33:09 ntpd[9424]: can't open /var/log/ntpstats/rawstats.20260222: Permission denied 2026-02-22T22:33:09.335 INFO:teuthology.orchestra.run.trial023.stderr:22 Feb 22:33:09 ntpd[9424]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2026-02-22T22:33:09.335 INFO:teuthology.orchestra.run.trial023.stderr:22 Feb 22:33:09 ntpd[9424]: can't open /var/log/ntpstats/rawstats.20260222: Permission denied 2026-02-22T22:33:09.401 INFO:teuthology.orchestra.run.trial023.stderr:22 Feb 22:33:09 ntpd[9424]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2026-02-22T22:33:09.401 INFO:teuthology.orchestra.run.trial023.stderr:22 Feb 22:33:09 ntpd[9424]: can't open /var/log/ntpstats/rawstats.20260222: Permission denied 2026-02-22T22:33:09.401 INFO:teuthology.orchestra.run.trial023.stderr:22 Feb 22:33:09 ntpd[9424]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2026-02-22T22:33:09.401 INFO:teuthology.orchestra.run.trial023.stderr:22 Feb 22:33:09 ntpd[9424]: can't open /var/log/ntpstats/peerstats.20260222: Permission denied 2026-02-22T22:33:09.406 INFO:teuthology.orchestra.run.trial023.stderr:22 Feb 22:33:09 ntpd[9424]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2026-02-22T22:33:09.406 INFO:teuthology.orchestra.run.trial023.stderr:22 Feb 22:33:09 ntpd[9424]: can't open /var/log/ntpstats/rawstats.20260222: Permission denied 2026-02-22T22:33:09.406 INFO:teuthology.orchestra.run.trial023.stderr:22 Feb 22:33:09 ntpd[9424]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2026-02-22T22:33:09.407 INFO:teuthology.orchestra.run.trial023.stderr:22 Feb 22:33:09 ntpd[9424]: can't open /var/log/ntpstats/peerstats.20260222: Permission denied 2026-02-22T22:33:10.334 INFO:teuthology.orchestra.run.trial023.stderr:22 Feb 22:33:10 ntpd[9424]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2026-02-22T22:33:10.334 INFO:teuthology.orchestra.run.trial023.stderr:22 Feb 22:33:10 ntpd[9424]: can't open /var/log/ntpstats/rawstats.20260222: Permission denied 2026-02-22T22:33:10.334 INFO:teuthology.orchestra.run.trial023.stderr:22 Feb 22:33:10 ntpd[9424]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2026-02-22T22:33:10.334 INFO:teuthology.orchestra.run.trial023.stderr:22 Feb 22:33:10 ntpd[9424]: can't open /var/log/ntpstats/rawstats.20260222: Permission denied 2026-02-22T22:33:10.371 INFO:teuthology.orchestra.run.trial023.stderr:22 Feb 22:33:10 ntpd[9424]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2026-02-22T22:33:10.371 INFO:teuthology.orchestra.run.trial023.stderr:22 Feb 22:33:10 ntpd[9424]: can't open /var/log/ntpstats/rawstats.20260222: Permission denied 2026-02-22T22:33:10.371 INFO:teuthology.orchestra.run.trial023.stderr:22 Feb 22:33:10 ntpd[9424]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2026-02-22T22:33:10.371 INFO:teuthology.orchestra.run.trial023.stderr:22 Feb 22:33:10 ntpd[9424]: can't open /var/log/ntpstats/peerstats.20260222: Permission denied 2026-02-22T22:33:10.398 INFO:teuthology.orchestra.run.trial023.stderr:22 Feb 22:33:10 ntpd[9424]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2026-02-22T22:33:10.398 INFO:teuthology.orchestra.run.trial023.stderr:22 Feb 22:33:10 ntpd[9424]: can't open /var/log/ntpstats/rawstats.20260222: Permission denied 2026-02-22T22:33:10.399 INFO:teuthology.orchestra.run.trial023.stderr:22 Feb 22:33:10 ntpd[9424]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2026-02-22T22:33:10.399 INFO:teuthology.orchestra.run.trial023.stderr:22 Feb 22:33:10 ntpd[9424]: can't open /var/log/ntpstats/peerstats.20260222: Permission denied 2026-02-22T22:33:11.334 INFO:teuthology.orchestra.run.trial023.stderr:22 Feb 22:33:11 ntpd[9424]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2026-02-22T22:33:11.335 INFO:teuthology.orchestra.run.trial023.stderr:22 Feb 22:33:11 ntpd[9424]: can't open /var/log/ntpstats/rawstats.20260222: Permission denied 2026-02-22T22:33:11.335 INFO:teuthology.orchestra.run.trial023.stderr:22 Feb 22:33:11 ntpd[9424]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2026-02-22T22:33:11.335 INFO:teuthology.orchestra.run.trial023.stderr:22 Feb 22:33:11 ntpd[9424]: can't open /var/log/ntpstats/rawstats.20260222: Permission denied 2026-02-22T22:33:11.401 INFO:teuthology.orchestra.run.trial023.stderr:22 Feb 22:33:11 ntpd[9424]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2026-02-22T22:33:11.401 INFO:teuthology.orchestra.run.trial023.stderr:22 Feb 22:33:11 ntpd[9424]: can't open /var/log/ntpstats/rawstats.20260222: Permission denied 2026-02-22T22:33:11.401 INFO:teuthology.orchestra.run.trial023.stderr:22 Feb 22:33:11 ntpd[9424]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2026-02-22T22:33:11.401 INFO:teuthology.orchestra.run.trial023.stderr:22 Feb 22:33:11 ntpd[9424]: can't open /var/log/ntpstats/peerstats.20260222: Permission denied 2026-02-22T22:33:11.406 INFO:teuthology.orchestra.run.trial023.stderr:22 Feb 22:33:11 ntpd[9424]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2026-02-22T22:33:11.406 INFO:teuthology.orchestra.run.trial023.stderr:22 Feb 22:33:11 ntpd[9424]: can't open /var/log/ntpstats/rawstats.20260222: Permission denied 2026-02-22T22:33:11.406 INFO:teuthology.orchestra.run.trial023.stderr:22 Feb 22:33:11 ntpd[9424]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2026-02-22T22:33:11.406 INFO:teuthology.orchestra.run.trial023.stderr:22 Feb 22:33:11 ntpd[9424]: can't open /var/log/ntpstats/peerstats.20260222: Permission denied 2026-02-22T22:33:12.334 INFO:teuthology.orchestra.run.trial023.stderr:22 Feb 22:33:12 ntpd[9424]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2026-02-22T22:33:12.334 INFO:teuthology.orchestra.run.trial023.stderr:22 Feb 22:33:12 ntpd[9424]: can't open /var/log/ntpstats/rawstats.20260222: Permission denied 2026-02-22T22:33:12.334 INFO:teuthology.orchestra.run.trial023.stderr:22 Feb 22:33:12 ntpd[9424]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2026-02-22T22:33:12.334 INFO:teuthology.orchestra.run.trial023.stderr:22 Feb 22:33:12 ntpd[9424]: can't open /var/log/ntpstats/rawstats.20260222: Permission denied 2026-02-22T22:33:12.370 INFO:teuthology.orchestra.run.trial023.stderr:22 Feb 22:33:12 ntpd[9424]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2026-02-22T22:33:12.371 INFO:teuthology.orchestra.run.trial023.stderr:22 Feb 22:33:12 ntpd[9424]: can't open /var/log/ntpstats/rawstats.20260222: Permission denied 2026-02-22T22:33:12.371 INFO:teuthology.orchestra.run.trial023.stderr:22 Feb 22:33:12 ntpd[9424]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2026-02-22T22:33:12.371 INFO:teuthology.orchestra.run.trial023.stderr:22 Feb 22:33:12 ntpd[9424]: can't open /var/log/ntpstats/peerstats.20260222: Permission denied 2026-02-22T22:33:12.371 INFO:teuthology.orchestra.run.trial023.stdout:22 Feb 22:33:12 ntpd[9424]: ntpd: time slew +0.002280 s 2026-02-22T22:33:12.371 INFO:teuthology.orchestra.run.trial023.stdout:ntpd: time slew +0.002280s 2026-02-22T22:33:12.371 INFO:teuthology.orchestra.run.trial023.stderr:22 Feb 22:33:12 ntpd[9424]: couldn't unlink /var/log/ntpstats/loopstats: Permission denied 2026-02-22T22:33:12.371 INFO:teuthology.orchestra.run.trial023.stderr:22 Feb 22:33:12 ntpd[9424]: can't open /var/log/ntpstats/loopstats.20260222: Permission denied 2026-02-22T22:33:12.437 INFO:teuthology.orchestra.run.trial023.stdout: remote refid st t when poll reach delay offset jitter 2026-02-22T22:33:12.437 INFO:teuthology.orchestra.run.trial023.stdout:============================================================================== 2026-02-22T22:33:12.437 INFO:teuthology.orchestra.run.trial023.stdout: nyc3.us.ntp.li .INIT. 16 u - 64 0 0.000 +0.000 0.000 2026-02-22T22:33:12.437 INFO:teuthology.orchestra.run.trial023.stdout: ntp.maxhost.io .INIT. 16 u - 64 0 0.000 +0.000 0.000 2026-02-22T22:33:12.437 INFO:teuthology.orchestra.run.trial023.stdout: LAX.CALTICK.NET .INIT. 16 u - 64 0 0.000 +0.000 0.000 2026-02-22T22:33:12.438 INFO:teuthology.orchestra.run.trial023.stdout: 24-111-79-186-d .INIT. 16 u - 64 0 0.000 +0.000 0.000 2026-02-22T22:33:12.438 INFO:teuthology.run_tasks:Running task install... 2026-02-22T22:33:12.454 DEBUG:teuthology.task.install:project ceph 2026-02-22T22:33:12.454 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': 'f90f4eaebb28b619cf76e80cbbe997656da69579'}} 2026-02-22T22:33:12.454 DEBUG:teuthology.task.install:config {'flavor': 'default', 'sha1': 'f90f4eaebb28b619cf76e80cbbe997656da69579'} 2026-02-22T22:33:12.454 INFO:teuthology.task.install:Using flavor: default 2026-02-22T22:33:12.463 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2026-02-22T22:33:12.463 INFO:teuthology.task.install:extra packages: [] 2026-02-22T22:33:12.464 DEBUG:teuthology.orchestra.run.trial023:> sudo apt-key list | grep Ceph 2026-02-22T22:33:12.530 INFO:teuthology.orchestra.run.trial023.stderr:Warning: apt-key is deprecated. Manage keyring files in trusted.gpg.d instead (see apt-key(8)). 2026-02-22T22:33:12.560 INFO:teuthology.orchestra.run.trial023.stdout:uid [ unknown] Ceph automated package build (Ceph automated package build) 2026-02-22T22:33:12.560 INFO:teuthology.orchestra.run.trial023.stdout:uid [ unknown] Ceph.com (release key) 2026-02-22T22:33:12.561 INFO:teuthology.task.install.deb:Installing packages: ceph, cephadm, ceph-mds, ceph-mgr, ceph-common, ceph-fuse, ceph-test, ceph-volume, radosgw, python3-rados, python3-rgw, python3-cephfs, python3-rbd, libcephfs2, libcephfs-dev, librados2, librbd1, rbd-fuse on remote deb x86_64 2026-02-22T22:33:12.562 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&sha1=f90f4eaebb28b619cf76e80cbbe997656da69579 2026-02-22T22:33:12.741 INFO:teuthology.task.install.deb:Pulling from https://2.chacra.ceph.com/r/ceph/wip-yuri6-testing-2026-02-22-1629-squid/f90f4eaebb28b619cf76e80cbbe997656da69579/ubuntu/jammy/flavors/default/ 2026-02-22T22:33:12.741 INFO:teuthology.task.install.deb:Package version is 19.2.3-689-gf90f4eae-1jammy 2026-02-22T22:33:12.882 DEBUG:teuthology.orchestra.run.trial023:> set -ex 2026-02-22T22:33:12.882 DEBUG:teuthology.orchestra.run.trial023:> sudo dd of=/etc/apt/sources.list.d/ceph.list 2026-02-22T22:33:12.892 DEBUG:teuthology.orchestra.run.trial023:> sudo apt-get update 2026-02-22T22:33:13.038 INFO:teuthology.orchestra.run.trial023.stdout:Ign:1 https://2.chacra.ceph.com/r/ceph/wip-yuri6-testing-2026-02-22-1629-squid/f90f4eaebb28b619cf76e80cbbe997656da69579/ubuntu/jammy/flavors/default jammy InRelease 2026-02-22T22:33:13.049 INFO:teuthology.orchestra.run.trial023.stdout:Get:2 https://2.chacra.ceph.com/r/ceph/wip-yuri6-testing-2026-02-22-1629-squid/f90f4eaebb28b619cf76e80cbbe997656da69579/ubuntu/jammy/flavors/default jammy Release [7662 B] 2026-02-22T22:33:13.062 INFO:teuthology.orchestra.run.trial023.stdout:Ign:3 https://2.chacra.ceph.com/r/ceph/wip-yuri6-testing-2026-02-22-1629-squid/f90f4eaebb28b619cf76e80cbbe997656da69579/ubuntu/jammy/flavors/default jammy Release.gpg 2026-02-22T22:33:13.073 INFO:teuthology.orchestra.run.trial023.stdout:Get:4 https://2.chacra.ceph.com/r/ceph/wip-yuri6-testing-2026-02-22-1629-squid/f90f4eaebb28b619cf76e80cbbe997656da69579/ubuntu/jammy/flavors/default jammy/main amd64 Packages [18.1 kB] 2026-02-22T22:33:13.399 INFO:teuthology.orchestra.run.trial023.stdout:Hit:5 https://archive.ubuntu.com/ubuntu jammy InRelease 2026-02-22T22:33:13.481 INFO:teuthology.orchestra.run.trial023.stdout:Hit:6 https://archive.ubuntu.com/ubuntu jammy-updates InRelease 2026-02-22T22:33:13.569 INFO:teuthology.orchestra.run.trial023.stdout:Hit:7 https://archive.ubuntu.com/ubuntu jammy-security InRelease 2026-02-22T22:33:13.657 INFO:teuthology.orchestra.run.trial023.stdout:Hit:8 https://archive.ubuntu.com/ubuntu jammy-backports InRelease 2026-02-22T22:33:13.799 INFO:teuthology.orchestra.run.trial023.stdout:Fetched 25.8 kB in 1s (34.6 kB/s) 2026-02-22T22:33:14.310 INFO:teuthology.orchestra.run.trial023.stdout:Reading package lists... 2026-02-22T22:33:14.322 DEBUG:teuthology.orchestra.run.trial023:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" install ceph=19.2.3-689-gf90f4eae-1jammy cephadm=19.2.3-689-gf90f4eae-1jammy ceph-mds=19.2.3-689-gf90f4eae-1jammy ceph-mgr=19.2.3-689-gf90f4eae-1jammy ceph-common=19.2.3-689-gf90f4eae-1jammy ceph-fuse=19.2.3-689-gf90f4eae-1jammy ceph-test=19.2.3-689-gf90f4eae-1jammy ceph-volume=19.2.3-689-gf90f4eae-1jammy radosgw=19.2.3-689-gf90f4eae-1jammy python3-rados=19.2.3-689-gf90f4eae-1jammy python3-rgw=19.2.3-689-gf90f4eae-1jammy python3-cephfs=19.2.3-689-gf90f4eae-1jammy python3-rbd=19.2.3-689-gf90f4eae-1jammy libcephfs2=19.2.3-689-gf90f4eae-1jammy libcephfs-dev=19.2.3-689-gf90f4eae-1jammy librados2=19.2.3-689-gf90f4eae-1jammy librbd1=19.2.3-689-gf90f4eae-1jammy rbd-fuse=19.2.3-689-gf90f4eae-1jammy 2026-02-22T22:33:14.354 INFO:teuthology.orchestra.run.trial023.stdout:Reading package lists... 2026-02-22T22:33:14.460 INFO:teuthology.orchestra.run.trial023.stdout:Building dependency tree... 2026-02-22T22:33:14.460 INFO:teuthology.orchestra.run.trial023.stdout:Reading state information... 2026-02-22T22:33:14.548 INFO:teuthology.orchestra.run.trial023.stdout:The following packages were automatically installed and are no longer required: 2026-02-22T22:33:14.548 INFO:teuthology.orchestra.run.trial023.stdout: kpartx libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 2026-02-22T22:33:14.548 INFO:teuthology.orchestra.run.trial023.stdout: libgfxdr0 libglusterfs0 libiscsi7 libpmemobj1 libsgutils2-2 sg3-utils 2026-02-22T22:33:14.548 INFO:teuthology.orchestra.run.trial023.stdout: sg3-utils-udev 2026-02-22T22:33:14.548 INFO:teuthology.orchestra.run.trial023.stdout:Use 'sudo apt autoremove' to remove them. 2026-02-22T22:33:14.548 INFO:teuthology.orchestra.run.trial023.stdout:The following additional packages will be installed: 2026-02-22T22:33:14.548 INFO:teuthology.orchestra.run.trial023.stdout: ceph-base ceph-mgr-cephadm ceph-mgr-dashboard ceph-mgr-diskprediction-local 2026-02-22T22:33:14.548 INFO:teuthology.orchestra.run.trial023.stdout: ceph-mgr-k8sevents ceph-mgr-modules-core ceph-mon ceph-osd jq 2026-02-22T22:33:14.549 INFO:teuthology.orchestra.run.trial023.stdout: libdouble-conversion3 libfuse2 libjq1 liblttng-ust1 liblua5.3-dev libnbd0 2026-02-22T22:33:14.549 INFO:teuthology.orchestra.run.trial023.stdout: liboath0 libonig5 libpcre2-16-0 libqt5core5a libqt5dbus5 libqt5network5 2026-02-22T22:33:14.549 INFO:teuthology.orchestra.run.trial023.stdout: libradosstriper1 librdkafka1 libreadline-dev librgw2 libsqlite3-mod-ceph 2026-02-22T22:33:14.549 INFO:teuthology.orchestra.run.trial023.stdout: libthrift-0.16.0 lua-any lua-sec lua-socket lua5.1 luarocks nvme-cli 2026-02-22T22:33:14.549 INFO:teuthology.orchestra.run.trial023.stdout: pkg-config python-asyncssh-doc python-pastedeploy-tpl python3-asyncssh 2026-02-22T22:33:14.549 INFO:teuthology.orchestra.run.trial023.stdout: python3-cachetools python3-ceph-argparse python3-ceph-common python3-cheroot 2026-02-22T22:33:14.549 INFO:teuthology.orchestra.run.trial023.stdout: python3-cherrypy3 python3-google-auth python3-iniconfig 2026-02-22T22:33:14.549 INFO:teuthology.orchestra.run.trial023.stdout: python3-jaraco.classes python3-jaraco.collections python3-jaraco.functools 2026-02-22T22:33:14.549 INFO:teuthology.orchestra.run.trial023.stdout: python3-jaraco.text python3-joblib python3-kubernetes python3-logutils 2026-02-22T22:33:14.549 INFO:teuthology.orchestra.run.trial023.stdout: python3-mako python3-natsort python3-paste python3-pastedeploy 2026-02-22T22:33:14.549 INFO:teuthology.orchestra.run.trial023.stdout: python3-pastescript python3-pecan python3-pluggy python3-portend 2026-02-22T22:33:14.549 INFO:teuthology.orchestra.run.trial023.stdout: python3-prettytable python3-psutil python3-py python3-pygments 2026-02-22T22:33:14.550 INFO:teuthology.orchestra.run.trial023.stdout: python3-pyinotify python3-pytest python3-repoze.lru 2026-02-22T22:33:14.550 INFO:teuthology.orchestra.run.trial023.stdout: python3-requests-oauthlib python3-routes python3-rsa python3-simplegeneric 2026-02-22T22:33:14.550 INFO:teuthology.orchestra.run.trial023.stdout: python3-simplejson python3-singledispatch python3-sklearn 2026-02-22T22:33:14.550 INFO:teuthology.orchestra.run.trial023.stdout: python3-sklearn-lib python3-tempita python3-tempora python3-threadpoolctl 2026-02-22T22:33:14.550 INFO:teuthology.orchestra.run.trial023.stdout: python3-toml python3-waitress python3-wcwidth python3-webob 2026-02-22T22:33:14.550 INFO:teuthology.orchestra.run.trial023.stdout: python3-websocket python3-webtest python3-werkzeug python3-zc.lockfile 2026-02-22T22:33:14.550 INFO:teuthology.orchestra.run.trial023.stdout: qttranslations5-l10n socat unzip xmlstarlet zip 2026-02-22T22:33:14.550 INFO:teuthology.orchestra.run.trial023.stdout:Suggested packages: 2026-02-22T22:33:14.550 INFO:teuthology.orchestra.run.trial023.stdout: python3-influxdb readline-doc python3-beaker python-mako-doc 2026-02-22T22:33:14.550 INFO:teuthology.orchestra.run.trial023.stdout: python-natsort-doc httpd-wsgi libapache2-mod-python libapache2-mod-scgi 2026-02-22T22:33:14.550 INFO:teuthology.orchestra.run.trial023.stdout: libjs-mochikit python-pecan-doc python-psutil-doc subversion 2026-02-22T22:33:14.551 INFO:teuthology.orchestra.run.trial023.stdout: python-pygments-doc ttf-bitstream-vera python-pyinotify-doc python3-dap 2026-02-22T22:33:14.551 INFO:teuthology.orchestra.run.trial023.stdout: python-sklearn-doc ipython3 python-waitress-doc python-webob-doc 2026-02-22T22:33:14.551 INFO:teuthology.orchestra.run.trial023.stdout: python-webtest-doc python-werkzeug-doc python3-watchdog 2026-02-22T22:33:14.551 INFO:teuthology.orchestra.run.trial023.stdout:Recommended packages: 2026-02-22T22:33:14.551 INFO:teuthology.orchestra.run.trial023.stdout: btrfs-tools 2026-02-22T22:33:14.582 INFO:teuthology.orchestra.run.trial023.stdout:The following NEW packages will be installed: 2026-02-22T22:33:14.582 INFO:teuthology.orchestra.run.trial023.stdout: ceph ceph-base ceph-common ceph-fuse ceph-mds ceph-mgr ceph-mgr-cephadm 2026-02-22T22:33:14.583 INFO:teuthology.orchestra.run.trial023.stdout: ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-k8sevents 2026-02-22T22:33:14.583 INFO:teuthology.orchestra.run.trial023.stdout: ceph-mgr-modules-core ceph-mon ceph-osd ceph-test ceph-volume cephadm jq 2026-02-22T22:33:14.583 INFO:teuthology.orchestra.run.trial023.stdout: libcephfs-dev libcephfs2 libdouble-conversion3 libfuse2 libjq1 liblttng-ust1 2026-02-22T22:33:14.583 INFO:teuthology.orchestra.run.trial023.stdout: liblua5.3-dev libnbd0 liboath0 libonig5 libpcre2-16-0 libqt5core5a 2026-02-22T22:33:14.583 INFO:teuthology.orchestra.run.trial023.stdout: libqt5dbus5 libqt5network5 librados2 libradosstriper1 librbd1 librdkafka1 2026-02-22T22:33:14.583 INFO:teuthology.orchestra.run.trial023.stdout: libreadline-dev librgw2 libsqlite3-mod-ceph libthrift-0.16.0 lua-any lua-sec 2026-02-22T22:33:14.583 INFO:teuthology.orchestra.run.trial023.stdout: lua-socket lua5.1 luarocks nvme-cli pkg-config python-asyncssh-doc 2026-02-22T22:33:14.583 INFO:teuthology.orchestra.run.trial023.stdout: python-pastedeploy-tpl python3-asyncssh python3-cachetools 2026-02-22T22:33:14.583 INFO:teuthology.orchestra.run.trial023.stdout: python3-ceph-argparse python3-ceph-common python3-cephfs python3-cheroot 2026-02-22T22:33:14.583 INFO:teuthology.orchestra.run.trial023.stdout: python3-cherrypy3 python3-google-auth python3-iniconfig 2026-02-22T22:33:14.583 INFO:teuthology.orchestra.run.trial023.stdout: python3-jaraco.classes python3-jaraco.collections python3-jaraco.functools 2026-02-22T22:33:14.583 INFO:teuthology.orchestra.run.trial023.stdout: python3-jaraco.text python3-joblib python3-kubernetes python3-logutils 2026-02-22T22:33:14.584 INFO:teuthology.orchestra.run.trial023.stdout: python3-mako python3-natsort python3-paste python3-pastedeploy 2026-02-22T22:33:14.584 INFO:teuthology.orchestra.run.trial023.stdout: python3-pastescript python3-pecan python3-pluggy python3-portend 2026-02-22T22:33:14.584 INFO:teuthology.orchestra.run.trial023.stdout: python3-prettytable python3-psutil python3-py python3-pygments 2026-02-22T22:33:14.584 INFO:teuthology.orchestra.run.trial023.stdout: python3-pyinotify python3-pytest python3-rados python3-rbd 2026-02-22T22:33:14.584 INFO:teuthology.orchestra.run.trial023.stdout: python3-repoze.lru python3-requests-oauthlib python3-rgw python3-routes 2026-02-22T22:33:14.584 INFO:teuthology.orchestra.run.trial023.stdout: python3-rsa python3-simplegeneric python3-simplejson python3-singledispatch 2026-02-22T22:33:14.584 INFO:teuthology.orchestra.run.trial023.stdout: python3-sklearn python3-sklearn-lib python3-tempita python3-tempora 2026-02-22T22:33:14.584 INFO:teuthology.orchestra.run.trial023.stdout: python3-threadpoolctl python3-toml python3-waitress python3-wcwidth 2026-02-22T22:33:14.584 INFO:teuthology.orchestra.run.trial023.stdout: python3-webob python3-websocket python3-webtest python3-werkzeug 2026-02-22T22:33:14.584 INFO:teuthology.orchestra.run.trial023.stdout: python3-zc.lockfile qttranslations5-l10n radosgw rbd-fuse socat unzip 2026-02-22T22:33:14.584 INFO:teuthology.orchestra.run.trial023.stdout: xmlstarlet zip 2026-02-22T22:33:14.661 INFO:teuthology.orchestra.run.trial023.stdout:0 upgraded, 108 newly installed, 0 to remove and 13 not upgraded. 2026-02-22T22:33:14.661 INFO:teuthology.orchestra.run.trial023.stdout:Need to get 178 MB of archives. 2026-02-22T22:33:14.661 INFO:teuthology.orchestra.run.trial023.stdout:After this operation, 809 MB of additional disk space will be used. 2026-02-22T22:33:14.661 INFO:teuthology.orchestra.run.trial023.stdout:Get:1 https://2.chacra.ceph.com/r/ceph/wip-yuri6-testing-2026-02-22-1629-squid/f90f4eaebb28b619cf76e80cbbe997656da69579/ubuntu/jammy/flavors/default jammy/main amd64 librados2 amd64 19.2.3-689-gf90f4eae-1jammy [3597 kB] 2026-02-22T22:33:14.764 INFO:teuthology.orchestra.run.trial023.stdout:Get:2 https://2.chacra.ceph.com/r/ceph/wip-yuri6-testing-2026-02-22-1629-squid/f90f4eaebb28b619cf76e80cbbe997656da69579/ubuntu/jammy/flavors/default jammy/main amd64 librbd1 amd64 19.2.3-689-gf90f4eae-1jammy [3256 kB] 2026-02-22T22:33:14.809 INFO:teuthology.orchestra.run.trial023.stdout:Get:3 https://2.chacra.ceph.com/r/ceph/wip-yuri6-testing-2026-02-22-1629-squid/f90f4eaebb28b619cf76e80cbbe997656da69579/ubuntu/jammy/flavors/default jammy/main amd64 libcephfs2 amd64 19.2.3-689-gf90f4eae-1jammy [979 kB] 2026-02-22T22:33:14.824 INFO:teuthology.orchestra.run.trial023.stdout:Get:4 https://2.chacra.ceph.com/r/ceph/wip-yuri6-testing-2026-02-22-1629-squid/f90f4eaebb28b619cf76e80cbbe997656da69579/ubuntu/jammy/flavors/default jammy/main amd64 python3-rados amd64 19.2.3-689-gf90f4eae-1jammy [357 kB] 2026-02-22T22:33:14.830 INFO:teuthology.orchestra.run.trial023.stdout:Get:5 https://2.chacra.ceph.com/r/ceph/wip-yuri6-testing-2026-02-22-1629-squid/f90f4eaebb28b619cf76e80cbbe997656da69579/ubuntu/jammy/flavors/default jammy/main amd64 python3-ceph-argparse all 19.2.3-689-gf90f4eae-1jammy [32.9 kB] 2026-02-22T22:33:14.831 INFO:teuthology.orchestra.run.trial023.stdout:Get:6 https://2.chacra.ceph.com/r/ceph/wip-yuri6-testing-2026-02-22-1629-squid/f90f4eaebb28b619cf76e80cbbe997656da69579/ubuntu/jammy/flavors/default jammy/main amd64 python3-cephfs amd64 19.2.3-689-gf90f4eae-1jammy [184 kB] 2026-02-22T22:33:14.834 INFO:teuthology.orchestra.run.trial023.stdout:Get:7 https://2.chacra.ceph.com/r/ceph/wip-yuri6-testing-2026-02-22-1629-squid/f90f4eaebb28b619cf76e80cbbe997656da69579/ubuntu/jammy/flavors/default jammy/main amd64 python3-ceph-common all 19.2.3-689-gf90f4eae-1jammy [70.0 kB] 2026-02-22T22:33:14.835 INFO:teuthology.orchestra.run.trial023.stdout:Get:8 https://2.chacra.ceph.com/r/ceph/wip-yuri6-testing-2026-02-22-1629-squid/f90f4eaebb28b619cf76e80cbbe997656da69579/ubuntu/jammy/flavors/default jammy/main amd64 python3-rbd amd64 19.2.3-689-gf90f4eae-1jammy [335 kB] 2026-02-22T22:33:14.840 INFO:teuthology.orchestra.run.trial023.stdout:Get:9 https://2.chacra.ceph.com/r/ceph/wip-yuri6-testing-2026-02-22-1629-squid/f90f4eaebb28b619cf76e80cbbe997656da69579/ubuntu/jammy/flavors/default jammy/main amd64 librgw2 amd64 19.2.3-689-gf90f4eae-1jammy [6935 kB] 2026-02-22T22:33:14.955 INFO:teuthology.orchestra.run.trial023.stdout:Get:10 https://2.chacra.ceph.com/r/ceph/wip-yuri6-testing-2026-02-22-1629-squid/f90f4eaebb28b619cf76e80cbbe997656da69579/ubuntu/jammy/flavors/default jammy/main amd64 python3-rgw amd64 19.2.3-689-gf90f4eae-1jammy [112 kB] 2026-02-22T22:33:14.956 INFO:teuthology.orchestra.run.trial023.stdout:Get:11 https://2.chacra.ceph.com/r/ceph/wip-yuri6-testing-2026-02-22-1629-squid/f90f4eaebb28b619cf76e80cbbe997656da69579/ubuntu/jammy/flavors/default jammy/main amd64 libradosstriper1 amd64 19.2.3-689-gf90f4eae-1jammy [470 kB] 2026-02-22T22:33:14.963 INFO:teuthology.orchestra.run.trial023.stdout:Get:12 https://2.chacra.ceph.com/r/ceph/wip-yuri6-testing-2026-02-22-1629-squid/f90f4eaebb28b619cf76e80cbbe997656da69579/ubuntu/jammy/flavors/default jammy/main amd64 ceph-common amd64 19.2.3-689-gf90f4eae-1jammy [26.5 MB] 2026-02-22T22:33:14.995 INFO:teuthology.orchestra.run.trial023.stdout:Get:13 https://archive.ubuntu.com/ubuntu jammy/main amd64 liblttng-ust1 amd64 2.13.1-1ubuntu1 [190 kB] 2026-02-22T22:33:15.402 INFO:teuthology.orchestra.run.trial023.stdout:Get:14 https://2.chacra.ceph.com/r/ceph/wip-yuri6-testing-2026-02-22-1629-squid/f90f4eaebb28b619cf76e80cbbe997656da69579/ubuntu/jammy/flavors/default jammy/main amd64 ceph-base amd64 19.2.3-689-gf90f4eae-1jammy [5178 kB] 2026-02-22T22:33:15.408 INFO:teuthology.orchestra.run.trial023.stdout:Get:15 https://archive.ubuntu.com/ubuntu jammy/universe amd64 libdouble-conversion3 amd64 3.1.7-4 [39.0 kB] 2026-02-22T22:33:15.421 INFO:teuthology.orchestra.run.trial023.stdout:Get:16 https://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libpcre2-16-0 amd64 10.39-3ubuntu0.1 [203 kB] 2026-02-22T22:33:15.483 INFO:teuthology.orchestra.run.trial023.stdout:Get:17 https://2.chacra.ceph.com/r/ceph/wip-yuri6-testing-2026-02-22-1629-squid/f90f4eaebb28b619cf76e80cbbe997656da69579/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-modules-core all 19.2.3-689-gf90f4eae-1jammy [248 kB] 2026-02-22T22:33:15.486 INFO:teuthology.orchestra.run.trial023.stdout:Get:18 https://2.chacra.ceph.com/r/ceph/wip-yuri6-testing-2026-02-22-1629-squid/f90f4eaebb28b619cf76e80cbbe997656da69579/ubuntu/jammy/flavors/default jammy/main amd64 libsqlite3-mod-ceph amd64 19.2.3-689-gf90f4eae-1jammy [125 kB] 2026-02-22T22:33:15.487 INFO:teuthology.orchestra.run.trial023.stdout:Get:19 https://2.chacra.ceph.com/r/ceph/wip-yuri6-testing-2026-02-22-1629-squid/f90f4eaebb28b619cf76e80cbbe997656da69579/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr amd64 19.2.3-689-gf90f4eae-1jammy [1081 kB] 2026-02-22T22:33:15.506 INFO:teuthology.orchestra.run.trial023.stdout:Get:20 https://2.chacra.ceph.com/r/ceph/wip-yuri6-testing-2026-02-22-1629-squid/f90f4eaebb28b619cf76e80cbbe997656da69579/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mon amd64 19.2.3-689-gf90f4eae-1jammy [6239 kB] 2026-02-22T22:33:15.507 INFO:teuthology.orchestra.run.trial023.stdout:Get:21 https://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5core5a amd64 5.15.3+dfsg-2ubuntu0.2 [2006 kB] 2026-02-22T22:33:15.609 INFO:teuthology.orchestra.run.trial023.stdout:Get:22 https://2.chacra.ceph.com/r/ceph/wip-yuri6-testing-2026-02-22-1629-squid/f90f4eaebb28b619cf76e80cbbe997656da69579/ubuntu/jammy/flavors/default jammy/main amd64 ceph-osd amd64 19.2.3-689-gf90f4eae-1jammy [23.0 MB] 2026-02-22T22:33:15.753 INFO:teuthology.orchestra.run.trial023.stdout:Get:23 https://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5dbus5 amd64 5.15.3+dfsg-2ubuntu0.2 [222 kB] 2026-02-22T22:33:15.766 INFO:teuthology.orchestra.run.trial023.stdout:Get:24 https://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5network5 amd64 5.15.3+dfsg-2ubuntu0.2 [731 kB] 2026-02-22T22:33:15.798 INFO:teuthology.orchestra.run.trial023.stdout:Get:25 https://archive.ubuntu.com/ubuntu jammy/universe amd64 libthrift-0.16.0 amd64 0.16.0-2 [267 kB] 2026-02-22T22:33:15.809 INFO:teuthology.orchestra.run.trial023.stdout:Get:26 https://archive.ubuntu.com/ubuntu jammy/universe amd64 libnbd0 amd64 1.10.5-1 [71.3 kB] 2026-02-22T22:33:15.811 INFO:teuthology.orchestra.run.trial023.stdout:Get:27 https://archive.ubuntu.com/ubuntu jammy/main amd64 python3-wcwidth all 0.2.5+dfsg1-1 [21.9 kB] 2026-02-22T22:33:15.812 INFO:teuthology.orchestra.run.trial023.stdout:Get:28 https://archive.ubuntu.com/ubuntu jammy/main amd64 python3-prettytable all 2.5.0-2 [31.3 kB] 2026-02-22T22:33:15.813 INFO:teuthology.orchestra.run.trial023.stdout:Get:29 https://archive.ubuntu.com/ubuntu jammy/universe amd64 librdkafka1 amd64 1.8.0-1build1 [633 kB] 2026-02-22T22:33:15.833 INFO:teuthology.orchestra.run.trial023.stdout:Get:30 https://archive.ubuntu.com/ubuntu jammy/main amd64 libreadline-dev amd64 8.1.2-1 [166 kB] 2026-02-22T22:33:15.837 INFO:teuthology.orchestra.run.trial023.stdout:Get:31 https://archive.ubuntu.com/ubuntu jammy/main amd64 liblua5.3-dev amd64 5.3.6-1build1 [167 kB] 2026-02-22T22:33:15.842 INFO:teuthology.orchestra.run.trial023.stdout:Get:32 https://archive.ubuntu.com/ubuntu jammy/universe amd64 lua5.1 amd64 5.1.5-8.1build4 [94.6 kB] 2026-02-22T22:33:15.926 INFO:teuthology.orchestra.run.trial023.stdout:Get:33 https://archive.ubuntu.com/ubuntu jammy/universe amd64 lua-any all 27ubuntu1 [5034 B] 2026-02-22T22:33:15.927 INFO:teuthology.orchestra.run.trial023.stdout:Get:34 https://archive.ubuntu.com/ubuntu jammy/main amd64 zip amd64 3.0-12build2 [176 kB] 2026-02-22T22:33:15.928 INFO:teuthology.orchestra.run.trial023.stdout:Get:35 https://archive.ubuntu.com/ubuntu jammy-updates/main amd64 unzip amd64 6.0-26ubuntu3.2 [175 kB] 2026-02-22T22:33:15.931 INFO:teuthology.orchestra.run.trial023.stdout:Get:36 https://archive.ubuntu.com/ubuntu jammy/universe amd64 luarocks all 3.8.0+dfsg1-1 [140 kB] 2026-02-22T22:33:15.933 INFO:teuthology.orchestra.run.trial023.stdout:Get:37 https://archive.ubuntu.com/ubuntu jammy-updates/main amd64 liboath0 amd64 2.6.7-3ubuntu0.1 [41.3 kB] 2026-02-22T22:33:15.933 INFO:teuthology.orchestra.run.trial023.stdout:Get:38 https://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.functools all 3.4.0-2 [9030 B] 2026-02-22T22:33:15.934 INFO:teuthology.orchestra.run.trial023.stdout:Get:39 https://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-cheroot all 8.5.2+ds1-1ubuntu3.1 [71.1 kB] 2026-02-22T22:33:15.934 INFO:teuthology.orchestra.run.trial023.stdout:Get:40 https://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.classes all 3.2.1-3 [6452 B] 2026-02-22T22:33:15.935 INFO:teuthology.orchestra.run.trial023.stdout:Get:41 https://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.text all 3.6.0-2 [8716 B] 2026-02-22T22:33:15.989 INFO:teuthology.orchestra.run.trial023.stdout:Get:42 https://2.chacra.ceph.com/r/ceph/wip-yuri6-testing-2026-02-22-1629-squid/f90f4eaebb28b619cf76e80cbbe997656da69579/ubuntu/jammy/flavors/default jammy/main amd64 ceph amd64 19.2.3-689-gf90f4eae-1jammy [14.2 kB] 2026-02-22T22:33:15.989 INFO:teuthology.orchestra.run.trial023.stdout:Get:43 https://2.chacra.ceph.com/r/ceph/wip-yuri6-testing-2026-02-22-1629-squid/f90f4eaebb28b619cf76e80cbbe997656da69579/ubuntu/jammy/flavors/default jammy/main amd64 ceph-fuse amd64 19.2.3-689-gf90f4eae-1jammy [1173 kB] 2026-02-22T22:33:16.004 INFO:teuthology.orchestra.run.trial023.stdout:Get:44 https://2.chacra.ceph.com/r/ceph/wip-yuri6-testing-2026-02-22-1629-squid/f90f4eaebb28b619cf76e80cbbe997656da69579/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mds amd64 19.2.3-689-gf90f4eae-1jammy [2503 kB] 2026-02-22T22:33:16.021 INFO:teuthology.orchestra.run.trial023.stdout:Get:45 https://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.collections all 3.4.0-2 [11.4 kB] 2026-02-22T22:33:16.022 INFO:teuthology.orchestra.run.trial023.stdout:Get:46 https://archive.ubuntu.com/ubuntu jammy/main amd64 python3-tempora all 4.1.2-1 [14.8 kB] 2026-02-22T22:33:16.022 INFO:teuthology.orchestra.run.trial023.stdout:Get:47 https://archive.ubuntu.com/ubuntu jammy/main amd64 python3-portend all 3.0.0-1 [7240 B] 2026-02-22T22:33:16.022 INFO:teuthology.orchestra.run.trial023.stdout:Get:48 https://archive.ubuntu.com/ubuntu jammy/main amd64 python3-zc.lockfile all 2.0-1 [8980 B] 2026-02-22T22:33:16.045 INFO:teuthology.orchestra.run.trial023.stdout:Get:49 https://2.chacra.ceph.com/r/ceph/wip-yuri6-testing-2026-02-22-1629-squid/f90f4eaebb28b619cf76e80cbbe997656da69579/ubuntu/jammy/flavors/default jammy/main amd64 cephadm amd64 19.2.3-689-gf90f4eae-1jammy [798 kB] 2026-02-22T22:33:16.059 INFO:teuthology.orchestra.run.trial023.stdout:Get:50 https://2.chacra.ceph.com/r/ceph/wip-yuri6-testing-2026-02-22-1629-squid/f90f4eaebb28b619cf76e80cbbe997656da69579/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-cephadm all 19.2.3-689-gf90f4eae-1jammy [157 kB] 2026-02-22T22:33:16.061 INFO:teuthology.orchestra.run.trial023.stdout:Get:51 https://2.chacra.ceph.com/r/ceph/wip-yuri6-testing-2026-02-22-1629-squid/f90f4eaebb28b619cf76e80cbbe997656da69579/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-dashboard all 19.2.3-689-gf90f4eae-1jammy [2396 kB] 2026-02-22T22:33:16.101 INFO:teuthology.orchestra.run.trial023.stdout:Get:52 https://2.chacra.ceph.com/r/ceph/wip-yuri6-testing-2026-02-22-1629-squid/f90f4eaebb28b619cf76e80cbbe997656da69579/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-diskprediction-local all 19.2.3-689-gf90f4eae-1jammy [8625 kB] 2026-02-22T22:33:16.109 INFO:teuthology.orchestra.run.trial023.stdout:Get:53 https://archive.ubuntu.com/ubuntu jammy/main amd64 python3-cherrypy3 all 18.6.1-4 [208 kB] 2026-02-22T22:33:16.112 INFO:teuthology.orchestra.run.trial023.stdout:Get:54 https://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-natsort all 8.0.2-1 [35.3 kB] 2026-02-22T22:33:16.113 INFO:teuthology.orchestra.run.trial023.stdout:Get:55 https://archive.ubuntu.com/ubuntu jammy/main amd64 python3-logutils all 0.3.3-8 [17.6 kB] 2026-02-22T22:33:16.113 INFO:teuthology.orchestra.run.trial023.stdout:Get:56 https://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-mako all 1.1.3+ds1-2ubuntu0.1 [60.5 kB] 2026-02-22T22:33:16.113 INFO:teuthology.orchestra.run.trial023.stdout:Get:57 https://archive.ubuntu.com/ubuntu jammy/main amd64 python3-simplegeneric all 0.8.1-3 [11.3 kB] 2026-02-22T22:33:16.114 INFO:teuthology.orchestra.run.trial023.stdout:Get:58 https://archive.ubuntu.com/ubuntu jammy/main amd64 python3-singledispatch all 3.4.0.3-3 [7320 B] 2026-02-22T22:33:16.197 INFO:teuthology.orchestra.run.trial023.stdout:Get:59 https://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-webob all 1:1.8.6-1.1ubuntu0.1 [86.7 kB] 2026-02-22T22:33:16.198 INFO:teuthology.orchestra.run.trial023.stdout:Get:60 https://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-waitress all 1.4.4-1.1ubuntu1.1 [47.0 kB] 2026-02-22T22:33:16.199 INFO:teuthology.orchestra.run.trial023.stdout:Get:61 https://archive.ubuntu.com/ubuntu jammy/main amd64 python3-tempita all 0.5.2-6ubuntu1 [15.1 kB] 2026-02-22T22:33:16.199 INFO:teuthology.orchestra.run.trial023.stdout:Get:62 https://archive.ubuntu.com/ubuntu jammy/main amd64 python3-paste all 3.5.0+dfsg1-1 [456 kB] 2026-02-22T22:33:16.242 INFO:teuthology.orchestra.run.trial023.stdout:Get:63 https://2.chacra.ceph.com/r/ceph/wip-yuri6-testing-2026-02-22-1629-squid/f90f4eaebb28b619cf76e80cbbe997656da69579/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-k8sevents all 19.2.3-689-gf90f4eae-1jammy [14.3 kB] 2026-02-22T22:33:16.242 INFO:teuthology.orchestra.run.trial023.stdout:Get:64 https://2.chacra.ceph.com/r/ceph/wip-yuri6-testing-2026-02-22-1629-squid/f90f4eaebb28b619cf76e80cbbe997656da69579/ubuntu/jammy/flavors/default jammy/main amd64 ceph-test amd64 19.2.3-689-gf90f4eae-1jammy [52.1 MB] 2026-02-22T22:33:16.286 INFO:teuthology.orchestra.run.trial023.stdout:Get:65 https://archive.ubuntu.com/ubuntu jammy/main amd64 python-pastedeploy-tpl all 2.1.1-1 [4892 B] 2026-02-22T22:33:16.286 INFO:teuthology.orchestra.run.trial023.stdout:Get:66 https://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pastedeploy all 2.1.1-1 [26.6 kB] 2026-02-22T22:33:16.286 INFO:teuthology.orchestra.run.trial023.stdout:Get:67 https://archive.ubuntu.com/ubuntu jammy/main amd64 python3-webtest all 2.0.35-1 [28.5 kB] 2026-02-22T22:33:16.287 INFO:teuthology.orchestra.run.trial023.stdout:Get:68 https://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pecan all 1.3.3-4ubuntu2 [87.3 kB] 2026-02-22T22:33:16.288 INFO:teuthology.orchestra.run.trial023.stdout:Get:69 https://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-werkzeug all 2.0.2+dfsg1-1ubuntu0.22.04.3 [181 kB] 2026-02-22T22:33:16.290 INFO:teuthology.orchestra.run.trial023.stdout:Get:70 https://archive.ubuntu.com/ubuntu jammy/universe amd64 libfuse2 amd64 2.9.9-5ubuntu3 [90.3 kB] 2026-02-22T22:33:16.374 INFO:teuthology.orchestra.run.trial023.stdout:Get:71 https://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 python3-asyncssh all 2.5.0-1ubuntu0.1 [189 kB] 2026-02-22T22:33:16.376 INFO:teuthology.orchestra.run.trial023.stdout:Get:72 https://archive.ubuntu.com/ubuntu jammy/main amd64 python3-repoze.lru all 0.7-2 [12.1 kB] 2026-02-22T22:33:16.376 INFO:teuthology.orchestra.run.trial023.stdout:Get:73 https://archive.ubuntu.com/ubuntu jammy/main amd64 python3-routes all 2.5.1-1ubuntu1 [89.0 kB] 2026-02-22T22:33:16.377 INFO:teuthology.orchestra.run.trial023.stdout:Get:74 https://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-sklearn-lib amd64 0.23.2-5ubuntu6 [2058 kB] 2026-02-22T22:33:16.462 INFO:teuthology.orchestra.run.trial023.stdout:Get:75 https://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-joblib all 0.17.0-4ubuntu1 [204 kB] 2026-02-22T22:33:16.465 INFO:teuthology.orchestra.run.trial023.stdout:Get:76 https://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-threadpoolctl all 3.1.0-1 [21.3 kB] 2026-02-22T22:33:16.465 INFO:teuthology.orchestra.run.trial023.stdout:Get:77 https://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-sklearn all 0.23.2-5ubuntu6 [1829 kB] 2026-02-22T22:33:16.490 INFO:teuthology.orchestra.run.trial023.stdout:Get:78 https://archive.ubuntu.com/ubuntu jammy/main amd64 python3-cachetools all 5.0.0-1 [9722 B] 2026-02-22T22:33:16.490 INFO:teuthology.orchestra.run.trial023.stdout:Get:79 https://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-rsa all 4.8-1 [28.4 kB] 2026-02-22T22:33:16.490 INFO:teuthology.orchestra.run.trial023.stdout:Get:80 https://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-google-auth all 1.5.1-3 [35.7 kB] 2026-02-22T22:33:16.550 INFO:teuthology.orchestra.run.trial023.stdout:Get:81 https://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-requests-oauthlib all 1.3.0+ds-0.1 [18.7 kB] 2026-02-22T22:33:16.550 INFO:teuthology.orchestra.run.trial023.stdout:Get:82 https://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-websocket all 1.2.3-1 [34.7 kB] 2026-02-22T22:33:16.551 INFO:teuthology.orchestra.run.trial023.stdout:Get:83 https://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-kubernetes all 12.0.1-1ubuntu1 [353 kB] 2026-02-22T22:33:16.555 INFO:teuthology.orchestra.run.trial023.stdout:Get:84 https://archive.ubuntu.com/ubuntu jammy/main amd64 libonig5 amd64 6.9.7.1-2build1 [172 kB] 2026-02-22T22:33:16.639 INFO:teuthology.orchestra.run.trial023.stdout:Get:85 https://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libjq1 amd64 1.6-2.1ubuntu3.1 [133 kB] 2026-02-22T22:33:16.640 INFO:teuthology.orchestra.run.trial023.stdout:Get:86 https://archive.ubuntu.com/ubuntu jammy-updates/main amd64 jq amd64 1.6-2.1ubuntu3.1 [52.5 kB] 2026-02-22T22:33:16.641 INFO:teuthology.orchestra.run.trial023.stdout:Get:87 https://archive.ubuntu.com/ubuntu jammy/main amd64 socat amd64 1.7.4.1-3ubuntu4 [349 kB] 2026-02-22T22:33:16.645 INFO:teuthology.orchestra.run.trial023.stdout:Get:88 https://archive.ubuntu.com/ubuntu jammy/universe amd64 xmlstarlet amd64 1.6.1-2.1 [265 kB] 2026-02-22T22:33:16.648 INFO:teuthology.orchestra.run.trial023.stdout:Get:89 https://archive.ubuntu.com/ubuntu jammy/universe amd64 lua-socket amd64 3.0~rc1+git+ac3201d-6 [78.9 kB] 2026-02-22T22:33:16.649 INFO:teuthology.orchestra.run.trial023.stdout:Get:90 https://archive.ubuntu.com/ubuntu jammy/universe amd64 lua-sec amd64 1.0.2-1 [37.6 kB] 2026-02-22T22:33:16.727 INFO:teuthology.orchestra.run.trial023.stdout:Get:91 https://archive.ubuntu.com/ubuntu jammy-updates/main amd64 nvme-cli amd64 1.16-3ubuntu0.3 [474 kB] 2026-02-22T22:33:16.733 INFO:teuthology.orchestra.run.trial023.stdout:Get:92 https://archive.ubuntu.com/ubuntu jammy/main amd64 pkg-config amd64 0.29.2-1ubuntu3 [48.2 kB] 2026-02-22T22:33:16.733 INFO:teuthology.orchestra.run.trial023.stdout:Get:93 https://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 python-asyncssh-doc all 2.5.0-1ubuntu0.1 [309 kB] 2026-02-22T22:33:16.737 INFO:teuthology.orchestra.run.trial023.stdout:Get:94 https://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-iniconfig all 1.1.1-2 [6024 B] 2026-02-22T22:33:16.815 INFO:teuthology.orchestra.run.trial023.stdout:Get:95 https://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pastescript all 2.0.2-4 [54.6 kB] 2026-02-22T22:33:16.816 INFO:teuthology.orchestra.run.trial023.stdout:Get:96 https://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-pluggy all 0.13.0-7.1 [19.0 kB] 2026-02-22T22:33:16.816 INFO:teuthology.orchestra.run.trial023.stdout:Get:97 https://archive.ubuntu.com/ubuntu jammy/main amd64 python3-psutil amd64 5.9.0-1build1 [158 kB] 2026-02-22T22:33:16.818 INFO:teuthology.orchestra.run.trial023.stdout:Get:98 https://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-py all 1.10.0-1 [71.9 kB] 2026-02-22T22:33:16.819 INFO:teuthology.orchestra.run.trial023.stdout:Get:99 https://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-pygments all 2.11.2+dfsg-2ubuntu0.1 [750 kB] 2026-02-22T22:33:16.828 INFO:teuthology.orchestra.run.trial023.stdout:Get:100 https://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyinotify all 0.9.6-1.3 [24.8 kB] 2026-02-22T22:33:16.904 INFO:teuthology.orchestra.run.trial023.stdout:Get:101 https://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-toml all 0.10.2-1 [16.5 kB] 2026-02-22T22:33:16.904 INFO:teuthology.orchestra.run.trial023.stdout:Get:102 https://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-pytest all 6.2.5-1ubuntu2 [214 kB] 2026-02-22T22:33:16.907 INFO:teuthology.orchestra.run.trial023.stdout:Get:103 https://archive.ubuntu.com/ubuntu jammy/main amd64 python3-simplejson amd64 3.17.6-1build1 [54.7 kB] 2026-02-22T22:33:16.907 INFO:teuthology.orchestra.run.trial023.stdout:Get:104 https://archive.ubuntu.com/ubuntu jammy/universe amd64 qttranslations5-l10n all 5.15.3-1 [1983 kB] 2026-02-22T22:33:17.104 INFO:teuthology.orchestra.run.trial023.stdout:Get:105 https://2.chacra.ceph.com/r/ceph/wip-yuri6-testing-2026-02-22-1629-squid/f90f4eaebb28b619cf76e80cbbe997656da69579/ubuntu/jammy/flavors/default jammy/main amd64 ceph-volume all 19.2.3-689-gf90f4eae-1jammy [135 kB] 2026-02-22T22:33:17.105 INFO:teuthology.orchestra.run.trial023.stdout:Get:106 https://2.chacra.ceph.com/r/ceph/wip-yuri6-testing-2026-02-22-1629-squid/f90f4eaebb28b619cf76e80cbbe997656da69579/ubuntu/jammy/flavors/default jammy/main amd64 libcephfs-dev amd64 19.2.3-689-gf90f4eae-1jammy [41.0 kB] 2026-02-22T22:33:17.105 INFO:teuthology.orchestra.run.trial023.stdout:Get:107 https://2.chacra.ceph.com/r/ceph/wip-yuri6-testing-2026-02-22-1629-squid/f90f4eaebb28b619cf76e80cbbe997656da69579/ubuntu/jammy/flavors/default jammy/main amd64 radosgw amd64 19.2.3-689-gf90f4eae-1jammy [13.7 MB] 2026-02-22T22:33:17.325 INFO:teuthology.orchestra.run.trial023.stdout:Get:108 https://2.chacra.ceph.com/r/ceph/wip-yuri6-testing-2026-02-22-1629-squid/f90f4eaebb28b619cf76e80cbbe997656da69579/ubuntu/jammy/flavors/default jammy/main amd64 rbd-fuse amd64 19.2.3-689-gf90f4eae-1jammy [92.3 kB] 2026-02-22T22:33:17.654 INFO:teuthology.orchestra.run.trial023.stdout:Fetched 178 MB in 3s (64.8 MB/s) 2026-02-22T22:33:17.671 INFO:teuthology.orchestra.run.trial023.stdout:Selecting previously unselected package liblttng-ust1:amd64. 2026-02-22T22:33:17.827 INFO:teuthology.orchestra.run.trial023.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 167442 files and directories currently installed.) 2026-02-22T22:33:17.829 INFO:teuthology.orchestra.run.trial023.stdout:Preparing to unpack .../000-liblttng-ust1_2.13.1-1ubuntu1_amd64.deb ... 2026-02-22T22:33:17.830 INFO:teuthology.orchestra.run.trial023.stdout:Unpacking liblttng-ust1:amd64 (2.13.1-1ubuntu1) ... 2026-02-22T22:33:17.851 INFO:teuthology.orchestra.run.trial023.stdout:Selecting previously unselected package libdouble-conversion3:amd64. 2026-02-22T22:33:17.863 INFO:teuthology.orchestra.run.trial023.stdout:Preparing to unpack .../001-libdouble-conversion3_3.1.7-4_amd64.deb ... 2026-02-22T22:33:17.864 INFO:teuthology.orchestra.run.trial023.stdout:Unpacking libdouble-conversion3:amd64 (3.1.7-4) ... 2026-02-22T22:33:17.880 INFO:teuthology.orchestra.run.trial023.stdout:Selecting previously unselected package libpcre2-16-0:amd64. 2026-02-22T22:33:17.892 INFO:teuthology.orchestra.run.trial023.stdout:Preparing to unpack .../002-libpcre2-16-0_10.39-3ubuntu0.1_amd64.deb ... 2026-02-22T22:33:17.893 INFO:teuthology.orchestra.run.trial023.stdout:Unpacking libpcre2-16-0:amd64 (10.39-3ubuntu0.1) ... 2026-02-22T22:33:17.912 INFO:teuthology.orchestra.run.trial023.stdout:Selecting previously unselected package libqt5core5a:amd64. 2026-02-22T22:33:17.920 INFO:teuthology.orchestra.run.trial023.stdout:Preparing to unpack .../003-libqt5core5a_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2026-02-22T22:33:17.925 INFO:teuthology.orchestra.run.trial023.stdout:Unpacking libqt5core5a:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2026-02-22T22:33:17.956 INFO:teuthology.orchestra.run.trial023.stdout:Selecting previously unselected package libqt5dbus5:amd64. 2026-02-22T22:33:17.965 INFO:teuthology.orchestra.run.trial023.stdout:Preparing to unpack .../004-libqt5dbus5_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2026-02-22T22:33:17.965 INFO:teuthology.orchestra.run.trial023.stdout:Unpacking libqt5dbus5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2026-02-22T22:33:17.981 INFO:teuthology.orchestra.run.trial023.stdout:Selecting previously unselected package libqt5network5:amd64. 2026-02-22T22:33:17.989 INFO:teuthology.orchestra.run.trial023.stdout:Preparing to unpack .../005-libqt5network5_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2026-02-22T22:33:17.989 INFO:teuthology.orchestra.run.trial023.stdout:Unpacking libqt5network5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2026-02-22T22:33:18.009 INFO:teuthology.orchestra.run.trial023.stdout:Selecting previously unselected package libthrift-0.16.0:amd64. 2026-02-22T22:33:18.017 INFO:teuthology.orchestra.run.trial023.stdout:Preparing to unpack .../006-libthrift-0.16.0_0.16.0-2_amd64.deb ... 2026-02-22T22:33:18.017 INFO:teuthology.orchestra.run.trial023.stdout:Unpacking libthrift-0.16.0:amd64 (0.16.0-2) ... 2026-02-22T22:33:18.034 INFO:teuthology.orchestra.run.trial023.stdout:Selecting previously unselected package librados2. 2026-02-22T22:33:18.042 INFO:teuthology.orchestra.run.trial023.stdout:Preparing to unpack .../007-librados2_19.2.3-689-gf90f4eae-1jammy_amd64.deb ... 2026-02-22T22:33:18.042 INFO:teuthology.orchestra.run.trial023.stdout:Unpacking librados2 (19.2.3-689-gf90f4eae-1jammy) ... 2026-02-22T22:33:18.083 INFO:teuthology.orchestra.run.trial023.stdout:Selecting previously unselected package libnbd0. 2026-02-22T22:33:18.091 INFO:teuthology.orchestra.run.trial023.stdout:Preparing to unpack .../008-libnbd0_1.10.5-1_amd64.deb ... 2026-02-22T22:33:18.091 INFO:teuthology.orchestra.run.trial023.stdout:Unpacking libnbd0 (1.10.5-1) ... 2026-02-22T22:33:18.106 INFO:teuthology.orchestra.run.trial023.stdout:Selecting previously unselected package librbd1. 2026-02-22T22:33:18.114 INFO:teuthology.orchestra.run.trial023.stdout:Preparing to unpack .../009-librbd1_19.2.3-689-gf90f4eae-1jammy_amd64.deb ... 2026-02-22T22:33:18.114 INFO:teuthology.orchestra.run.trial023.stdout:Unpacking librbd1 (19.2.3-689-gf90f4eae-1jammy) ... 2026-02-22T22:33:18.156 INFO:teuthology.orchestra.run.trial023.stdout:Selecting previously unselected package libcephfs2. 2026-02-22T22:33:18.164 INFO:teuthology.orchestra.run.trial023.stdout:Preparing to unpack .../010-libcephfs2_19.2.3-689-gf90f4eae-1jammy_amd64.deb ... 2026-02-22T22:33:18.165 INFO:teuthology.orchestra.run.trial023.stdout:Unpacking libcephfs2 (19.2.3-689-gf90f4eae-1jammy) ... 2026-02-22T22:33:18.185 INFO:teuthology.orchestra.run.trial023.stdout:Selecting previously unselected package python3-rados. 2026-02-22T22:33:18.193 INFO:teuthology.orchestra.run.trial023.stdout:Preparing to unpack .../011-python3-rados_19.2.3-689-gf90f4eae-1jammy_amd64.deb ... 2026-02-22T22:33:18.193 INFO:teuthology.orchestra.run.trial023.stdout:Unpacking python3-rados (19.2.3-689-gf90f4eae-1jammy) ... 2026-02-22T22:33:18.211 INFO:teuthology.orchestra.run.trial023.stdout:Selecting previously unselected package python3-ceph-argparse. 2026-02-22T22:33:18.219 INFO:teuthology.orchestra.run.trial023.stdout:Preparing to unpack .../012-python3-ceph-argparse_19.2.3-689-gf90f4eae-1jammy_all.deb ... 2026-02-22T22:33:18.219 INFO:teuthology.orchestra.run.trial023.stdout:Unpacking python3-ceph-argparse (19.2.3-689-gf90f4eae-1jammy) ... 2026-02-22T22:33:18.233 INFO:teuthology.orchestra.run.trial023.stdout:Selecting previously unselected package python3-cephfs. 2026-02-22T22:33:18.241 INFO:teuthology.orchestra.run.trial023.stdout:Preparing to unpack .../013-python3-cephfs_19.2.3-689-gf90f4eae-1jammy_amd64.deb ... 2026-02-22T22:33:18.242 INFO:teuthology.orchestra.run.trial023.stdout:Unpacking python3-cephfs (19.2.3-689-gf90f4eae-1jammy) ... 2026-02-22T22:33:18.257 INFO:teuthology.orchestra.run.trial023.stdout:Selecting previously unselected package python3-ceph-common. 2026-02-22T22:33:18.265 INFO:teuthology.orchestra.run.trial023.stdout:Preparing to unpack .../014-python3-ceph-common_19.2.3-689-gf90f4eae-1jammy_all.deb ... 2026-02-22T22:33:18.266 INFO:teuthology.orchestra.run.trial023.stdout:Unpacking python3-ceph-common (19.2.3-689-gf90f4eae-1jammy) ... 2026-02-22T22:33:18.283 INFO:teuthology.orchestra.run.trial023.stdout:Selecting previously unselected package python3-wcwidth. 2026-02-22T22:33:18.291 INFO:teuthology.orchestra.run.trial023.stdout:Preparing to unpack .../015-python3-wcwidth_0.2.5+dfsg1-1_all.deb ... 2026-02-22T22:33:18.291 INFO:teuthology.orchestra.run.trial023.stdout:Unpacking python3-wcwidth (0.2.5+dfsg1-1) ... 2026-02-22T22:33:18.307 INFO:teuthology.orchestra.run.trial023.stdout:Selecting previously unselected package python3-prettytable. 2026-02-22T22:33:18.315 INFO:teuthology.orchestra.run.trial023.stdout:Preparing to unpack .../016-python3-prettytable_2.5.0-2_all.deb ... 2026-02-22T22:33:18.315 INFO:teuthology.orchestra.run.trial023.stdout:Unpacking python3-prettytable (2.5.0-2) ... 2026-02-22T22:33:18.329 INFO:teuthology.orchestra.run.trial023.stdout:Selecting previously unselected package python3-rbd. 2026-02-22T22:33:18.337 INFO:teuthology.orchestra.run.trial023.stdout:Preparing to unpack .../017-python3-rbd_19.2.3-689-gf90f4eae-1jammy_amd64.deb ... 2026-02-22T22:33:18.337 INFO:teuthology.orchestra.run.trial023.stdout:Unpacking python3-rbd (19.2.3-689-gf90f4eae-1jammy) ... 2026-02-22T22:33:18.355 INFO:teuthology.orchestra.run.trial023.stdout:Selecting previously unselected package librdkafka1:amd64. 2026-02-22T22:33:18.363 INFO:teuthology.orchestra.run.trial023.stdout:Preparing to unpack .../018-librdkafka1_1.8.0-1build1_amd64.deb ... 2026-02-22T22:33:18.363 INFO:teuthology.orchestra.run.trial023.stdout:Unpacking librdkafka1:amd64 (1.8.0-1build1) ... 2026-02-22T22:33:18.381 INFO:teuthology.orchestra.run.trial023.stdout:Selecting previously unselected package libreadline-dev:amd64. 2026-02-22T22:33:18.389 INFO:teuthology.orchestra.run.trial023.stdout:Preparing to unpack .../019-libreadline-dev_8.1.2-1_amd64.deb ... 2026-02-22T22:33:18.389 INFO:teuthology.orchestra.run.trial023.stdout:Unpacking libreadline-dev:amd64 (8.1.2-1) ... 2026-02-22T22:33:18.406 INFO:teuthology.orchestra.run.trial023.stdout:Selecting previously unselected package liblua5.3-dev:amd64. 2026-02-22T22:33:18.414 INFO:teuthology.orchestra.run.trial023.stdout:Preparing to unpack .../020-liblua5.3-dev_5.3.6-1build1_amd64.deb ... 2026-02-22T22:33:18.415 INFO:teuthology.orchestra.run.trial023.stdout:Unpacking liblua5.3-dev:amd64 (5.3.6-1build1) ... 2026-02-22T22:33:18.432 INFO:teuthology.orchestra.run.trial023.stdout:Selecting previously unselected package lua5.1. 2026-02-22T22:33:18.440 INFO:teuthology.orchestra.run.trial023.stdout:Preparing to unpack .../021-lua5.1_5.1.5-8.1build4_amd64.deb ... 2026-02-22T22:33:18.440 INFO:teuthology.orchestra.run.trial023.stdout:Unpacking lua5.1 (5.1.5-8.1build4) ... 2026-02-22T22:33:18.458 INFO:teuthology.orchestra.run.trial023.stdout:Selecting previously unselected package lua-any. 2026-02-22T22:33:18.466 INFO:teuthology.orchestra.run.trial023.stdout:Preparing to unpack .../022-lua-any_27ubuntu1_all.deb ... 2026-02-22T22:33:18.467 INFO:teuthology.orchestra.run.trial023.stdout:Unpacking lua-any (27ubuntu1) ... 2026-02-22T22:33:18.480 INFO:teuthology.orchestra.run.trial023.stdout:Selecting previously unselected package zip. 2026-02-22T22:33:18.488 INFO:teuthology.orchestra.run.trial023.stdout:Preparing to unpack .../023-zip_3.0-12build2_amd64.deb ... 2026-02-22T22:33:18.488 INFO:teuthology.orchestra.run.trial023.stdout:Unpacking zip (3.0-12build2) ... 2026-02-22T22:33:18.504 INFO:teuthology.orchestra.run.trial023.stdout:Selecting previously unselected package unzip. 2026-02-22T22:33:18.513 INFO:teuthology.orchestra.run.trial023.stdout:Preparing to unpack .../024-unzip_6.0-26ubuntu3.2_amd64.deb ... 2026-02-22T22:33:18.513 INFO:teuthology.orchestra.run.trial023.stdout:Unpacking unzip (6.0-26ubuntu3.2) ... 2026-02-22T22:33:18.531 INFO:teuthology.orchestra.run.trial023.stdout:Selecting previously unselected package luarocks. 2026-02-22T22:33:18.539 INFO:teuthology.orchestra.run.trial023.stdout:Preparing to unpack .../025-luarocks_3.8.0+dfsg1-1_all.deb ... 2026-02-22T22:33:18.539 INFO:teuthology.orchestra.run.trial023.stdout:Unpacking luarocks (3.8.0+dfsg1-1) ... 2026-02-22T22:33:18.577 INFO:teuthology.orchestra.run.trial023.stdout:Selecting previously unselected package librgw2. 2026-02-22T22:33:18.585 INFO:teuthology.orchestra.run.trial023.stdout:Preparing to unpack .../026-librgw2_19.2.3-689-gf90f4eae-1jammy_amd64.deb ... 2026-02-22T22:33:18.586 INFO:teuthology.orchestra.run.trial023.stdout:Unpacking librgw2 (19.2.3-689-gf90f4eae-1jammy) ... 2026-02-22T22:33:18.662 INFO:teuthology.orchestra.run.trial023.stdout:Selecting previously unselected package python3-rgw. 2026-02-22T22:33:18.670 INFO:teuthology.orchestra.run.trial023.stdout:Preparing to unpack .../027-python3-rgw_19.2.3-689-gf90f4eae-1jammy_amd64.deb ... 2026-02-22T22:33:18.671 INFO:teuthology.orchestra.run.trial023.stdout:Unpacking python3-rgw (19.2.3-689-gf90f4eae-1jammy) ... 2026-02-22T22:33:18.686 INFO:teuthology.orchestra.run.trial023.stdout:Selecting previously unselected package liboath0:amd64. 2026-02-22T22:33:18.694 INFO:teuthology.orchestra.run.trial023.stdout:Preparing to unpack .../028-liboath0_2.6.7-3ubuntu0.1_amd64.deb ... 2026-02-22T22:33:18.695 INFO:teuthology.orchestra.run.trial023.stdout:Unpacking liboath0:amd64 (2.6.7-3ubuntu0.1) ... 2026-02-22T22:33:18.710 INFO:teuthology.orchestra.run.trial023.stdout:Selecting previously unselected package libradosstriper1. 2026-02-22T22:33:18.718 INFO:teuthology.orchestra.run.trial023.stdout:Preparing to unpack .../029-libradosstriper1_19.2.3-689-gf90f4eae-1jammy_amd64.deb ... 2026-02-22T22:33:18.719 INFO:teuthology.orchestra.run.trial023.stdout:Unpacking libradosstriper1 (19.2.3-689-gf90f4eae-1jammy) ... 2026-02-22T22:33:18.737 INFO:teuthology.orchestra.run.trial023.stdout:Selecting previously unselected package ceph-common. 2026-02-22T22:33:18.745 INFO:teuthology.orchestra.run.trial023.stdout:Preparing to unpack .../030-ceph-common_19.2.3-689-gf90f4eae-1jammy_amd64.deb ... 2026-02-22T22:33:18.746 INFO:teuthology.orchestra.run.trial023.stdout:Unpacking ceph-common (19.2.3-689-gf90f4eae-1jammy) ... 2026-02-22T22:33:18.967 INFO:teuthology.orchestra.run.trial023.stdout:Selecting previously unselected package ceph-base. 2026-02-22T22:33:18.975 INFO:teuthology.orchestra.run.trial023.stdout:Preparing to unpack .../031-ceph-base_19.2.3-689-gf90f4eae-1jammy_amd64.deb ... 2026-02-22T22:33:18.980 INFO:teuthology.orchestra.run.trial023.stdout:Unpacking ceph-base (19.2.3-689-gf90f4eae-1jammy) ... 2026-02-22T22:33:19.044 INFO:teuthology.orchestra.run.trial023.stdout:Selecting previously unselected package python3-jaraco.functools. 2026-02-22T22:33:19.053 INFO:teuthology.orchestra.run.trial023.stdout:Preparing to unpack .../032-python3-jaraco.functools_3.4.0-2_all.deb ... 2026-02-22T22:33:19.054 INFO:teuthology.orchestra.run.trial023.stdout:Unpacking python3-jaraco.functools (3.4.0-2) ... 2026-02-22T22:33:19.068 INFO:teuthology.orchestra.run.trial023.stdout:Selecting previously unselected package python3-cheroot. 2026-02-22T22:33:19.076 INFO:teuthology.orchestra.run.trial023.stdout:Preparing to unpack .../033-python3-cheroot_8.5.2+ds1-1ubuntu3.1_all.deb ... 2026-02-22T22:33:19.077 INFO:teuthology.orchestra.run.trial023.stdout:Unpacking python3-cheroot (8.5.2+ds1-1ubuntu3.1) ... 2026-02-22T22:33:19.093 INFO:teuthology.orchestra.run.trial023.stdout:Selecting previously unselected package python3-jaraco.classes. 2026-02-22T22:33:19.102 INFO:teuthology.orchestra.run.trial023.stdout:Preparing to unpack .../034-python3-jaraco.classes_3.2.1-3_all.deb ... 2026-02-22T22:33:19.103 INFO:teuthology.orchestra.run.trial023.stdout:Unpacking python3-jaraco.classes (3.2.1-3) ... 2026-02-22T22:33:19.117 INFO:teuthology.orchestra.run.trial023.stdout:Selecting previously unselected package python3-jaraco.text. 2026-02-22T22:33:19.125 INFO:teuthology.orchestra.run.trial023.stdout:Preparing to unpack .../035-python3-jaraco.text_3.6.0-2_all.deb ... 2026-02-22T22:33:19.126 INFO:teuthology.orchestra.run.trial023.stdout:Unpacking python3-jaraco.text (3.6.0-2) ... 2026-02-22T22:33:19.141 INFO:teuthology.orchestra.run.trial023.stdout:Selecting previously unselected package python3-jaraco.collections. 2026-02-22T22:33:19.149 INFO:teuthology.orchestra.run.trial023.stdout:Preparing to unpack .../036-python3-jaraco.collections_3.4.0-2_all.deb ... 2026-02-22T22:33:19.150 INFO:teuthology.orchestra.run.trial023.stdout:Unpacking python3-jaraco.collections (3.4.0-2) ... 2026-02-22T22:33:19.164 INFO:teuthology.orchestra.run.trial023.stdout:Selecting previously unselected package python3-tempora. 2026-02-22T22:33:19.172 INFO:teuthology.orchestra.run.trial023.stdout:Preparing to unpack .../037-python3-tempora_4.1.2-1_all.deb ... 2026-02-22T22:33:19.173 INFO:teuthology.orchestra.run.trial023.stdout:Unpacking python3-tempora (4.1.2-1) ... 2026-02-22T22:33:19.187 INFO:teuthology.orchestra.run.trial023.stdout:Selecting previously unselected package python3-portend. 2026-02-22T22:33:19.196 INFO:teuthology.orchestra.run.trial023.stdout:Preparing to unpack .../038-python3-portend_3.0.0-1_all.deb ... 2026-02-22T22:33:19.196 INFO:teuthology.orchestra.run.trial023.stdout:Unpacking python3-portend (3.0.0-1) ... 2026-02-22T22:33:19.210 INFO:teuthology.orchestra.run.trial023.stdout:Selecting previously unselected package python3-zc.lockfile. 2026-02-22T22:33:19.219 INFO:teuthology.orchestra.run.trial023.stdout:Preparing to unpack .../039-python3-zc.lockfile_2.0-1_all.deb ... 2026-02-22T22:33:19.219 INFO:teuthology.orchestra.run.trial023.stdout:Unpacking python3-zc.lockfile (2.0-1) ... 2026-02-22T22:33:19.233 INFO:teuthology.orchestra.run.trial023.stdout:Selecting previously unselected package python3-cherrypy3. 2026-02-22T22:33:19.240 INFO:teuthology.orchestra.run.trial023.stdout:Preparing to unpack .../040-python3-cherrypy3_18.6.1-4_all.deb ... 2026-02-22T22:33:19.241 INFO:teuthology.orchestra.run.trial023.stdout:Unpacking python3-cherrypy3 (18.6.1-4) ... 2026-02-22T22:33:19.264 INFO:teuthology.orchestra.run.trial023.stdout:Selecting previously unselected package python3-natsort. 2026-02-22T22:33:19.272 INFO:teuthology.orchestra.run.trial023.stdout:Preparing to unpack .../041-python3-natsort_8.0.2-1_all.deb ... 2026-02-22T22:33:19.272 INFO:teuthology.orchestra.run.trial023.stdout:Unpacking python3-natsort (8.0.2-1) ... 2026-02-22T22:33:19.289 INFO:teuthology.orchestra.run.trial023.stdout:Selecting previously unselected package python3-logutils. 2026-02-22T22:33:19.297 INFO:teuthology.orchestra.run.trial023.stdout:Preparing to unpack .../042-python3-logutils_0.3.3-8_all.deb ... 2026-02-22T22:33:19.297 INFO:teuthology.orchestra.run.trial023.stdout:Unpacking python3-logutils (0.3.3-8) ... 2026-02-22T22:33:19.312 INFO:teuthology.orchestra.run.trial023.stdout:Selecting previously unselected package python3-mako. 2026-02-22T22:33:19.320 INFO:teuthology.orchestra.run.trial023.stdout:Preparing to unpack .../043-python3-mako_1.1.3+ds1-2ubuntu0.1_all.deb ... 2026-02-22T22:33:19.321 INFO:teuthology.orchestra.run.trial023.stdout:Unpacking python3-mako (1.1.3+ds1-2ubuntu0.1) ... 2026-02-22T22:33:19.339 INFO:teuthology.orchestra.run.trial023.stdout:Selecting previously unselected package python3-simplegeneric. 2026-02-22T22:33:19.348 INFO:teuthology.orchestra.run.trial023.stdout:Preparing to unpack .../044-python3-simplegeneric_0.8.1-3_all.deb ... 2026-02-22T22:33:19.348 INFO:teuthology.orchestra.run.trial023.stdout:Unpacking python3-simplegeneric (0.8.1-3) ... 2026-02-22T22:33:19.363 INFO:teuthology.orchestra.run.trial023.stdout:Selecting previously unselected package python3-singledispatch. 2026-02-22T22:33:19.371 INFO:teuthology.orchestra.run.trial023.stdout:Preparing to unpack .../045-python3-singledispatch_3.4.0.3-3_all.deb ... 2026-02-22T22:33:19.372 INFO:teuthology.orchestra.run.trial023.stdout:Unpacking python3-singledispatch (3.4.0.3-3) ... 2026-02-22T22:33:19.385 INFO:teuthology.orchestra.run.trial023.stdout:Selecting previously unselected package python3-webob. 2026-02-22T22:33:19.393 INFO:teuthology.orchestra.run.trial023.stdout:Preparing to unpack .../046-python3-webob_1%3a1.8.6-1.1ubuntu0.1_all.deb ... 2026-02-22T22:33:19.393 INFO:teuthology.orchestra.run.trial023.stdout:Unpacking python3-webob (1:1.8.6-1.1ubuntu0.1) ... 2026-02-22T22:33:19.409 INFO:teuthology.orchestra.run.trial023.stdout:Selecting previously unselected package python3-waitress. 2026-02-22T22:33:19.417 INFO:teuthology.orchestra.run.trial023.stdout:Preparing to unpack .../047-python3-waitress_1.4.4-1.1ubuntu1.1_all.deb ... 2026-02-22T22:33:19.419 INFO:teuthology.orchestra.run.trial023.stdout:Unpacking python3-waitress (1.4.4-1.1ubuntu1.1) ... 2026-02-22T22:33:19.436 INFO:teuthology.orchestra.run.trial023.stdout:Selecting previously unselected package python3-tempita. 2026-02-22T22:33:19.444 INFO:teuthology.orchestra.run.trial023.stdout:Preparing to unpack .../048-python3-tempita_0.5.2-6ubuntu1_all.deb ... 2026-02-22T22:33:19.445 INFO:teuthology.orchestra.run.trial023.stdout:Unpacking python3-tempita (0.5.2-6ubuntu1) ... 2026-02-22T22:33:19.460 INFO:teuthology.orchestra.run.trial023.stdout:Selecting previously unselected package python3-paste. 2026-02-22T22:33:19.468 INFO:teuthology.orchestra.run.trial023.stdout:Preparing to unpack .../049-python3-paste_3.5.0+dfsg1-1_all.deb ... 2026-02-22T22:33:19.469 INFO:teuthology.orchestra.run.trial023.stdout:Unpacking python3-paste (3.5.0+dfsg1-1) ... 2026-02-22T22:33:19.496 INFO:teuthology.orchestra.run.trial023.stdout:Selecting previously unselected package python-pastedeploy-tpl. 2026-02-22T22:33:19.505 INFO:teuthology.orchestra.run.trial023.stdout:Preparing to unpack .../050-python-pastedeploy-tpl_2.1.1-1_all.deb ... 2026-02-22T22:33:19.505 INFO:teuthology.orchestra.run.trial023.stdout:Unpacking python-pastedeploy-tpl (2.1.1-1) ... 2026-02-22T22:33:19.519 INFO:teuthology.orchestra.run.trial023.stdout:Selecting previously unselected package python3-pastedeploy. 2026-02-22T22:33:19.527 INFO:teuthology.orchestra.run.trial023.stdout:Preparing to unpack .../051-python3-pastedeploy_2.1.1-1_all.deb ... 2026-02-22T22:33:19.528 INFO:teuthology.orchestra.run.trial023.stdout:Unpacking python3-pastedeploy (2.1.1-1) ... 2026-02-22T22:33:19.543 INFO:teuthology.orchestra.run.trial023.stdout:Selecting previously unselected package python3-webtest. 2026-02-22T22:33:19.551 INFO:teuthology.orchestra.run.trial023.stdout:Preparing to unpack .../052-python3-webtest_2.0.35-1_all.deb ... 2026-02-22T22:33:19.552 INFO:teuthology.orchestra.run.trial023.stdout:Unpacking python3-webtest (2.0.35-1) ... 2026-02-22T22:33:19.567 INFO:teuthology.orchestra.run.trial023.stdout:Selecting previously unselected package python3-pecan. 2026-02-22T22:33:19.575 INFO:teuthology.orchestra.run.trial023.stdout:Preparing to unpack .../053-python3-pecan_1.3.3-4ubuntu2_all.deb ... 2026-02-22T22:33:19.576 INFO:teuthology.orchestra.run.trial023.stdout:Unpacking python3-pecan (1.3.3-4ubuntu2) ... 2026-02-22T22:33:19.600 INFO:teuthology.orchestra.run.trial023.stdout:Selecting previously unselected package python3-werkzeug. 2026-02-22T22:33:19.608 INFO:teuthology.orchestra.run.trial023.stdout:Preparing to unpack .../054-python3-werkzeug_2.0.2+dfsg1-1ubuntu0.22.04.3_all.deb ... 2026-02-22T22:33:19.608 INFO:teuthology.orchestra.run.trial023.stdout:Unpacking python3-werkzeug (2.0.2+dfsg1-1ubuntu0.22.04.3) ... 2026-02-22T22:33:19.628 INFO:teuthology.orchestra.run.trial023.stdout:Selecting previously unselected package ceph-mgr-modules-core. 2026-02-22T22:33:19.636 INFO:teuthology.orchestra.run.trial023.stdout:Preparing to unpack .../055-ceph-mgr-modules-core_19.2.3-689-gf90f4eae-1jammy_all.deb ... 2026-02-22T22:33:19.637 INFO:teuthology.orchestra.run.trial023.stdout:Unpacking ceph-mgr-modules-core (19.2.3-689-gf90f4eae-1jammy) ... 2026-02-22T22:33:19.664 INFO:teuthology.orchestra.run.trial023.stdout:Selecting previously unselected package libsqlite3-mod-ceph. 2026-02-22T22:33:19.672 INFO:teuthology.orchestra.run.trial023.stdout:Preparing to unpack .../056-libsqlite3-mod-ceph_19.2.3-689-gf90f4eae-1jammy_amd64.deb ... 2026-02-22T22:33:19.673 INFO:teuthology.orchestra.run.trial023.stdout:Unpacking libsqlite3-mod-ceph (19.2.3-689-gf90f4eae-1jammy) ... 2026-02-22T22:33:19.688 INFO:teuthology.orchestra.run.trial023.stdout:Selecting previously unselected package ceph-mgr. 2026-02-22T22:33:19.696 INFO:teuthology.orchestra.run.trial023.stdout:Preparing to unpack .../057-ceph-mgr_19.2.3-689-gf90f4eae-1jammy_amd64.deb ... 2026-02-22T22:33:19.696 INFO:teuthology.orchestra.run.trial023.stdout:Unpacking ceph-mgr (19.2.3-689-gf90f4eae-1jammy) ... 2026-02-22T22:33:19.718 INFO:teuthology.orchestra.run.trial023.stdout:Selecting previously unselected package ceph-mon. 2026-02-22T22:33:19.727 INFO:teuthology.orchestra.run.trial023.stdout:Preparing to unpack .../058-ceph-mon_19.2.3-689-gf90f4eae-1jammy_amd64.deb ... 2026-02-22T22:33:19.727 INFO:teuthology.orchestra.run.trial023.stdout:Unpacking ceph-mon (19.2.3-689-gf90f4eae-1jammy) ... 2026-02-22T22:33:19.784 INFO:teuthology.orchestra.run.trial023.stdout:Selecting previously unselected package libfuse2:amd64. 2026-02-22T22:33:19.792 INFO:teuthology.orchestra.run.trial023.stdout:Preparing to unpack .../059-libfuse2_2.9.9-5ubuntu3_amd64.deb ... 2026-02-22T22:33:19.793 INFO:teuthology.orchestra.run.trial023.stdout:Unpacking libfuse2:amd64 (2.9.9-5ubuntu3) ... 2026-02-22T22:33:19.808 INFO:teuthology.orchestra.run.trial023.stdout:Selecting previously unselected package ceph-osd. 2026-02-22T22:33:19.816 INFO:teuthology.orchestra.run.trial023.stdout:Preparing to unpack .../060-ceph-osd_19.2.3-689-gf90f4eae-1jammy_amd64.deb ... 2026-02-22T22:33:19.816 INFO:teuthology.orchestra.run.trial023.stdout:Unpacking ceph-osd (19.2.3-689-gf90f4eae-1jammy) ... 2026-02-22T22:33:19.970 INFO:teuthology.orchestra.run.trial023.stdout:Selecting previously unselected package ceph. 2026-02-22T22:33:19.980 INFO:teuthology.orchestra.run.trial023.stdout:Preparing to unpack .../061-ceph_19.2.3-689-gf90f4eae-1jammy_amd64.deb ... 2026-02-22T22:33:19.980 INFO:teuthology.orchestra.run.trial023.stdout:Unpacking ceph (19.2.3-689-gf90f4eae-1jammy) ... 2026-02-22T22:33:19.992 INFO:teuthology.orchestra.run.trial023.stdout:Selecting previously unselected package ceph-fuse. 2026-02-22T22:33:20.000 INFO:teuthology.orchestra.run.trial023.stdout:Preparing to unpack .../062-ceph-fuse_19.2.3-689-gf90f4eae-1jammy_amd64.deb ... 2026-02-22T22:33:20.001 INFO:teuthology.orchestra.run.trial023.stdout:Unpacking ceph-fuse (19.2.3-689-gf90f4eae-1jammy) ... 2026-02-22T22:33:20.024 INFO:teuthology.orchestra.run.trial023.stdout:Selecting previously unselected package ceph-mds. 2026-02-22T22:33:20.032 INFO:teuthology.orchestra.run.trial023.stdout:Preparing to unpack .../063-ceph-mds_19.2.3-689-gf90f4eae-1jammy_amd64.deb ... 2026-02-22T22:33:20.033 INFO:teuthology.orchestra.run.trial023.stdout:Unpacking ceph-mds (19.2.3-689-gf90f4eae-1jammy) ... 2026-02-22T22:33:20.067 INFO:teuthology.orchestra.run.trial023.stdout:Selecting previously unselected package cephadm. 2026-02-22T22:33:20.075 INFO:teuthology.orchestra.run.trial023.stdout:Preparing to unpack .../064-cephadm_19.2.3-689-gf90f4eae-1jammy_amd64.deb ... 2026-02-22T22:33:20.075 INFO:teuthology.orchestra.run.trial023.stdout:Unpacking cephadm (19.2.3-689-gf90f4eae-1jammy) ... 2026-02-22T22:33:20.092 INFO:teuthology.orchestra.run.trial023.stdout:Selecting previously unselected package python3-asyncssh. 2026-02-22T22:33:20.099 INFO:teuthology.orchestra.run.trial023.stdout:Preparing to unpack .../065-python3-asyncssh_2.5.0-1ubuntu0.1_all.deb ... 2026-02-22T22:33:20.099 INFO:teuthology.orchestra.run.trial023.stdout:Unpacking python3-asyncssh (2.5.0-1ubuntu0.1) ... 2026-02-22T22:33:20.121 INFO:teuthology.orchestra.run.trial023.stdout:Selecting previously unselected package ceph-mgr-cephadm. 2026-02-22T22:33:20.129 INFO:teuthology.orchestra.run.trial023.stdout:Preparing to unpack .../066-ceph-mgr-cephadm_19.2.3-689-gf90f4eae-1jammy_all.deb ... 2026-02-22T22:33:20.129 INFO:teuthology.orchestra.run.trial023.stdout:Unpacking ceph-mgr-cephadm (19.2.3-689-gf90f4eae-1jammy) ... 2026-02-22T22:33:20.149 INFO:teuthology.orchestra.run.trial023.stdout:Selecting previously unselected package python3-repoze.lru. 2026-02-22T22:33:20.158 INFO:teuthology.orchestra.run.trial023.stdout:Preparing to unpack .../067-python3-repoze.lru_0.7-2_all.deb ... 2026-02-22T22:33:20.159 INFO:teuthology.orchestra.run.trial023.stdout:Unpacking python3-repoze.lru (0.7-2) ... 2026-02-22T22:33:20.174 INFO:teuthology.orchestra.run.trial023.stdout:Selecting previously unselected package python3-routes. 2026-02-22T22:33:20.182 INFO:teuthology.orchestra.run.trial023.stdout:Preparing to unpack .../068-python3-routes_2.5.1-1ubuntu1_all.deb ... 2026-02-22T22:33:20.182 INFO:teuthology.orchestra.run.trial023.stdout:Unpacking python3-routes (2.5.1-1ubuntu1) ... 2026-02-22T22:33:20.201 INFO:teuthology.orchestra.run.trial023.stdout:Selecting previously unselected package ceph-mgr-dashboard. 2026-02-22T22:33:20.210 INFO:teuthology.orchestra.run.trial023.stdout:Preparing to unpack .../069-ceph-mgr-dashboard_19.2.3-689-gf90f4eae-1jammy_all.deb ... 2026-02-22T22:33:20.210 INFO:teuthology.orchestra.run.trial023.stdout:Unpacking ceph-mgr-dashboard (19.2.3-689-gf90f4eae-1jammy) ... 2026-02-22T22:33:20.417 INFO:teuthology.orchestra.run.trial023.stdout:Selecting previously unselected package python3-sklearn-lib:amd64. 2026-02-22T22:33:20.426 INFO:teuthology.orchestra.run.trial023.stdout:Preparing to unpack .../070-python3-sklearn-lib_0.23.2-5ubuntu6_amd64.deb ... 2026-02-22T22:33:20.427 INFO:teuthology.orchestra.run.trial023.stdout:Unpacking python3-sklearn-lib:amd64 (0.23.2-5ubuntu6) ... 2026-02-22T22:33:20.464 INFO:teuthology.orchestra.run.trial023.stdout:Selecting previously unselected package python3-joblib. 2026-02-22T22:33:20.473 INFO:teuthology.orchestra.run.trial023.stdout:Preparing to unpack .../071-python3-joblib_0.17.0-4ubuntu1_all.deb ... 2026-02-22T22:33:20.473 INFO:teuthology.orchestra.run.trial023.stdout:Unpacking python3-joblib (0.17.0-4ubuntu1) ... 2026-02-22T22:33:20.500 INFO:teuthology.orchestra.run.trial023.stdout:Selecting previously unselected package python3-threadpoolctl. 2026-02-22T22:33:20.509 INFO:teuthology.orchestra.run.trial023.stdout:Preparing to unpack .../072-python3-threadpoolctl_3.1.0-1_all.deb ... 2026-02-22T22:33:20.510 INFO:teuthology.orchestra.run.trial023.stdout:Unpacking python3-threadpoolctl (3.1.0-1) ... 2026-02-22T22:33:20.526 INFO:teuthology.orchestra.run.trial023.stdout:Selecting previously unselected package python3-sklearn. 2026-02-22T22:33:20.534 INFO:teuthology.orchestra.run.trial023.stdout:Preparing to unpack .../073-python3-sklearn_0.23.2-5ubuntu6_all.deb ... 2026-02-22T22:33:20.535 INFO:teuthology.orchestra.run.trial023.stdout:Unpacking python3-sklearn (0.23.2-5ubuntu6) ... 2026-02-22T22:33:20.618 INFO:teuthology.orchestra.run.trial023.stdout:Selecting previously unselected package ceph-mgr-diskprediction-local. 2026-02-22T22:33:20.627 INFO:teuthology.orchestra.run.trial023.stdout:Preparing to unpack .../074-ceph-mgr-diskprediction-local_19.2.3-689-gf90f4eae-1jammy_all.deb ... 2026-02-22T22:33:20.628 INFO:teuthology.orchestra.run.trial023.stdout:Unpacking ceph-mgr-diskprediction-local (19.2.3-689-gf90f4eae-1jammy) ... 2026-02-22T22:33:20.797 INFO:teuthology.orchestra.run.trial023.stdout:Selecting previously unselected package python3-cachetools. 2026-02-22T22:33:20.805 INFO:teuthology.orchestra.run.trial023.stdout:Preparing to unpack .../075-python3-cachetools_5.0.0-1_all.deb ... 2026-02-22T22:33:20.806 INFO:teuthology.orchestra.run.trial023.stdout:Unpacking python3-cachetools (5.0.0-1) ... 2026-02-22T22:33:20.821 INFO:teuthology.orchestra.run.trial023.stdout:Selecting previously unselected package python3-rsa. 2026-02-22T22:33:20.829 INFO:teuthology.orchestra.run.trial023.stdout:Preparing to unpack .../076-python3-rsa_4.8-1_all.deb ... 2026-02-22T22:33:20.830 INFO:teuthology.orchestra.run.trial023.stdout:Unpacking python3-rsa (4.8-1) ... 2026-02-22T22:33:20.847 INFO:teuthology.orchestra.run.trial023.stdout:Selecting previously unselected package python3-google-auth. 2026-02-22T22:33:20.856 INFO:teuthology.orchestra.run.trial023.stdout:Preparing to unpack .../077-python3-google-auth_1.5.1-3_all.deb ... 2026-02-22T22:33:20.856 INFO:teuthology.orchestra.run.trial023.stdout:Unpacking python3-google-auth (1.5.1-3) ... 2026-02-22T22:33:20.873 INFO:teuthology.orchestra.run.trial023.stdout:Selecting previously unselected package python3-requests-oauthlib. 2026-02-22T22:33:20.882 INFO:teuthology.orchestra.run.trial023.stdout:Preparing to unpack .../078-python3-requests-oauthlib_1.3.0+ds-0.1_all.deb ... 2026-02-22T22:33:20.882 INFO:teuthology.orchestra.run.trial023.stdout:Unpacking python3-requests-oauthlib (1.3.0+ds-0.1) ... 2026-02-22T22:33:20.898 INFO:teuthology.orchestra.run.trial023.stdout:Selecting previously unselected package python3-websocket. 2026-02-22T22:33:20.907 INFO:teuthology.orchestra.run.trial023.stdout:Preparing to unpack .../079-python3-websocket_1.2.3-1_all.deb ... 2026-02-22T22:33:20.907 INFO:teuthology.orchestra.run.trial023.stdout:Unpacking python3-websocket (1.2.3-1) ... 2026-02-22T22:33:20.924 INFO:teuthology.orchestra.run.trial023.stdout:Selecting previously unselected package python3-kubernetes. 2026-02-22T22:33:20.933 INFO:teuthology.orchestra.run.trial023.stdout:Preparing to unpack .../080-python3-kubernetes_12.0.1-1ubuntu1_all.deb ... 2026-02-22T22:33:20.934 INFO:teuthology.orchestra.run.trial023.stdout:Unpacking python3-kubernetes (12.0.1-1ubuntu1) ... 2026-02-22T22:33:21.025 INFO:teuthology.orchestra.run.trial023.stdout:Selecting previously unselected package ceph-mgr-k8sevents. 2026-02-22T22:33:21.034 INFO:teuthology.orchestra.run.trial023.stdout:Preparing to unpack .../081-ceph-mgr-k8sevents_19.2.3-689-gf90f4eae-1jammy_all.deb ... 2026-02-22T22:33:21.035 INFO:teuthology.orchestra.run.trial023.stdout:Unpacking ceph-mgr-k8sevents (19.2.3-689-gf90f4eae-1jammy) ... 2026-02-22T22:33:21.049 INFO:teuthology.orchestra.run.trial023.stdout:Selecting previously unselected package libonig5:amd64. 2026-02-22T22:33:21.057 INFO:teuthology.orchestra.run.trial023.stdout:Preparing to unpack .../082-libonig5_6.9.7.1-2build1_amd64.deb ... 2026-02-22T22:33:21.065 INFO:teuthology.orchestra.run.trial023.stdout:Unpacking libonig5:amd64 (6.9.7.1-2build1) ... 2026-02-22T22:33:21.081 INFO:teuthology.orchestra.run.trial023.stdout:Selecting previously unselected package libjq1:amd64. 2026-02-22T22:33:21.090 INFO:teuthology.orchestra.run.trial023.stdout:Preparing to unpack .../083-libjq1_1.6-2.1ubuntu3.1_amd64.deb ... 2026-02-22T22:33:21.090 INFO:teuthology.orchestra.run.trial023.stdout:Unpacking libjq1:amd64 (1.6-2.1ubuntu3.1) ... 2026-02-22T22:33:21.106 INFO:teuthology.orchestra.run.trial023.stdout:Selecting previously unselected package jq. 2026-02-22T22:33:21.115 INFO:teuthology.orchestra.run.trial023.stdout:Preparing to unpack .../084-jq_1.6-2.1ubuntu3.1_amd64.deb ... 2026-02-22T22:33:21.115 INFO:teuthology.orchestra.run.trial023.stdout:Unpacking jq (1.6-2.1ubuntu3.1) ... 2026-02-22T22:33:21.130 INFO:teuthology.orchestra.run.trial023.stdout:Selecting previously unselected package socat. 2026-02-22T22:33:21.139 INFO:teuthology.orchestra.run.trial023.stdout:Preparing to unpack .../085-socat_1.7.4.1-3ubuntu4_amd64.deb ... 2026-02-22T22:33:21.139 INFO:teuthology.orchestra.run.trial023.stdout:Unpacking socat (1.7.4.1-3ubuntu4) ... 2026-02-22T22:33:21.158 INFO:teuthology.orchestra.run.trial023.stdout:Selecting previously unselected package xmlstarlet. 2026-02-22T22:33:21.167 INFO:teuthology.orchestra.run.trial023.stdout:Preparing to unpack .../086-xmlstarlet_1.6.1-2.1_amd64.deb ... 2026-02-22T22:33:21.167 INFO:teuthology.orchestra.run.trial023.stdout:Unpacking xmlstarlet (1.6.1-2.1) ... 2026-02-22T22:33:21.204 INFO:teuthology.orchestra.run.trial023.stdout:Selecting previously unselected package ceph-test. 2026-02-22T22:33:21.212 INFO:teuthology.orchestra.run.trial023.stdout:Preparing to unpack .../087-ceph-test_19.2.3-689-gf90f4eae-1jammy_amd64.deb ... 2026-02-22T22:33:21.213 INFO:teuthology.orchestra.run.trial023.stdout:Unpacking ceph-test (19.2.3-689-gf90f4eae-1jammy) ... 2026-02-22T22:33:21.685 INFO:teuthology.orchestra.run.trial023.stdout:Selecting previously unselected package ceph-volume. 2026-02-22T22:33:21.694 INFO:teuthology.orchestra.run.trial023.stdout:Preparing to unpack .../088-ceph-volume_19.2.3-689-gf90f4eae-1jammy_all.deb ... 2026-02-22T22:33:21.694 INFO:teuthology.orchestra.run.trial023.stdout:Unpacking ceph-volume (19.2.3-689-gf90f4eae-1jammy) ... 2026-02-22T22:33:21.717 INFO:teuthology.orchestra.run.trial023.stdout:Selecting previously unselected package libcephfs-dev. 2026-02-22T22:33:21.726 INFO:teuthology.orchestra.run.trial023.stdout:Preparing to unpack .../089-libcephfs-dev_19.2.3-689-gf90f4eae-1jammy_amd64.deb ... 2026-02-22T22:33:21.726 INFO:teuthology.orchestra.run.trial023.stdout:Unpacking libcephfs-dev (19.2.3-689-gf90f4eae-1jammy) ... 2026-02-22T22:33:21.742 INFO:teuthology.orchestra.run.trial023.stdout:Selecting previously unselected package lua-socket:amd64. 2026-02-22T22:33:21.750 INFO:teuthology.orchestra.run.trial023.stdout:Preparing to unpack .../090-lua-socket_3.0~rc1+git+ac3201d-6_amd64.deb ... 2026-02-22T22:33:21.751 INFO:teuthology.orchestra.run.trial023.stdout:Unpacking lua-socket:amd64 (3.0~rc1+git+ac3201d-6) ... 2026-02-22T22:33:21.772 INFO:teuthology.orchestra.run.trial023.stdout:Selecting previously unselected package lua-sec:amd64. 2026-02-22T22:33:21.780 INFO:teuthology.orchestra.run.trial023.stdout:Preparing to unpack .../091-lua-sec_1.0.2-1_amd64.deb ... 2026-02-22T22:33:21.780 INFO:teuthology.orchestra.run.trial023.stdout:Unpacking lua-sec:amd64 (1.0.2-1) ... 2026-02-22T22:33:21.797 INFO:teuthology.orchestra.run.trial023.stdout:Selecting previously unselected package nvme-cli. 2026-02-22T22:33:21.807 INFO:teuthology.orchestra.run.trial023.stdout:Preparing to unpack .../092-nvme-cli_1.16-3ubuntu0.3_amd64.deb ... 2026-02-22T22:33:21.807 INFO:teuthology.orchestra.run.trial023.stdout:Unpacking nvme-cli (1.16-3ubuntu0.3) ... 2026-02-22T22:33:21.838 INFO:teuthology.orchestra.run.trial023.stdout:Selecting previously unselected package pkg-config. 2026-02-22T22:33:21.847 INFO:teuthology.orchestra.run.trial023.stdout:Preparing to unpack .../093-pkg-config_0.29.2-1ubuntu3_amd64.deb ... 2026-02-22T22:33:21.847 INFO:teuthology.orchestra.run.trial023.stdout:Unpacking pkg-config (0.29.2-1ubuntu3) ... 2026-02-22T22:33:21.865 INFO:teuthology.orchestra.run.trial023.stdout:Selecting previously unselected package python-asyncssh-doc. 2026-02-22T22:33:21.873 INFO:teuthology.orchestra.run.trial023.stdout:Preparing to unpack .../094-python-asyncssh-doc_2.5.0-1ubuntu0.1_all.deb ... 2026-02-22T22:33:21.874 INFO:teuthology.orchestra.run.trial023.stdout:Unpacking python-asyncssh-doc (2.5.0-1ubuntu0.1) ... 2026-02-22T22:33:21.904 INFO:teuthology.orchestra.run.trial023.stdout:Selecting previously unselected package python3-iniconfig. 2026-02-22T22:33:21.912 INFO:teuthology.orchestra.run.trial023.stdout:Preparing to unpack .../095-python3-iniconfig_1.1.1-2_all.deb ... 2026-02-22T22:33:21.913 INFO:teuthology.orchestra.run.trial023.stdout:Unpacking python3-iniconfig (1.1.1-2) ... 2026-02-22T22:33:21.927 INFO:teuthology.orchestra.run.trial023.stdout:Selecting previously unselected package python3-pastescript. 2026-02-22T22:33:21.936 INFO:teuthology.orchestra.run.trial023.stdout:Preparing to unpack .../096-python3-pastescript_2.0.2-4_all.deb ... 2026-02-22T22:33:21.936 INFO:teuthology.orchestra.run.trial023.stdout:Unpacking python3-pastescript (2.0.2-4) ... 2026-02-22T22:33:21.954 INFO:teuthology.orchestra.run.trial023.stdout:Selecting previously unselected package python3-pluggy. 2026-02-22T22:33:21.963 INFO:teuthology.orchestra.run.trial023.stdout:Preparing to unpack .../097-python3-pluggy_0.13.0-7.1_all.deb ... 2026-02-22T22:33:21.963 INFO:teuthology.orchestra.run.trial023.stdout:Unpacking python3-pluggy (0.13.0-7.1) ... 2026-02-22T22:33:21.979 INFO:teuthology.orchestra.run.trial023.stdout:Selecting previously unselected package python3-psutil. 2026-02-22T22:33:21.988 INFO:teuthology.orchestra.run.trial023.stdout:Preparing to unpack .../098-python3-psutil_5.9.0-1build1_amd64.deb ... 2026-02-22T22:33:21.988 INFO:teuthology.orchestra.run.trial023.stdout:Unpacking python3-psutil (5.9.0-1build1) ... 2026-02-22T22:33:22.006 INFO:teuthology.orchestra.run.trial023.stdout:Selecting previously unselected package python3-py. 2026-02-22T22:33:22.015 INFO:teuthology.orchestra.run.trial023.stdout:Preparing to unpack .../099-python3-py_1.10.0-1_all.deb ... 2026-02-22T22:33:22.015 INFO:teuthology.orchestra.run.trial023.stdout:Unpacking python3-py (1.10.0-1) ... 2026-02-22T22:33:22.035 INFO:teuthology.orchestra.run.trial023.stdout:Selecting previously unselected package python3-pygments. 2026-02-22T22:33:22.043 INFO:teuthology.orchestra.run.trial023.stdout:Preparing to unpack .../100-python3-pygments_2.11.2+dfsg-2ubuntu0.1_all.deb ... 2026-02-22T22:33:22.043 INFO:teuthology.orchestra.run.trial023.stdout:Unpacking python3-pygments (2.11.2+dfsg-2ubuntu0.1) ... 2026-02-22T22:33:22.086 INFO:teuthology.orchestra.run.trial023.stdout:Selecting previously unselected package python3-pyinotify. 2026-02-22T22:33:22.095 INFO:teuthology.orchestra.run.trial023.stdout:Preparing to unpack .../101-python3-pyinotify_0.9.6-1.3_all.deb ... 2026-02-22T22:33:22.095 INFO:teuthology.orchestra.run.trial023.stdout:Unpacking python3-pyinotify (0.9.6-1.3) ... 2026-02-22T22:33:22.111 INFO:teuthology.orchestra.run.trial023.stdout:Selecting previously unselected package python3-toml. 2026-02-22T22:33:22.120 INFO:teuthology.orchestra.run.trial023.stdout:Preparing to unpack .../102-python3-toml_0.10.2-1_all.deb ... 2026-02-22T22:33:22.121 INFO:teuthology.orchestra.run.trial023.stdout:Unpacking python3-toml (0.10.2-1) ... 2026-02-22T22:33:22.136 INFO:teuthology.orchestra.run.trial023.stdout:Selecting previously unselected package python3-pytest. 2026-02-22T22:33:22.144 INFO:teuthology.orchestra.run.trial023.stdout:Preparing to unpack .../103-python3-pytest_6.2.5-1ubuntu2_all.deb ... 2026-02-22T22:33:22.144 INFO:teuthology.orchestra.run.trial023.stdout:Unpacking python3-pytest (6.2.5-1ubuntu2) ... 2026-02-22T22:33:22.166 INFO:teuthology.orchestra.run.trial023.stdout:Selecting previously unselected package python3-simplejson. 2026-02-22T22:33:22.175 INFO:teuthology.orchestra.run.trial023.stdout:Preparing to unpack .../104-python3-simplejson_3.17.6-1build1_amd64.deb ... 2026-02-22T22:33:22.175 INFO:teuthology.orchestra.run.trial023.stdout:Unpacking python3-simplejson (3.17.6-1build1) ... 2026-02-22T22:33:22.192 INFO:teuthology.orchestra.run.trial023.stdout:Selecting previously unselected package qttranslations5-l10n. 2026-02-22T22:33:22.201 INFO:teuthology.orchestra.run.trial023.stdout:Preparing to unpack .../105-qttranslations5-l10n_5.15.3-1_all.deb ... 2026-02-22T22:33:22.201 INFO:teuthology.orchestra.run.trial023.stdout:Unpacking qttranslations5-l10n (5.15.3-1) ... 2026-02-22T22:33:22.268 INFO:teuthology.orchestra.run.trial023.stdout:Selecting previously unselected package radosgw. 2026-02-22T22:33:22.276 INFO:teuthology.orchestra.run.trial023.stdout:Preparing to unpack .../106-radosgw_19.2.3-689-gf90f4eae-1jammy_amd64.deb ... 2026-02-22T22:33:22.277 INFO:teuthology.orchestra.run.trial023.stdout:Unpacking radosgw (19.2.3-689-gf90f4eae-1jammy) ... 2026-02-22T22:33:22.391 INFO:teuthology.orchestra.run.trial023.stdout:Selecting previously unselected package rbd-fuse. 2026-02-22T22:33:22.400 INFO:teuthology.orchestra.run.trial023.stdout:Preparing to unpack .../107-rbd-fuse_19.2.3-689-gf90f4eae-1jammy_amd64.deb ... 2026-02-22T22:33:22.400 INFO:teuthology.orchestra.run.trial023.stdout:Unpacking rbd-fuse (19.2.3-689-gf90f4eae-1jammy) ... 2026-02-22T22:33:22.430 INFO:teuthology.orchestra.run.trial023.stdout:Setting up python3-iniconfig (1.1.1-2) ... 2026-02-22T22:33:22.499 INFO:teuthology.orchestra.run.trial023.stdout:Setting up libdouble-conversion3:amd64 (3.1.7-4) ... 2026-02-22T22:33:22.499 INFO:teuthology.orchestra.run.trial023.stdout:Setting up nvme-cli (1.16-3ubuntu0.3) ... 2026-02-22T22:33:22.569 INFO:teuthology.orchestra.run.trial023.stdout:Created symlink /etc/systemd/system/default.target.wants/nvmefc-boot-connections.service → /lib/systemd/system/nvmefc-boot-connections.service. 2026-02-22T22:33:22.832 INFO:teuthology.orchestra.run.trial023.stdout:Created symlink /etc/systemd/system/default.target.wants/nvmf-autoconnect.service → /lib/systemd/system/nvmf-autoconnect.service. 2026-02-22T22:33:23.200 INFO:teuthology.orchestra.run.trial023.stdout:nvmf-connect.target is a disabled or a static unit, not starting it. 2026-02-22T22:33:23.210 INFO:teuthology.orchestra.run.trial023.stdout:Setting up cephadm (19.2.3-689-gf90f4eae-1jammy) ... 2026-02-22T22:33:23.254 INFO:teuthology.orchestra.run.trial023.stdout:Adding system user cephadm....done 2026-02-22T22:33:23.265 INFO:teuthology.orchestra.run.trial023.stdout:Setting up python3-waitress (1.4.4-1.1ubuntu1.1) ... 2026-02-22T22:33:23.337 INFO:teuthology.orchestra.run.trial023.stdout:Setting up python3-jaraco.classes (3.2.1-3) ... 2026-02-22T22:33:23.410 INFO:teuthology.orchestra.run.trial023.stdout:Setting up python-asyncssh-doc (2.5.0-1ubuntu0.1) ... 2026-02-22T22:33:23.411 INFO:teuthology.orchestra.run.trial023.stdout:Setting up python3-jaraco.functools (3.4.0-2) ... 2026-02-22T22:33:23.483 INFO:teuthology.orchestra.run.trial023.stdout:Setting up python3-repoze.lru (0.7-2) ... 2026-02-22T22:33:23.558 INFO:teuthology.orchestra.run.trial023.stdout:Setting up liboath0:amd64 (2.6.7-3ubuntu0.1) ... 2026-02-22T22:33:23.559 INFO:teuthology.orchestra.run.trial023.stdout:Setting up python3-py (1.10.0-1) ... 2026-02-22T22:33:23.656 INFO:teuthology.orchestra.run.trial023.stdout:Setting up python3-joblib (0.17.0-4ubuntu1) ... 2026-02-22T22:33:23.781 INFO:teuthology.orchestra.run.trial023.stdout:Setting up python3-cachetools (5.0.0-1) ... 2026-02-22T22:33:23.853 INFO:teuthology.orchestra.run.trial023.stdout:Setting up unzip (6.0-26ubuntu3.2) ... 2026-02-22T22:33:23.862 INFO:teuthology.orchestra.run.trial023.stdout:Setting up python3-pyinotify (0.9.6-1.3) ... 2026-02-22T22:33:23.939 INFO:teuthology.orchestra.run.trial023.stdout:Setting up python3-threadpoolctl (3.1.0-1) ... 2026-02-22T22:33:24.013 INFO:teuthology.orchestra.run.trial023.stdout:Setting up python3-ceph-argparse (19.2.3-689-gf90f4eae-1jammy) ... 2026-02-22T22:33:24.081 INFO:teuthology.orchestra.run.trial023.stdout:Setting up python3-sklearn-lib:amd64 (0.23.2-5ubuntu6) ... 2026-02-22T22:33:24.082 INFO:teuthology.orchestra.run.trial023.stdout:Setting up libnbd0 (1.10.5-1) ... 2026-02-22T22:33:24.083 INFO:teuthology.orchestra.run.trial023.stdout:Setting up lua-socket:amd64 (3.0~rc1+git+ac3201d-6) ... 2026-02-22T22:33:24.084 INFO:teuthology.orchestra.run.trial023.stdout:Setting up libreadline-dev:amd64 (8.1.2-1) ... 2026-02-22T22:33:24.085 INFO:teuthology.orchestra.run.trial023.stdout:Setting up libfuse2:amd64 (2.9.9-5ubuntu3) ... 2026-02-22T22:33:24.086 INFO:teuthology.orchestra.run.trial023.stdout:Setting up lua5.1 (5.1.5-8.1build4) ... 2026-02-22T22:33:24.102 INFO:teuthology.orchestra.run.trial023.stdout:update-alternatives: using /usr/bin/lua5.1 to provide /usr/bin/lua (lua-interpreter) in auto mode 2026-02-22T22:33:24.104 INFO:teuthology.orchestra.run.trial023.stdout:update-alternatives: using /usr/bin/luac5.1 to provide /usr/bin/luac (lua-compiler) in auto mode 2026-02-22T22:33:24.105 INFO:teuthology.orchestra.run.trial023.stdout:Setting up libpcre2-16-0:amd64 (10.39-3ubuntu0.1) ... 2026-02-22T22:33:24.106 INFO:teuthology.orchestra.run.trial023.stdout:Setting up python3-psutil (5.9.0-1build1) ... 2026-02-22T22:33:24.227 INFO:teuthology.orchestra.run.trial023.stdout:Setting up python3-natsort (8.0.2-1) ... 2026-02-22T22:33:24.305 INFO:teuthology.orchestra.run.trial023.stdout:Setting up python3-routes (2.5.1-1ubuntu1) ... 2026-02-22T22:33:24.384 INFO:teuthology.orchestra.run.trial023.stdout:Setting up python3-simplejson (3.17.6-1build1) ... 2026-02-22T22:33:24.471 INFO:teuthology.orchestra.run.trial023.stdout:Setting up zip (3.0-12build2) ... 2026-02-22T22:33:24.472 INFO:teuthology.orchestra.run.trial023.stdout:Setting up python3-pygments (2.11.2+dfsg-2ubuntu0.1) ... 2026-02-22T22:33:24.727 INFO:teuthology.orchestra.run.trial023.stdout:Setting up python3-tempita (0.5.2-6ubuntu1) ... 2026-02-22T22:33:24.803 INFO:teuthology.orchestra.run.trial023.stdout:Setting up python-pastedeploy-tpl (2.1.1-1) ... 2026-02-22T22:33:24.804 INFO:teuthology.orchestra.run.trial023.stdout:Setting up qttranslations5-l10n (5.15.3-1) ... 2026-02-22T22:33:24.805 INFO:teuthology.orchestra.run.trial023.stdout:Setting up python3-wcwidth (0.2.5+dfsg1-1) ... 2026-02-22T22:33:24.904 INFO:teuthology.orchestra.run.trial023.stdout:Setting up python3-asyncssh (2.5.0-1ubuntu0.1) ... 2026-02-22T22:33:25.027 INFO:teuthology.orchestra.run.trial023.stdout:Setting up python3-paste (3.5.0+dfsg1-1) ... 2026-02-22T22:33:25.156 INFO:teuthology.orchestra.run.trial023.stdout:Setting up python3-cheroot (8.5.2+ds1-1ubuntu3.1) ... 2026-02-22T22:33:25.236 INFO:teuthology.orchestra.run.trial023.stdout:Setting up python3-werkzeug (2.0.2+dfsg1-1ubuntu0.22.04.3) ... 2026-02-22T22:33:25.345 INFO:teuthology.orchestra.run.trial023.stdout:Setting up python3-jaraco.text (3.6.0-2) ... 2026-02-22T22:33:25.415 INFO:teuthology.orchestra.run.trial023.stdout:Setting up socat (1.7.4.1-3ubuntu4) ... 2026-02-22T22:33:25.416 INFO:teuthology.orchestra.run.trial023.stdout:Setting up python3-ceph-common (19.2.3-689-gf90f4eae-1jammy) ... 2026-02-22T22:33:25.498 INFO:teuthology.orchestra.run.trial023.stdout:Setting up python3-sklearn (0.23.2-5ubuntu6) ... 2026-02-22T22:33:26.006 INFO:teuthology.orchestra.run.trial023.stdout:Setting up pkg-config (0.29.2-1ubuntu3) ... 2026-02-22T22:33:26.028 INFO:teuthology.orchestra.run.trial023.stdout:Setting up libqt5core5a:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2026-02-22T22:33:26.032 INFO:teuthology.orchestra.run.trial023.stdout:Setting up python3-toml (0.10.2-1) ... 2026-02-22T22:33:26.107 INFO:teuthology.orchestra.run.trial023.stdout:Setting up librdkafka1:amd64 (1.8.0-1build1) ... 2026-02-22T22:33:26.108 INFO:teuthology.orchestra.run.trial023.stdout:Setting up xmlstarlet (1.6.1-2.1) ... 2026-02-22T22:33:26.109 INFO:teuthology.orchestra.run.trial023.stdout:Setting up python3-pluggy (0.13.0-7.1) ... 2026-02-22T22:33:26.184 INFO:teuthology.orchestra.run.trial023.stdout:Setting up python3-zc.lockfile (2.0-1) ... 2026-02-22T22:33:26.259 INFO:teuthology.orchestra.run.trial023.stdout:Setting up libqt5dbus5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2026-02-22T22:33:26.259 INFO:teuthology.orchestra.run.trial023.stdout:Setting up python3-rsa (4.8-1) ... 2026-02-22T22:33:26.340 INFO:teuthology.orchestra.run.trial023.stdout:Setting up python3-singledispatch (3.4.0.3-3) ... 2026-02-22T22:33:26.412 INFO:teuthology.orchestra.run.trial023.stdout:Setting up python3-logutils (0.3.3-8) ... 2026-02-22T22:33:26.487 INFO:teuthology.orchestra.run.trial023.stdout:Setting up python3-tempora (4.1.2-1) ... 2026-02-22T22:33:26.561 INFO:teuthology.orchestra.run.trial023.stdout:Setting up python3-simplegeneric (0.8.1-3) ... 2026-02-22T22:33:26.635 INFO:teuthology.orchestra.run.trial023.stdout:Setting up python3-prettytable (2.5.0-2) ... 2026-02-22T22:33:26.713 INFO:teuthology.orchestra.run.trial023.stdout:Setting up liblttng-ust1:amd64 (2.13.1-1ubuntu1) ... 2026-02-22T22:33:26.714 INFO:teuthology.orchestra.run.trial023.stdout:Setting up python3-websocket (1.2.3-1) ... 2026-02-22T22:33:26.794 INFO:teuthology.orchestra.run.trial023.stdout:Setting up libonig5:amd64 (6.9.7.1-2build1) ... 2026-02-22T22:33:26.795 INFO:teuthology.orchestra.run.trial023.stdout:Setting up python3-requests-oauthlib (1.3.0+ds-0.1) ... 2026-02-22T22:33:26.876 INFO:teuthology.orchestra.run.trial023.stdout:Setting up python3-mako (1.1.3+ds1-2ubuntu0.1) ... 2026-02-22T22:33:26.961 INFO:teuthology.orchestra.run.trial023.stdout:Setting up python3-webob (1:1.8.6-1.1ubuntu0.1) ... 2026-02-22T22:33:27.047 INFO:teuthology.orchestra.run.trial023.stdout:Setting up python3-jaraco.collections (3.4.0-2) ... 2026-02-22T22:33:27.122 INFO:teuthology.orchestra.run.trial023.stdout:Setting up liblua5.3-dev:amd64 (5.3.6-1build1) ... 2026-02-22T22:33:27.123 INFO:teuthology.orchestra.run.trial023.stdout:Setting up lua-sec:amd64 (1.0.2-1) ... 2026-02-22T22:33:27.124 INFO:teuthology.orchestra.run.trial023.stdout:Setting up libjq1:amd64 (1.6-2.1ubuntu3.1) ... 2026-02-22T22:33:27.125 INFO:teuthology.orchestra.run.trial023.stdout:Setting up python3-pytest (6.2.5-1ubuntu2) ... 2026-02-22T22:33:27.262 INFO:teuthology.orchestra.run.trial023.stdout:Setting up python3-pastedeploy (2.1.1-1) ... 2026-02-22T22:33:27.341 INFO:teuthology.orchestra.run.trial023.stdout:Setting up lua-any (27ubuntu1) ... 2026-02-22T22:33:27.342 INFO:teuthology.orchestra.run.trial023.stdout:Setting up python3-portend (3.0.0-1) ... 2026-02-22T22:33:27.419 INFO:teuthology.orchestra.run.trial023.stdout:Setting up libqt5network5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2026-02-22T22:33:27.420 INFO:teuthology.orchestra.run.trial023.stdout:Setting up python3-google-auth (1.5.1-3) ... 2026-02-22T22:33:27.504 INFO:teuthology.orchestra.run.trial023.stdout:Setting up jq (1.6-2.1ubuntu3.1) ... 2026-02-22T22:33:27.505 INFO:teuthology.orchestra.run.trial023.stdout:Setting up python3-webtest (2.0.35-1) ... 2026-02-22T22:33:27.586 INFO:teuthology.orchestra.run.trial023.stdout:Setting up python3-cherrypy3 (18.6.1-4) ... 2026-02-22T22:33:27.725 INFO:teuthology.orchestra.run.trial023.stdout:Setting up python3-pastescript (2.0.2-4) ... 2026-02-22T22:33:27.818 INFO:teuthology.orchestra.run.trial023.stdout:Setting up python3-pecan (1.3.3-4ubuntu2) ... 2026-02-22T22:33:27.935 INFO:teuthology.orchestra.run.trial023.stdout:Setting up libthrift-0.16.0:amd64 (0.16.0-2) ... 2026-02-22T22:33:27.936 INFO:teuthology.orchestra.run.trial023.stdout:Setting up librados2 (19.2.3-689-gf90f4eae-1jammy) ... 2026-02-22T22:33:27.937 INFO:teuthology.orchestra.run.trial023.stdout:Setting up libsqlite3-mod-ceph (19.2.3-689-gf90f4eae-1jammy) ... 2026-02-22T22:33:27.938 INFO:teuthology.orchestra.run.trial023.stdout:Setting up python3-kubernetes (12.0.1-1ubuntu1) ... 2026-02-22T22:33:28.463 INFO:teuthology.orchestra.run.trial023.stdout:Setting up luarocks (3.8.0+dfsg1-1) ... 2026-02-22T22:33:28.474 INFO:teuthology.orchestra.run.trial023.stdout:Setting up libcephfs2 (19.2.3-689-gf90f4eae-1jammy) ... 2026-02-22T22:33:28.474 INFO:teuthology.orchestra.run.trial023.stdout:Setting up libradosstriper1 (19.2.3-689-gf90f4eae-1jammy) ... 2026-02-22T22:33:28.475 INFO:teuthology.orchestra.run.trial023.stdout:Setting up librbd1 (19.2.3-689-gf90f4eae-1jammy) ... 2026-02-22T22:33:28.476 INFO:teuthology.orchestra.run.trial023.stdout:Setting up ceph-mgr-modules-core (19.2.3-689-gf90f4eae-1jammy) ... 2026-02-22T22:33:28.477 INFO:teuthology.orchestra.run.trial023.stdout:Setting up ceph-fuse (19.2.3-689-gf90f4eae-1jammy) ... 2026-02-22T22:33:28.546 INFO:teuthology.orchestra.run.trial023.stdout:Created symlink /etc/systemd/system/remote-fs.target.wants/ceph-fuse.target → /lib/systemd/system/ceph-fuse.target. 2026-02-22T22:33:28.547 INFO:teuthology.orchestra.run.trial023.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-fuse.target → /lib/systemd/system/ceph-fuse.target. 2026-02-22T22:33:28.937 INFO:teuthology.orchestra.run.trial023.stdout:Setting up libcephfs-dev (19.2.3-689-gf90f4eae-1jammy) ... 2026-02-22T22:33:28.938 INFO:teuthology.orchestra.run.trial023.stdout:Setting up python3-rados (19.2.3-689-gf90f4eae-1jammy) ... 2026-02-22T22:33:28.939 INFO:teuthology.orchestra.run.trial023.stdout:Setting up librgw2 (19.2.3-689-gf90f4eae-1jammy) ... 2026-02-22T22:33:28.939 INFO:teuthology.orchestra.run.trial023.stdout:Setting up python3-rbd (19.2.3-689-gf90f4eae-1jammy) ... 2026-02-22T22:33:28.940 INFO:teuthology.orchestra.run.trial023.stdout:Setting up rbd-fuse (19.2.3-689-gf90f4eae-1jammy) ... 2026-02-22T22:33:28.941 INFO:teuthology.orchestra.run.trial023.stdout:Setting up python3-rgw (19.2.3-689-gf90f4eae-1jammy) ... 2026-02-22T22:33:28.942 INFO:teuthology.orchestra.run.trial023.stdout:Setting up python3-cephfs (19.2.3-689-gf90f4eae-1jammy) ... 2026-02-22T22:33:28.943 INFO:teuthology.orchestra.run.trial023.stdout:Setting up ceph-common (19.2.3-689-gf90f4eae-1jammy) ... 2026-02-22T22:33:28.971 INFO:teuthology.orchestra.run.trial023.stdout:Adding group ceph....done 2026-02-22T22:33:29.006 INFO:teuthology.orchestra.run.trial023.stdout:Adding system user ceph....done 2026-02-22T22:33:29.016 INFO:teuthology.orchestra.run.trial023.stdout:Setting system user ceph properties....done 2026-02-22T22:33:29.022 INFO:teuthology.orchestra.run.trial023.stdout:chown: cannot access '/var/log/ceph/*.log*': No such file or directory 2026-02-22T22:33:29.110 INFO:teuthology.orchestra.run.trial023.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /lib/systemd/system/ceph.target. 2026-02-22T22:33:29.378 INFO:teuthology.orchestra.run.trial023.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/rbdmap.service → /lib/systemd/system/rbdmap.service. 2026-02-22T22:33:29.749 INFO:teuthology.orchestra.run.trial023.stdout:Setting up ceph-test (19.2.3-689-gf90f4eae-1jammy) ... 2026-02-22T22:33:29.750 INFO:teuthology.orchestra.run.trial023.stdout:Setting up radosgw (19.2.3-689-gf90f4eae-1jammy) ... 2026-02-22T22:33:30.003 INFO:teuthology.orchestra.run.trial023.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /lib/systemd/system/ceph-radosgw.target. 2026-02-22T22:33:30.003 INFO:teuthology.orchestra.run.trial023.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /lib/systemd/system/ceph-radosgw.target. 2026-02-22T22:33:30.349 INFO:teuthology.orchestra.run.trial023.stdout:Setting up ceph-base (19.2.3-689-gf90f4eae-1jammy) ... 2026-02-22T22:33:30.440 INFO:teuthology.orchestra.run.trial023.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /lib/systemd/system/ceph-crash.service. 2026-02-22T22:33:30.811 INFO:teuthology.orchestra.run.trial023.stdout:Setting up ceph-mds (19.2.3-689-gf90f4eae-1jammy) ... 2026-02-22T22:33:30.871 INFO:teuthology.orchestra.run.trial023.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /lib/systemd/system/ceph-mds.target. 2026-02-22T22:33:30.871 INFO:teuthology.orchestra.run.trial023.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /lib/systemd/system/ceph-mds.target. 2026-02-22T22:33:31.213 INFO:teuthology.orchestra.run.trial023.stdout:Setting up ceph-mgr (19.2.3-689-gf90f4eae-1jammy) ... 2026-02-22T22:33:31.290 INFO:teuthology.orchestra.run.trial023.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /lib/systemd/system/ceph-mgr.target. 2026-02-22T22:33:31.290 INFO:teuthology.orchestra.run.trial023.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /lib/systemd/system/ceph-mgr.target. 2026-02-22T22:33:31.636 INFO:teuthology.orchestra.run.trial023.stdout:Setting up ceph-osd (19.2.3-689-gf90f4eae-1jammy) ... 2026-02-22T22:33:31.732 INFO:teuthology.orchestra.run.trial023.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /lib/systemd/system/ceph-osd.target. 2026-02-22T22:33:31.732 INFO:teuthology.orchestra.run.trial023.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /lib/systemd/system/ceph-osd.target. 2026-02-22T22:33:32.081 INFO:teuthology.orchestra.run.trial023.stdout:Setting up ceph-mgr-k8sevents (19.2.3-689-gf90f4eae-1jammy) ... 2026-02-22T22:33:32.081 INFO:teuthology.orchestra.run.trial023.stdout:Setting up ceph-mgr-diskprediction-local (19.2.3-689-gf90f4eae-1jammy) ... 2026-02-22T22:33:32.093 INFO:teuthology.orchestra.run.trial023.stdout:Setting up ceph-mon (19.2.3-689-gf90f4eae-1jammy) ... 2026-02-22T22:33:32.166 INFO:teuthology.orchestra.run.trial023.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /lib/systemd/system/ceph-mon.target. 2026-02-22T22:33:32.166 INFO:teuthology.orchestra.run.trial023.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /lib/systemd/system/ceph-mon.target. 2026-02-22T22:33:32.555 INFO:teuthology.orchestra.run.trial023.stdout:Setting up ceph-mgr-cephadm (19.2.3-689-gf90f4eae-1jammy) ... 2026-02-22T22:33:32.566 INFO:teuthology.orchestra.run.trial023.stdout:Setting up ceph (19.2.3-689-gf90f4eae-1jammy) ... 2026-02-22T22:33:32.567 INFO:teuthology.orchestra.run.trial023.stdout:Setting up ceph-mgr-dashboard (19.2.3-689-gf90f4eae-1jammy) ... 2026-02-22T22:33:32.578 INFO:teuthology.orchestra.run.trial023.stdout:Setting up ceph-volume (19.2.3-689-gf90f4eae-1jammy) ... 2026-02-22T22:33:32.691 INFO:teuthology.orchestra.run.trial023.stdout:Processing triggers for mailcap (3.70+nmu1ubuntu1) ... 2026-02-22T22:33:32.698 INFO:teuthology.orchestra.run.trial023.stdout:Processing triggers for libc-bin (2.35-0ubuntu3.13) ... 2026-02-22T22:33:32.938 INFO:teuthology.orchestra.run.trial023.stdout:Processing triggers for man-db (2.10.2-1) ... 2026-02-22T22:33:32.959 INFO:teuthology.orchestra.run.trial023.stdout:Processing triggers for install-info (6.8-4build1) ... 2026-02-22T22:33:33.447 INFO:teuthology.orchestra.run.trial023.stdout: 2026-02-22T22:33:33.447 INFO:teuthology.orchestra.run.trial023.stdout:Running kernel seems to be up-to-date. 2026-02-22T22:33:33.447 INFO:teuthology.orchestra.run.trial023.stdout: 2026-02-22T22:33:33.447 INFO:teuthology.orchestra.run.trial023.stdout:The processor microcode seems to be up-to-date. 2026-02-22T22:33:33.448 INFO:teuthology.orchestra.run.trial023.stdout: 2026-02-22T22:33:33.448 INFO:teuthology.orchestra.run.trial023.stdout:No services need to be restarted. 2026-02-22T22:33:33.448 INFO:teuthology.orchestra.run.trial023.stdout: 2026-02-22T22:33:33.448 INFO:teuthology.orchestra.run.trial023.stdout:No containers need to be restarted. 2026-02-22T22:33:33.448 INFO:teuthology.orchestra.run.trial023.stdout: 2026-02-22T22:33:33.448 INFO:teuthology.orchestra.run.trial023.stdout:No user sessions are running outdated binaries. 2026-02-22T22:33:33.449 INFO:teuthology.orchestra.run.trial023.stdout: 2026-02-22T22:33:33.449 INFO:teuthology.orchestra.run.trial023.stdout:No VM guests are running outdated hypervisor (qemu) binaries on this host. 2026-02-22T22:33:34.248 INFO:teuthology.orchestra.run.trial023.stderr:W: --force-yes is deprecated, use one of the options starting with --allow instead. 2026-02-22T22:33:34.255 DEBUG:teuthology.parallel:result is None 2026-02-22T22:33:34.255 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&sha1=f90f4eaebb28b619cf76e80cbbe997656da69579 2026-02-22T22:33:34.422 DEBUG:teuthology.orchestra.run.trial023:> dpkg-query -W -f '${Version}' ceph 2026-02-22T22:33:34.434 INFO:teuthology.orchestra.run.trial023.stdout:19.2.3-689-gf90f4eae-1jammy 2026-02-22T22:33:34.434 INFO:teuthology.packaging:The installed version of ceph is 19.2.3-689-gf90f4eae-1jammy 2026-02-22T22:33:34.435 INFO:teuthology.task.install:The correct ceph version 19.2.3-689-gf90f4eae-1jammy is installed. 2026-02-22T22:33:34.437 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2026-02-22T22:33:34.437 DEBUG:teuthology.orchestra.run.trial023:> set -ex 2026-02-22T22:33:34.437 DEBUG:teuthology.orchestra.run.trial023:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2026-02-22T22:33:34.485 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2026-02-22T22:33:34.485 DEBUG:teuthology.orchestra.run.trial023:> set -ex 2026-02-22T22:33:34.485 DEBUG:teuthology.orchestra.run.trial023:> sudo dd of=/usr/bin/daemon-helper 2026-02-22T22:33:34.533 DEBUG:teuthology.orchestra.run.trial023:> sudo chmod a=rx -- /usr/bin/daemon-helper 2026-02-22T22:33:34.581 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2026-02-22T22:33:34.582 DEBUG:teuthology.orchestra.run.trial023:> set -ex 2026-02-22T22:33:34.582 DEBUG:teuthology.orchestra.run.trial023:> sudo dd of=/usr/bin/adjust-ulimits 2026-02-22T22:33:34.631 DEBUG:teuthology.orchestra.run.trial023:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2026-02-22T22:33:34.680 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2026-02-22T22:33:34.681 DEBUG:teuthology.orchestra.run.trial023:> set -ex 2026-02-22T22:33:34.681 DEBUG:teuthology.orchestra.run.trial023:> sudo dd of=/usr/bin/stdin-killer 2026-02-22T22:33:34.731 DEBUG:teuthology.orchestra.run.trial023:> sudo chmod a=rx -- /usr/bin/stdin-killer 2026-02-22T22:33:34.781 INFO:teuthology.run_tasks:Running task ceph... 2026-02-22T22:33:34.902 INFO:tasks.ceph:Making ceph log dir writeable by non-root... 2026-02-22T22:33:34.902 DEBUG:teuthology.orchestra.run.trial023:> sudo chmod 777 /var/log/ceph 2026-02-22T22:33:34.911 INFO:tasks.ceph:Disabling ceph logrotate... 2026-02-22T22:33:34.911 DEBUG:teuthology.orchestra.run.trial023:> sudo rm -f -- /etc/logrotate.d/ceph 2026-02-22T22:33:34.960 INFO:tasks.ceph:Creating extra log directories... 2026-02-22T22:33:34.960 DEBUG:teuthology.orchestra.run.trial023:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2026-02-22T22:33:35.018 INFO:tasks.ceph:Creating ceph cluster ceph... 2026-02-22T22:33:35.018 INFO:tasks.ceph:config {'conf': {'mon': {'debug auth': 30, 'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon scrub interval': 300}, 'global': {'mon client directed command retry': 5, 'mon election default strategy': 3, 'ms inject socket failures': 5000, 'ms type': 'async'}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'osd': {'bluestore block size': 96636764160, 'bluestore compression algorithm': 'zstd', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'bluestore zero block detection': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd mclock override recovery settings': True, 'osd mclock profile': 'high_recovery_ops', 'osd mclock skip benchmark': True, 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random'}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['no reply from', 'overall HEALTH_', '\\(MON_DOWN\\)', '\\(MGR_DOWN\\)', '\\(OSDMAP_FLAGS\\)', '\\(OSD_', '\\(PG_', '\\(POOL_APP_NOT_ENABLED\\)', '\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME', '\\(MON_DOWN\\)'], 'cpu_profile': set(), 'cluster': 'ceph', 'mon_bind_msgr2': True, 'mon_bind_addrvec': True} 2026-02-22T22:33:35.018 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/mnt/teuthology/yuriw-2026-02-22_19:35:40-rados-wip-yuri6-testing-2026-02-22-1629-squid-distro-default-trial/64252', 'branch': 'wip-yuri6-testing-2026-02-22-1629-squid', 'description': 'rados/singleton/{all/rebuild-mondb mon_election/connectivity msgr-failures/few msgr/async objectstore/bluestore-comp-zstd rados supported-random-distro$/{ubuntu_latest}}', 'email': 'yweinste@redhat.com', 'first_in_suite': False, 'flavor': 'default', 'job_id': '64252', 'kernel': {'branch': 'distro', 'kdb': True, 'sha1': 'distro'}, 'ktype': 'distro', 'last_in_suite': False, 'machine_type': 'trial', 'name': 'yuriw-2026-02-22_19:35:40-rados-wip-yuri6-testing-2026-02-22-1629-squid-distro-default-trial', 'no_nested_subset': False, 'openstack': [{'volumes': {'count': 3, 'size': 10}}], 'os_type': 'ubuntu', 'os_version': '22.04', 'overrides': {'admin_socket': {'branch': 'wip-yuri6-testing-2026-02-22-1629-squid'}, 'ceph': {'conf': {'global': {'mon client directed command retry': 5, 'mon election default strategy': 3, 'ms inject socket failures': 5000, 'ms type': 'async'}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon scrub interval': 300}, 'osd': {'bluestore block size': 96636764160, 'bluestore compression algorithm': 'zstd', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'bluestore zero block detection': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd mclock override recovery settings': True, 'osd mclock profile': 'high_recovery_ops', 'osd mclock skip benchmark': True, 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random'}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME', '\\(MON_DOWN\\)'], 'sha1': 'f90f4eaebb28b619cf76e80cbbe997656da69579'}, 'ceph-deploy': {'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {}}}, 'install': {'ceph': {'flavor': 'default', 'sha1': 'f90f4eaebb28b619cf76e80cbbe997656da69579'}}, 'thrashosds': {'bdev_inject_crash': 2, 'bdev_inject_crash_probability': 0.5}, 'workunit': {'branch': 'wip-yuri6-testing-2026-02-22-1629-squid', 'sha1': 'f90f4eaebb28b619cf76e80cbbe997656da69579'}}, 'owner': 'scheduled_yuriw@soko04.front.sepia.ceph.com', 'priority': 69, 'repo': 'https://github.com/ceph/ceph-ci.git', 'roles': [['mon.a', 'mon.b', 'mon.c', 'mgr.x', 'osd.0', 'osd.1', 'osd.2', 'client.0']], 'seed': 5907, 'sha1': 'f90f4eaebb28b619cf76e80cbbe997656da69579', 'sleep_before_teardown': 0, 'subset': '111/120000', 'suite': 'rados', 'suite_branch': 'wip-yuri6-testing-2026-02-22-1629-squid', 'suite_path': '/home/teuthworker/src/github.com_ceph_ceph-c_f90f4eaebb28b619cf76e80cbbe997656da69579/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://github.com/ceph/ceph-ci.git', 'suite_sha1': 'f90f4eaebb28b619cf76e80cbbe997656da69579', 'targets': {'trial023.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBFMzy9ICbBYbIp1CetICBj9of3PshaRAUhq0K0Ih30IB4Xeslfz5yFV6XioYr4lsJZDpn6s07UkGKI6BFd5jdIY='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'branch': 'distro', 'kdb': True, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': None}, {'ceph': {'conf': {'mon': {'debug auth': 30, 'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon scrub interval': 300}, 'global': {'mon client directed command retry': 5, 'mon election default strategy': 3, 'ms inject socket failures': 5000, 'ms type': 'async'}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'osd': {'bluestore block size': 96636764160, 'bluestore compression algorithm': 'zstd', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'bluestore zero block detection': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd mclock override recovery settings': True, 'osd mclock profile': 'high_recovery_ops', 'osd mclock skip benchmark': True, 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random'}}, 'log-ignorelist': ['no reply from', 'overall HEALTH_', '\\(MON_DOWN\\)', '\\(MGR_DOWN\\)', '\\(OSDMAP_FLAGS\\)', '\\(OSD_', '\\(PG_', '\\(POOL_APP_NOT_ENABLED\\)', '\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME', '\\(MON_DOWN\\)'], 'pre-mgr-commands': ['sudo ceph config set mgr mgr_pool false --force'], 'flavor': 'default', 'fs': 'xfs', 'sha1': 'f90f4eaebb28b619cf76e80cbbe997656da69579', 'cluster': 'ceph'}}, {'full_sequential': [{'radosbench': {'clients': ['client.0'], 'time': 30}}, {'rebuild_mondb': None}, {'radosbench': {'clients': ['client.0'], 'time': 30}}]}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': []}, 'teuthology_branch': 'main', 'teuthology_sha1': 'fa17720d0088c3ac28e473468bfc79eeaff5cd38', 'timestamp': '2026-02-22_19:35:40', 'tube': 'trial', 'user': 'yuriw', 'verbose': True, 'worker_log': '/home/teuthworker/mnt/teuthology/worker_logs/dispatcher.trial.3318653'} 2026-02-22T22:33:35.018 DEBUG:teuthology.orchestra.run.trial023:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2026-02-22T22:33:35.053 DEBUG:teuthology.orchestra.run.trial023:> sudo install -d -m0777 -- /var/run/ceph 2026-02-22T22:33:35.103 DEBUG:teuthology.orchestra.run.trial023:> set -ex 2026-02-22T22:33:35.103 DEBUG:teuthology.orchestra.run.trial023:> dd if=/scratch_devs of=/dev/stdout 2026-02-22T22:33:35.150 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2026-02-22T22:33:35.150 DEBUG:teuthology.orchestra.run.trial023:> stat /dev/vg_nvme/lv_1 2026-02-22T22:33:35.197 INFO:teuthology.orchestra.run.trial023.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2026-02-22T22:33:35.198 INFO:teuthology.orchestra.run.trial023.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2026-02-22T22:33:35.198 INFO:teuthology.orchestra.run.trial023.stdout:Device: 5h/5d Inode: 912 Links: 1 2026-02-22T22:33:35.198 INFO:teuthology.orchestra.run.trial023.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2026-02-22T22:33:35.198 INFO:teuthology.orchestra.run.trial023.stdout:Access: 2026-02-22 22:32:52.219484768 +0000 2026-02-22T22:33:35.198 INFO:teuthology.orchestra.run.trial023.stdout:Modify: 2026-02-22 22:32:52.215484834 +0000 2026-02-22T22:33:35.198 INFO:teuthology.orchestra.run.trial023.stdout:Change: 2026-02-22 22:32:52.215484834 +0000 2026-02-22T22:33:35.198 INFO:teuthology.orchestra.run.trial023.stdout: Birth: - 2026-02-22T22:33:35.198 DEBUG:teuthology.orchestra.run.trial023:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2026-02-22T22:33:35.248 INFO:teuthology.orchestra.run.trial023.stderr:1+0 records in 2026-02-22T22:33:35.249 INFO:teuthology.orchestra.run.trial023.stderr:1+0 records out 2026-02-22T22:33:35.249 INFO:teuthology.orchestra.run.trial023.stderr:512 bytes copied, 0.000198018 s, 2.6 MB/s 2026-02-22T22:33:35.250 DEBUG:teuthology.orchestra.run.trial023:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2026-02-22T22:33:35.299 DEBUG:teuthology.orchestra.run.trial023:> stat /dev/vg_nvme/lv_2 2026-02-22T22:33:35.345 INFO:teuthology.orchestra.run.trial023.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2026-02-22T22:33:35.345 INFO:teuthology.orchestra.run.trial023.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2026-02-22T22:33:35.346 INFO:teuthology.orchestra.run.trial023.stdout:Device: 5h/5d Inode: 939 Links: 1 2026-02-22T22:33:35.346 INFO:teuthology.orchestra.run.trial023.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2026-02-22T22:33:35.346 INFO:teuthology.orchestra.run.trial023.stdout:Access: 2026-02-22 22:32:52.691476742 +0000 2026-02-22T22:33:35.346 INFO:teuthology.orchestra.run.trial023.stdout:Modify: 2026-02-22 22:32:52.527479530 +0000 2026-02-22T22:33:35.346 INFO:teuthology.orchestra.run.trial023.stdout:Change: 2026-02-22 22:32:52.527479530 +0000 2026-02-22T22:33:35.346 INFO:teuthology.orchestra.run.trial023.stdout: Birth: - 2026-02-22T22:33:35.346 DEBUG:teuthology.orchestra.run.trial023:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2026-02-22T22:33:35.397 INFO:teuthology.orchestra.run.trial023.stderr:1+0 records in 2026-02-22T22:33:35.397 INFO:teuthology.orchestra.run.trial023.stderr:1+0 records out 2026-02-22T22:33:35.397 INFO:teuthology.orchestra.run.trial023.stderr:512 bytes copied, 0.000181257 s, 2.8 MB/s 2026-02-22T22:33:35.398 DEBUG:teuthology.orchestra.run.trial023:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2026-02-22T22:33:35.447 DEBUG:teuthology.orchestra.run.trial023:> stat /dev/vg_nvme/lv_3 2026-02-22T22:33:35.494 INFO:teuthology.orchestra.run.trial023.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2026-02-22T22:33:35.494 INFO:teuthology.orchestra.run.trial023.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2026-02-22T22:33:35.494 INFO:teuthology.orchestra.run.trial023.stdout:Device: 5h/5d Inode: 950 Links: 1 2026-02-22T22:33:35.494 INFO:teuthology.orchestra.run.trial023.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2026-02-22T22:33:35.494 INFO:teuthology.orchestra.run.trial023.stdout:Access: 2026-02-22 22:32:52.991471642 +0000 2026-02-22T22:33:35.495 INFO:teuthology.orchestra.run.trial023.stdout:Modify: 2026-02-22 22:32:52.855473956 +0000 2026-02-22T22:33:35.495 INFO:teuthology.orchestra.run.trial023.stdout:Change: 2026-02-22 22:32:52.855473956 +0000 2026-02-22T22:33:35.495 INFO:teuthology.orchestra.run.trial023.stdout: Birth: - 2026-02-22T22:33:35.495 DEBUG:teuthology.orchestra.run.trial023:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2026-02-22T22:33:35.545 INFO:teuthology.orchestra.run.trial023.stderr:1+0 records in 2026-02-22T22:33:35.546 INFO:teuthology.orchestra.run.trial023.stderr:1+0 records out 2026-02-22T22:33:35.546 INFO:teuthology.orchestra.run.trial023.stderr:512 bytes copied, 0.000171418 s, 3.0 MB/s 2026-02-22T22:33:35.547 DEBUG:teuthology.orchestra.run.trial023:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2026-02-22T22:33:35.595 DEBUG:teuthology.orchestra.run.trial023:> stat /dev/vg_nvme/lv_4 2026-02-22T22:33:35.641 INFO:teuthology.orchestra.run.trial023.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2026-02-22T22:33:35.642 INFO:teuthology.orchestra.run.trial023.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2026-02-22T22:33:35.642 INFO:teuthology.orchestra.run.trial023.stdout:Device: 5h/5d Inode: 976 Links: 1 2026-02-22T22:33:35.642 INFO:teuthology.orchestra.run.trial023.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2026-02-22T22:33:35.642 INFO:teuthology.orchestra.run.trial023.stdout:Access: 2026-02-22 22:32:53.351465522 +0000 2026-02-22T22:33:35.642 INFO:teuthology.orchestra.run.trial023.stdout:Modify: 2026-02-22 22:32:53.159468787 +0000 2026-02-22T22:33:35.642 INFO:teuthology.orchestra.run.trial023.stdout:Change: 2026-02-22 22:32:53.159468787 +0000 2026-02-22T22:33:35.643 INFO:teuthology.orchestra.run.trial023.stdout: Birth: - 2026-02-22T22:33:35.643 DEBUG:teuthology.orchestra.run.trial023:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2026-02-22T22:33:35.692 INFO:teuthology.orchestra.run.trial023.stderr:1+0 records in 2026-02-22T22:33:35.693 INFO:teuthology.orchestra.run.trial023.stderr:1+0 records out 2026-02-22T22:33:35.693 INFO:teuthology.orchestra.run.trial023.stderr:512 bytes copied, 0.000162812 s, 3.1 MB/s 2026-02-22T22:33:35.694 DEBUG:teuthology.orchestra.run.trial023:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2026-02-22T22:33:35.743 INFO:tasks.ceph:osd dev map: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2026-02-22T22:33:35.743 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@trial023.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'}} 2026-02-22T22:33:35.743 INFO:tasks.ceph:Generating config... 2026-02-22T22:33:35.746 INFO:tasks.ceph:[mon] debug auth = 30 2026-02-22T22:33:35.746 INFO:tasks.ceph:[mon] debug mon = 20 2026-02-22T22:33:35.746 INFO:tasks.ceph:[mon] debug ms = 1 2026-02-22T22:33:35.746 INFO:tasks.ceph:[mon] debug paxos = 20 2026-02-22T22:33:35.746 INFO:tasks.ceph:[mon] mon scrub interval = 300 2026-02-22T22:33:35.746 INFO:tasks.ceph:[global] mon client directed command retry = 5 2026-02-22T22:33:35.746 INFO:tasks.ceph:[global] mon election default strategy = 3 2026-02-22T22:33:35.747 INFO:tasks.ceph:[global] ms inject socket failures = 5000 2026-02-22T22:33:35.747 INFO:tasks.ceph:[global] ms type = async 2026-02-22T22:33:35.747 INFO:tasks.ceph:[mgr] debug mgr = 20 2026-02-22T22:33:35.747 INFO:tasks.ceph:[mgr] debug ms = 1 2026-02-22T22:33:35.747 INFO:tasks.ceph:[osd] bluestore block size = 96636764160 2026-02-22T22:33:35.747 INFO:tasks.ceph:[osd] bluestore compression algorithm = zstd 2026-02-22T22:33:35.747 INFO:tasks.ceph:[osd] bluestore compression mode = aggressive 2026-02-22T22:33:35.747 INFO:tasks.ceph:[osd] bluestore fsck on mount = True 2026-02-22T22:33:35.747 INFO:tasks.ceph:[osd] bluestore zero block detection = True 2026-02-22T22:33:35.747 INFO:tasks.ceph:[osd] debug bluefs = 20 2026-02-22T22:33:35.747 INFO:tasks.ceph:[osd] debug bluestore = 20 2026-02-22T22:33:35.748 INFO:tasks.ceph:[osd] debug ms = 1 2026-02-22T22:33:35.748 INFO:tasks.ceph:[osd] debug osd = 20 2026-02-22T22:33:35.748 INFO:tasks.ceph:[osd] debug rocksdb = 10 2026-02-22T22:33:35.748 INFO:tasks.ceph:[osd] mon osd backfillfull_ratio = 0.85 2026-02-22T22:33:35.748 INFO:tasks.ceph:[osd] mon osd full ratio = 0.9 2026-02-22T22:33:35.748 INFO:tasks.ceph:[osd] mon osd nearfull ratio = 0.8 2026-02-22T22:33:35.748 INFO:tasks.ceph:[osd] osd debug verify cached snaps = True 2026-02-22T22:33:35.748 INFO:tasks.ceph:[osd] osd debug verify missing on start = True 2026-02-22T22:33:35.748 INFO:tasks.ceph:[osd] osd failsafe full ratio = 0.95 2026-02-22T22:33:35.748 INFO:tasks.ceph:[osd] osd mclock override recovery settings = True 2026-02-22T22:33:35.748 INFO:tasks.ceph:[osd] osd mclock profile = high_recovery_ops 2026-02-22T22:33:35.748 INFO:tasks.ceph:[osd] osd mclock skip benchmark = True 2026-02-22T22:33:35.748 INFO:tasks.ceph:[osd] osd objectstore = bluestore 2026-02-22T22:33:35.749 INFO:tasks.ceph:[osd] osd op queue = debug_random 2026-02-22T22:33:35.749 INFO:tasks.ceph:[osd] osd op queue cut off = debug_random 2026-02-22T22:33:35.749 INFO:tasks.ceph:Setting up mon.a... 2026-02-22T22:33:35.749 DEBUG:teuthology.orchestra.run.trial023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/ceph.keyring 2026-02-22T22:33:35.802 INFO:teuthology.orchestra.run.trial023.stdout:creating /etc/ceph/ceph.keyring 2026-02-22T22:33:35.806 DEBUG:teuthology.orchestra.run.trial023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/ceph.keyring 2026-02-22T22:33:35.873 DEBUG:teuthology.orchestra.run.trial023:> sudo chmod 0644 /etc/ceph/ceph.keyring 2026-02-22T22:33:35.922 DEBUG:tasks.ceph:Ceph mon addresses: [('mon.a', '10.20.193.23'), ('mon.b', '[v2:10.20.193.23:3301,v1:10.20.193.23:6790]'), ('mon.c', '[v2:10.20.193.23:3302,v1:10.20.193.23:6791]')] 2026-02-22T22:33:35.922 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'cephx', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '10.20.193.23,[v2:10.20.193.23:3301,v1:10.20.193.23:6790],[v2:10.20.193.23:3302,v1:10.20.193.23:6791]', 'mon client directed command retry': 5, 'mon election default strategy': 3, 'ms inject socket failures': 5000, 'ms type': 'async'}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd mclock profile': 'high_recovery_ops', 'osd mclock skip benchmark': True, 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd debug trim objects': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'bluestore block size': 96636764160, 'bluestore compression algorithm': 'zstd', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'bluestore zero block detection': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd mclock override recovery settings': True, 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random'}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false'}, 'mon': {'debug ms': 1, 'debug mon': 20, 'debug paxos': 20, 'debug auth': 30, 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'mon_down_mkfs_grace': '2m', 'mon_warn_on_filestore_osds': 'false', 'mon scrub interval': 300}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok'}, 'mon.a': {}, 'mon.b': {}, 'mon.c': {}} 2026-02-22T22:33:35.924 DEBUG:teuthology.orchestra.run.trial023:> set -ex 2026-02-22T22:33:35.924 DEBUG:teuthology.orchestra.run.trial023:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2026-02-22T22:33:35.970 DEBUG:teuthology.orchestra.run.trial023:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --add a 10.20.193.23 --addv b '[v2:10.20.193.23:3301,v1:10.20.193.23:6790]' --addv c '[v2:10.20.193.23:3302,v1:10.20.193.23:6791]' --print /home/ubuntu/cephtest/ceph.monmap 2026-02-22T22:33:36.032 INFO:teuthology.orchestra.run.trial023.stdout:monmaptool: monmap file /home/ubuntu/cephtest/ceph.monmap 2026-02-22T22:33:36.032 INFO:teuthology.orchestra.run.trial023.stdout:monmaptool: generated fsid a296be36-532d-4148-b28e-0183b3968057 2026-02-22T22:33:36.032 INFO:teuthology.orchestra.run.trial023.stdout:setting min_mon_release = quincy 2026-02-22T22:33:36.032 INFO:teuthology.orchestra.run.trial023.stdout:epoch 0 2026-02-22T22:33:36.032 INFO:teuthology.orchestra.run.trial023.stdout:fsid a296be36-532d-4148-b28e-0183b3968057 2026-02-22T22:33:36.032 INFO:teuthology.orchestra.run.trial023.stdout:last_changed 2026-02-22T22:33:36.033736+0000 2026-02-22T22:33:36.032 INFO:teuthology.orchestra.run.trial023.stdout:created 2026-02-22T22:33:36.033736+0000 2026-02-22T22:33:36.033 INFO:teuthology.orchestra.run.trial023.stdout:min_mon_release 17 (quincy) 2026-02-22T22:33:36.033 INFO:teuthology.orchestra.run.trial023.stdout:election_strategy: 3 2026-02-22T22:33:36.033 INFO:teuthology.orchestra.run.trial023.stdout:0: [v2:10.20.193.23:3300/0,v1:10.20.193.23:6789/0] mon.a 2026-02-22T22:33:36.033 INFO:teuthology.orchestra.run.trial023.stdout:1: [v2:10.20.193.23:3301/0,v1:10.20.193.23:6790/0] mon.b 2026-02-22T22:33:36.033 INFO:teuthology.orchestra.run.trial023.stdout:2: [v2:10.20.193.23:3302/0,v1:10.20.193.23:6791/0] mon.c 2026-02-22T22:33:36.033 INFO:teuthology.orchestra.run.trial023.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/ceph.monmap (3 monitors) 2026-02-22T22:33:36.034 DEBUG:teuthology.orchestra.run.trial023:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2026-02-22T22:33:36.085 INFO:tasks.ceph:Writing /etc/ceph/ceph.conf for FSID a296be36-532d-4148-b28e-0183b3968057... 2026-02-22T22:33:36.086 DEBUG:teuthology.orchestra.run.trial023:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2026-02-22T22:33:36.144 INFO:teuthology.orchestra.run.trial023.stdout:[global] 2026-02-22T22:33:36.144 INFO:teuthology.orchestra.run.trial023.stdout: chdir = "" 2026-02-22T22:33:36.144 INFO:teuthology.orchestra.run.trial023.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2026-02-22T22:33:36.144 INFO:teuthology.orchestra.run.trial023.stdout: auth supported = cephx 2026-02-22T22:33:36.144 INFO:teuthology.orchestra.run.trial023.stdout: 2026-02-22T22:33:36.144 INFO:teuthology.orchestra.run.trial023.stdout: filestore xattr use omap = true 2026-02-22T22:33:36.145 INFO:teuthology.orchestra.run.trial023.stdout: 2026-02-22T22:33:36.145 INFO:teuthology.orchestra.run.trial023.stdout: mon clock drift allowed = 1.000 2026-02-22T22:33:36.145 INFO:teuthology.orchestra.run.trial023.stdout: 2026-02-22T22:33:36.145 INFO:teuthology.orchestra.run.trial023.stdout: osd crush chooseleaf type = 0 2026-02-22T22:33:36.145 INFO:teuthology.orchestra.run.trial023.stdout: auth debug = true 2026-02-22T22:33:36.145 INFO:teuthology.orchestra.run.trial023.stdout: 2026-02-22T22:33:36.145 INFO:teuthology.orchestra.run.trial023.stdout: ms die on old message = true 2026-02-22T22:33:36.145 INFO:teuthology.orchestra.run.trial023.stdout: ms die on bug = true 2026-02-22T22:33:36.145 INFO:teuthology.orchestra.run.trial023.stdout: 2026-02-22T22:33:36.145 INFO:teuthology.orchestra.run.trial023.stdout: mon max pg per osd = 10000 # >= luminous 2026-02-22T22:33:36.145 INFO:teuthology.orchestra.run.trial023.stdout: mon pg warn max object skew = 0 2026-02-22T22:33:36.145 INFO:teuthology.orchestra.run.trial023.stdout: 2026-02-22T22:33:36.145 INFO:teuthology.orchestra.run.trial023.stdout: # disable pg_autoscaler by default for new pools 2026-02-22T22:33:36.145 INFO:teuthology.orchestra.run.trial023.stdout: osd_pool_default_pg_autoscale_mode = off 2026-02-22T22:33:36.145 INFO:teuthology.orchestra.run.trial023.stdout: 2026-02-22T22:33:36.145 INFO:teuthology.orchestra.run.trial023.stdout: osd pool default size = 2 2026-02-22T22:33:36.145 INFO:teuthology.orchestra.run.trial023.stdout: 2026-02-22T22:33:36.145 INFO:teuthology.orchestra.run.trial023.stdout: mon osd allow primary affinity = true 2026-02-22T22:33:36.145 INFO:teuthology.orchestra.run.trial023.stdout: mon osd allow pg remap = true 2026-02-22T22:33:36.145 INFO:teuthology.orchestra.run.trial023.stdout: mon warn on legacy crush tunables = false 2026-02-22T22:33:36.145 INFO:teuthology.orchestra.run.trial023.stdout: mon warn on crush straw calc version zero = false 2026-02-22T22:33:36.145 INFO:teuthology.orchestra.run.trial023.stdout: mon warn on no sortbitwise = false 2026-02-22T22:33:36.145 INFO:teuthology.orchestra.run.trial023.stdout: mon warn on osd down out interval zero = false 2026-02-22T22:33:36.146 INFO:teuthology.orchestra.run.trial023.stdout: mon warn on too few osds = false 2026-02-22T22:33:36.146 INFO:teuthology.orchestra.run.trial023.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2026-02-22T22:33:36.146 INFO:teuthology.orchestra.run.trial023.stdout: mon_warn_on_pool_no_redundancy = false 2026-02-22T22:33:36.146 INFO:teuthology.orchestra.run.trial023.stdout: mon_allow_pool_size_one = true 2026-02-22T22:33:36.146 INFO:teuthology.orchestra.run.trial023.stdout: 2026-02-22T22:33:36.146 INFO:teuthology.orchestra.run.trial023.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2026-02-22T22:33:36.146 INFO:teuthology.orchestra.run.trial023.stdout: 2026-02-22T22:33:36.146 INFO:teuthology.orchestra.run.trial023.stdout: osd default data pool replay window = 5 2026-02-22T22:33:36.146 INFO:teuthology.orchestra.run.trial023.stdout: 2026-02-22T22:33:36.146 INFO:teuthology.orchestra.run.trial023.stdout: mon allow pool delete = true 2026-02-22T22:33:36.146 INFO:teuthology.orchestra.run.trial023.stdout: 2026-02-22T22:33:36.146 INFO:teuthology.orchestra.run.trial023.stdout: mon cluster log file level = debug 2026-02-22T22:33:36.146 INFO:teuthology.orchestra.run.trial023.stdout: debug asserts on shutdown = true 2026-02-22T22:33:36.146 INFO:teuthology.orchestra.run.trial023.stdout: mon health detail to clog = false 2026-02-22T22:33:36.146 INFO:teuthology.orchestra.run.trial023.stdout: mon host = "10.20.193.23,[v2:10.20.193.23:3301,v1:10.20.193.23:6790],[v2:10.20.193.23:3302,v1:10.20.193.23:6791]" 2026-02-22T22:33:36.146 INFO:teuthology.orchestra.run.trial023.stdout: mon client directed command retry = 5 2026-02-22T22:33:36.146 INFO:teuthology.orchestra.run.trial023.stdout: mon election default strategy = 3 2026-02-22T22:33:36.146 INFO:teuthology.orchestra.run.trial023.stdout: ms inject socket failures = 5000 2026-02-22T22:33:36.146 INFO:teuthology.orchestra.run.trial023.stdout: ms type = async 2026-02-22T22:33:36.146 INFO:teuthology.orchestra.run.trial023.stdout: fsid = a296be36-532d-4148-b28e-0183b3968057 2026-02-22T22:33:36.146 INFO:teuthology.orchestra.run.trial023.stdout: 2026-02-22T22:33:36.146 INFO:teuthology.orchestra.run.trial023.stdout:[osd] 2026-02-22T22:33:36.146 INFO:teuthology.orchestra.run.trial023.stdout: osd journal size = 100 2026-02-22T22:33:36.146 INFO:teuthology.orchestra.run.trial023.stdout: 2026-02-22T22:33:36.146 INFO:teuthology.orchestra.run.trial023.stdout: osd scrub load threshold = 5.0 2026-02-22T22:33:36.147 INFO:teuthology.orchestra.run.trial023.stdout: osd scrub max interval = 600 2026-02-22T22:33:36.147 INFO:teuthology.orchestra.run.trial023.stdout: osd mclock profile = high_recovery_ops 2026-02-22T22:33:36.147 INFO:teuthology.orchestra.run.trial023.stdout: osd mclock skip benchmark = True 2026-02-22T22:33:36.147 INFO:teuthology.orchestra.run.trial023.stdout: 2026-02-22T22:33:36.147 INFO:teuthology.orchestra.run.trial023.stdout: osd recover clone overlap = true 2026-02-22T22:33:36.147 INFO:teuthology.orchestra.run.trial023.stdout: osd recovery max chunk = 1048576 2026-02-22T22:33:36.147 INFO:teuthology.orchestra.run.trial023.stdout: 2026-02-22T22:33:36.147 INFO:teuthology.orchestra.run.trial023.stdout: osd debug shutdown = true 2026-02-22T22:33:36.147 INFO:teuthology.orchestra.run.trial023.stdout: osd debug op order = true 2026-02-22T22:33:36.147 INFO:teuthology.orchestra.run.trial023.stdout: osd debug verify stray on activate = true 2026-02-22T22:33:36.147 INFO:teuthology.orchestra.run.trial023.stdout: osd debug trim objects = true 2026-02-22T22:33:36.147 INFO:teuthology.orchestra.run.trial023.stdout: 2026-02-22T22:33:36.147 INFO:teuthology.orchestra.run.trial023.stdout: osd open classes on start = true 2026-02-22T22:33:36.147 INFO:teuthology.orchestra.run.trial023.stdout: osd debug pg log writeout = true 2026-02-22T22:33:36.147 INFO:teuthology.orchestra.run.trial023.stdout: 2026-02-22T22:33:36.147 INFO:teuthology.orchestra.run.trial023.stdout: osd deep scrub update digest min age = 30 2026-02-22T22:33:36.147 INFO:teuthology.orchestra.run.trial023.stdout: 2026-02-22T22:33:36.147 INFO:teuthology.orchestra.run.trial023.stdout: osd map max advance = 10 2026-02-22T22:33:36.147 INFO:teuthology.orchestra.run.trial023.stdout: 2026-02-22T22:33:36.147 INFO:teuthology.orchestra.run.trial023.stdout: journal zero on create = true 2026-02-22T22:33:36.147 INFO:teuthology.orchestra.run.trial023.stdout: 2026-02-22T22:33:36.147 INFO:teuthology.orchestra.run.trial023.stdout: filestore ondisk finisher threads = 3 2026-02-22T22:33:36.147 INFO:teuthology.orchestra.run.trial023.stdout: filestore apply finisher threads = 3 2026-02-22T22:33:36.147 INFO:teuthology.orchestra.run.trial023.stdout: 2026-02-22T22:33:36.147 INFO:teuthology.orchestra.run.trial023.stdout: bdev debug aio = true 2026-02-22T22:33:36.148 INFO:teuthology.orchestra.run.trial023.stdout: osd debug misdirected ops = true 2026-02-22T22:33:36.148 INFO:teuthology.orchestra.run.trial023.stdout: bluestore block size = 96636764160 2026-02-22T22:33:36.148 INFO:teuthology.orchestra.run.trial023.stdout: bluestore compression algorithm = zstd 2026-02-22T22:33:36.148 INFO:teuthology.orchestra.run.trial023.stdout: bluestore compression mode = aggressive 2026-02-22T22:33:36.148 INFO:teuthology.orchestra.run.trial023.stdout: bluestore fsck on mount = True 2026-02-22T22:33:36.148 INFO:teuthology.orchestra.run.trial023.stdout: bluestore zero block detection = True 2026-02-22T22:33:36.148 INFO:teuthology.orchestra.run.trial023.stdout: debug bluefs = 20 2026-02-22T22:33:36.148 INFO:teuthology.orchestra.run.trial023.stdout: debug bluestore = 20 2026-02-22T22:33:36.148 INFO:teuthology.orchestra.run.trial023.stdout: debug ms = 1 2026-02-22T22:33:36.148 INFO:teuthology.orchestra.run.trial023.stdout: debug osd = 20 2026-02-22T22:33:36.148 INFO:teuthology.orchestra.run.trial023.stdout: debug rocksdb = 10 2026-02-22T22:33:36.148 INFO:teuthology.orchestra.run.trial023.stdout: mon osd backfillfull_ratio = 0.85 2026-02-22T22:33:36.148 INFO:teuthology.orchestra.run.trial023.stdout: mon osd full ratio = 0.9 2026-02-22T22:33:36.148 INFO:teuthology.orchestra.run.trial023.stdout: mon osd nearfull ratio = 0.8 2026-02-22T22:33:36.148 INFO:teuthology.orchestra.run.trial023.stdout: osd debug verify cached snaps = True 2026-02-22T22:33:36.148 INFO:teuthology.orchestra.run.trial023.stdout: osd debug verify missing on start = True 2026-02-22T22:33:36.148 INFO:teuthology.orchestra.run.trial023.stdout: osd failsafe full ratio = 0.95 2026-02-22T22:33:36.148 INFO:teuthology.orchestra.run.trial023.stdout: osd mclock override recovery settings = True 2026-02-22T22:33:36.148 INFO:teuthology.orchestra.run.trial023.stdout: osd objectstore = bluestore 2026-02-22T22:33:36.148 INFO:teuthology.orchestra.run.trial023.stdout: osd op queue = debug_random 2026-02-22T22:33:36.148 INFO:teuthology.orchestra.run.trial023.stdout: osd op queue cut off = debug_random 2026-02-22T22:33:36.148 INFO:teuthology.orchestra.run.trial023.stdout: 2026-02-22T22:33:36.148 INFO:teuthology.orchestra.run.trial023.stdout:[mgr] 2026-02-22T22:33:36.148 INFO:teuthology.orchestra.run.trial023.stdout: debug ms = 1 2026-02-22T22:33:36.148 INFO:teuthology.orchestra.run.trial023.stdout: debug mgr = 20 2026-02-22T22:33:36.148 INFO:teuthology.orchestra.run.trial023.stdout: debug mon = 20 2026-02-22T22:33:36.149 INFO:teuthology.orchestra.run.trial023.stdout: debug auth = 20 2026-02-22T22:33:36.149 INFO:teuthology.orchestra.run.trial023.stdout: mon reweight min pgs per osd = 4 2026-02-22T22:33:36.149 INFO:teuthology.orchestra.run.trial023.stdout: mon reweight min bytes per osd = 10 2026-02-22T22:33:36.149 INFO:teuthology.orchestra.run.trial023.stdout: mgr/telemetry/nag = false 2026-02-22T22:33:36.149 INFO:teuthology.orchestra.run.trial023.stdout: 2026-02-22T22:33:36.149 INFO:teuthology.orchestra.run.trial023.stdout:[mon] 2026-02-22T22:33:36.149 INFO:teuthology.orchestra.run.trial023.stdout: debug ms = 1 2026-02-22T22:33:36.149 INFO:teuthology.orchestra.run.trial023.stdout: debug mon = 20 2026-02-22T22:33:36.149 INFO:teuthology.orchestra.run.trial023.stdout: debug paxos = 20 2026-02-22T22:33:36.149 INFO:teuthology.orchestra.run.trial023.stdout: debug auth = 30 2026-02-22T22:33:36.149 INFO:teuthology.orchestra.run.trial023.stdout: mon data avail warn = 5 2026-02-22T22:33:36.149 INFO:teuthology.orchestra.run.trial023.stdout: mon mgr mkfs grace = 240 2026-02-22T22:33:36.149 INFO:teuthology.orchestra.run.trial023.stdout: mon reweight min pgs per osd = 4 2026-02-22T22:33:36.149 INFO:teuthology.orchestra.run.trial023.stdout: mon osd reporter subtree level = osd 2026-02-22T22:33:36.149 INFO:teuthology.orchestra.run.trial023.stdout: mon osd prime pg temp = true 2026-02-22T22:33:36.149 INFO:teuthology.orchestra.run.trial023.stdout: mon reweight min bytes per osd = 10 2026-02-22T22:33:36.149 INFO:teuthology.orchestra.run.trial023.stdout: 2026-02-22T22:33:36.149 INFO:teuthology.orchestra.run.trial023.stdout: # rotate auth tickets quickly to exercise renewal paths 2026-02-22T22:33:36.149 INFO:teuthology.orchestra.run.trial023.stdout: auth mon ticket ttl = 660 # 11m 2026-02-22T22:33:36.149 INFO:teuthology.orchestra.run.trial023.stdout: auth service ticket ttl = 240 # 4m 2026-02-22T22:33:36.149 INFO:teuthology.orchestra.run.trial023.stdout: 2026-02-22T22:33:36.149 INFO:teuthology.orchestra.run.trial023.stdout: # don't complain about insecure global_id in the test suite 2026-02-22T22:33:36.150 INFO:teuthology.orchestra.run.trial023.stdout: mon_warn_on_insecure_global_id_reclaim = false 2026-02-22T22:33:36.150 INFO:teuthology.orchestra.run.trial023.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2026-02-22T22:33:36.150 INFO:teuthology.orchestra.run.trial023.stdout: 2026-02-22T22:33:36.150 INFO:teuthology.orchestra.run.trial023.stdout: # 1m isn't quite enough 2026-02-22T22:33:36.150 INFO:teuthology.orchestra.run.trial023.stdout: mon_down_mkfs_grace = 2m 2026-02-22T22:33:36.150 INFO:teuthology.orchestra.run.trial023.stdout: 2026-02-22T22:33:36.150 INFO:teuthology.orchestra.run.trial023.stdout: mon_warn_on_filestore_osds = false 2026-02-22T22:33:36.150 INFO:teuthology.orchestra.run.trial023.stdout: mon scrub interval = 300 2026-02-22T22:33:36.150 INFO:teuthology.orchestra.run.trial023.stdout: 2026-02-22T22:33:36.150 INFO:teuthology.orchestra.run.trial023.stdout:[client] 2026-02-22T22:33:36.150 INFO:teuthology.orchestra.run.trial023.stdout: rgw cache enabled = true 2026-02-22T22:33:36.150 INFO:teuthology.orchestra.run.trial023.stdout: rgw enable ops log = true 2026-02-22T22:33:36.150 INFO:teuthology.orchestra.run.trial023.stdout: rgw enable usage log = true 2026-02-22T22:33:36.150 INFO:teuthology.orchestra.run.trial023.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2026-02-22T22:33:36.150 INFO:teuthology.orchestra.run.trial023.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2026-02-22T22:33:36.150 INFO:teuthology.orchestra.run.trial023.stdout:[mon.a] 2026-02-22T22:33:36.150 INFO:teuthology.orchestra.run.trial023.stdout:[mon.b] 2026-02-22T22:33:36.150 INFO:teuthology.orchestra.run.trial023.stdout:[mon.c] 2026-02-22T22:33:36.153 INFO:tasks.ceph:Creating admin key on mon.a... 2026-02-22T22:33:36.153 DEBUG:teuthology.orchestra.run.trial023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/ceph.keyring 2026-02-22T22:33:36.219 INFO:tasks.ceph:Copying monmap to all nodes... 2026-02-22T22:33:36.220 DEBUG:teuthology.orchestra.run.trial023:> set -ex 2026-02-22T22:33:36.220 DEBUG:teuthology.orchestra.run.trial023:> dd if=/etc/ceph/ceph.keyring of=/dev/stdout 2026-02-22T22:33:36.266 DEBUG:teuthology.orchestra.run.trial023:> set -ex 2026-02-22T22:33:36.266 DEBUG:teuthology.orchestra.run.trial023:> dd if=/home/ubuntu/cephtest/ceph.monmap of=/dev/stdout 2026-02-22T22:33:36.314 INFO:tasks.ceph:Sending monmap to node ubuntu@trial023.front.sepia.ceph.com 2026-02-22T22:33:36.314 DEBUG:teuthology.orchestra.run.trial023:> set -ex 2026-02-22T22:33:36.314 DEBUG:teuthology.orchestra.run.trial023:> sudo dd of=/etc/ceph/ceph.keyring 2026-02-22T22:33:36.314 DEBUG:teuthology.orchestra.run.trial023:> sudo chmod 0644 /etc/ceph/ceph.keyring 2026-02-22T22:33:36.371 DEBUG:teuthology.orchestra.run.trial023:> set -ex 2026-02-22T22:33:36.372 DEBUG:teuthology.orchestra.run.trial023:> dd of=/home/ubuntu/cephtest/ceph.monmap 2026-02-22T22:33:36.418 INFO:tasks.ceph:Setting up mon nodes... 2026-02-22T22:33:36.419 INFO:tasks.ceph:Setting up mgr nodes... 2026-02-22T22:33:36.419 DEBUG:teuthology.orchestra.run.trial023:> sudo mkdir -p /var/lib/ceph/mgr/ceph-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/ceph-x/keyring 2026-02-22T22:33:36.483 INFO:teuthology.orchestra.run.trial023.stdout:creating /var/lib/ceph/mgr/ceph-x/keyring 2026-02-22T22:33:36.486 INFO:tasks.ceph:Setting up mds nodes... 2026-02-22T22:33:36.486 INFO:tasks.ceph_client:Setting up client nodes... 2026-02-22T22:33:36.486 DEBUG:teuthology.orchestra.run.trial023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/ceph.client.0.keyring && sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2026-02-22T22:33:36.546 INFO:teuthology.orchestra.run.trial023.stdout:creating /etc/ceph/ceph.client.0.keyring 2026-02-22T22:33:36.553 INFO:tasks.ceph:Running mkfs on osd nodes... 2026-02-22T22:33:36.554 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@trial023.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'}} 2026-02-22T22:33:36.554 DEBUG:teuthology.orchestra.run.trial023:> sudo mkdir -p /var/lib/ceph/osd/ceph-0 2026-02-22T22:33:36.604 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2026-02-22T22:33:36.605 INFO:tasks.ceph:role: osd.0 2026-02-22T22:33:36.605 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@trial023.front.sepia.ceph.com 2026-02-22T22:33:36.605 DEBUG:teuthology.orchestra.run.trial023:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2026-02-22T22:33:36.654 INFO:teuthology.orchestra.run.trial023.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=45875200 blks 2026-02-22T22:33:36.654 INFO:teuthology.orchestra.run.trial023.stdout: = sectsz=4096 attr=2, projid32bit=1 2026-02-22T22:33:36.654 INFO:teuthology.orchestra.run.trial023.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2026-02-22T22:33:36.655 INFO:teuthology.orchestra.run.trial023.stdout: = reflink=1 bigtime=0 inobtcount=0 2026-02-22T22:33:36.655 INFO:teuthology.orchestra.run.trial023.stdout:data = bsize=4096 blocks=183500800, imaxpct=25 2026-02-22T22:33:36.655 INFO:teuthology.orchestra.run.trial023.stdout: = sunit=0 swidth=0 blks 2026-02-22T22:33:36.655 INFO:teuthology.orchestra.run.trial023.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2026-02-22T22:33:36.655 INFO:teuthology.orchestra.run.trial023.stdout:log =internal log bsize=4096 blocks=89600, version=2 2026-02-22T22:33:36.655 INFO:teuthology.orchestra.run.trial023.stdout: = sectsz=4096 sunit=1 blks, lazy-count=1 2026-02-22T22:33:36.655 INFO:teuthology.orchestra.run.trial023.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2026-02-22T22:33:37.235 INFO:teuthology.orchestra.run.trial023.stdout:Discarding blocks...Done. 2026-02-22T22:33:37.237 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@trial023.front.sepia.ceph.com -o noatime 2026-02-22T22:33:37.237 DEBUG:teuthology.orchestra.run.trial023:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-0 2026-02-22T22:33:37.294 DEBUG:teuthology.orchestra.run.trial023:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-0 2026-02-22T22:33:37.344 DEBUG:teuthology.orchestra.run:got remote process result: 1 2026-02-22T22:33:37.344 INFO:teuthology.orchestra.run.trial023.stderr:sudo: /sbin/restorecon: command not found 2026-02-22T22:33:37.345 DEBUG:teuthology.orchestra.run.trial023:> sudo mkdir -p /var/lib/ceph/osd/ceph-1 2026-02-22T22:33:37.393 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2026-02-22T22:33:37.393 INFO:tasks.ceph:role: osd.1 2026-02-22T22:33:37.394 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@trial023.front.sepia.ceph.com 2026-02-22T22:33:37.394 DEBUG:teuthology.orchestra.run.trial023:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2026-02-22T22:33:37.447 INFO:teuthology.orchestra.run.trial023.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=45875200 blks 2026-02-22T22:33:37.447 INFO:teuthology.orchestra.run.trial023.stdout: = sectsz=4096 attr=2, projid32bit=1 2026-02-22T22:33:37.447 INFO:teuthology.orchestra.run.trial023.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2026-02-22T22:33:37.447 INFO:teuthology.orchestra.run.trial023.stdout: = reflink=1 bigtime=0 inobtcount=0 2026-02-22T22:33:37.447 INFO:teuthology.orchestra.run.trial023.stdout:data = bsize=4096 blocks=183500800, imaxpct=25 2026-02-22T22:33:37.447 INFO:teuthology.orchestra.run.trial023.stdout: = sunit=0 swidth=0 blks 2026-02-22T22:33:37.447 INFO:teuthology.orchestra.run.trial023.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2026-02-22T22:33:37.448 INFO:teuthology.orchestra.run.trial023.stdout:log =internal log bsize=4096 blocks=89600, version=2 2026-02-22T22:33:37.448 INFO:teuthology.orchestra.run.trial023.stdout: = sectsz=4096 sunit=1 blks, lazy-count=1 2026-02-22T22:33:37.448 INFO:teuthology.orchestra.run.trial023.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2026-02-22T22:33:38.014 INFO:teuthology.orchestra.run.trial023.stdout:Discarding blocks...Done. 2026-02-22T22:33:38.016 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@trial023.front.sepia.ceph.com -o noatime 2026-02-22T22:33:38.016 DEBUG:teuthology.orchestra.run.trial023:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-1 2026-02-22T22:33:38.075 DEBUG:teuthology.orchestra.run.trial023:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-1 2026-02-22T22:33:38.123 DEBUG:teuthology.orchestra.run:got remote process result: 1 2026-02-22T22:33:38.124 INFO:teuthology.orchestra.run.trial023.stderr:sudo: /sbin/restorecon: command not found 2026-02-22T22:33:38.124 DEBUG:teuthology.orchestra.run.trial023:> sudo mkdir -p /var/lib/ceph/osd/ceph-2 2026-02-22T22:33:38.174 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2026-02-22T22:33:38.174 INFO:tasks.ceph:role: osd.2 2026-02-22T22:33:38.174 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@trial023.front.sepia.ceph.com 2026-02-22T22:33:38.174 DEBUG:teuthology.orchestra.run.trial023:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2026-02-22T22:33:38.227 INFO:teuthology.orchestra.run.trial023.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=45875200 blks 2026-02-22T22:33:38.227 INFO:teuthology.orchestra.run.trial023.stdout: = sectsz=4096 attr=2, projid32bit=1 2026-02-22T22:33:38.227 INFO:teuthology.orchestra.run.trial023.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2026-02-22T22:33:38.227 INFO:teuthology.orchestra.run.trial023.stdout: = reflink=1 bigtime=0 inobtcount=0 2026-02-22T22:33:38.227 INFO:teuthology.orchestra.run.trial023.stdout:data = bsize=4096 blocks=183500800, imaxpct=25 2026-02-22T22:33:38.227 INFO:teuthology.orchestra.run.trial023.stdout: = sunit=0 swidth=0 blks 2026-02-22T22:33:38.227 INFO:teuthology.orchestra.run.trial023.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2026-02-22T22:33:38.227 INFO:teuthology.orchestra.run.trial023.stdout:log =internal log bsize=4096 blocks=89600, version=2 2026-02-22T22:33:38.228 INFO:teuthology.orchestra.run.trial023.stdout: = sectsz=4096 sunit=1 blks, lazy-count=1 2026-02-22T22:33:38.228 INFO:teuthology.orchestra.run.trial023.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2026-02-22T22:33:38.794 INFO:teuthology.orchestra.run.trial023.stdout:Discarding blocks...Done. 2026-02-22T22:33:38.795 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@trial023.front.sepia.ceph.com -o noatime 2026-02-22T22:33:38.796 DEBUG:teuthology.orchestra.run.trial023:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-2 2026-02-22T22:33:38.851 DEBUG:teuthology.orchestra.run.trial023:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-2 2026-02-22T22:33:38.899 INFO:teuthology.orchestra.run.trial023.stderr:sudo: /sbin/restorecon: command not found 2026-02-22T22:33:38.899 DEBUG:teuthology.orchestra.run:got remote process result: 1 2026-02-22T22:33:38.899 DEBUG:teuthology.orchestra.run.trial023:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/ceph.monmap 2026-02-22T22:33:38.963 INFO:teuthology.orchestra.run.trial023.stderr:2026-02-22T22:33:38.960+0000 7f0c40a978c0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-0/keyring: can't open /var/lib/ceph/osd/ceph-0/keyring: (2) No such file or directory 2026-02-22T22:33:38.964 INFO:teuthology.orchestra.run.trial023.stderr:2026-02-22T22:33:38.960+0000 7f0c40a978c0 -1 created new key in keyring /var/lib/ceph/osd/ceph-0/keyring 2026-02-22T22:33:38.964 INFO:teuthology.orchestra.run.trial023.stderr:2026-02-22T22:33:38.960+0000 7f0c40a978c0 -1 bdev(0x55b69949d800 /var/lib/ceph/osd/ceph-0/block) open stat got: (1) Operation not permitted 2026-02-22T22:33:38.964 INFO:teuthology.orchestra.run.trial023.stderr:2026-02-22T22:33:38.960+0000 7f0c40a978c0 -1 bluestore(/var/lib/ceph/osd/ceph-0) _read_fsid unparsable uuid 2026-02-22T22:33:42.062 DEBUG:teuthology.orchestra.run.trial023:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2026-02-22T22:33:42.115 DEBUG:teuthology.orchestra.run.trial023:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/ceph.monmap 2026-02-22T22:33:42.182 INFO:teuthology.orchestra.run.trial023.stderr:2026-02-22T22:33:42.180+0000 7fb359a278c0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-1/keyring: can't open /var/lib/ceph/osd/ceph-1/keyring: (2) No such file or directory 2026-02-22T22:33:42.183 INFO:teuthology.orchestra.run.trial023.stderr:2026-02-22T22:33:42.180+0000 7fb359a278c0 -1 created new key in keyring /var/lib/ceph/osd/ceph-1/keyring 2026-02-22T22:33:42.183 INFO:teuthology.orchestra.run.trial023.stderr:2026-02-22T22:33:42.180+0000 7fb359a278c0 -1 bdev(0x55e84d659800 /var/lib/ceph/osd/ceph-1/block) open stat got: (1) Operation not permitted 2026-02-22T22:33:42.184 INFO:teuthology.orchestra.run.trial023.stderr:2026-02-22T22:33:42.180+0000 7fb359a278c0 -1 bluestore(/var/lib/ceph/osd/ceph-1) _read_fsid unparsable uuid 2026-02-22T22:33:45.285 DEBUG:teuthology.orchestra.run.trial023:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2026-02-22T22:33:45.333 DEBUG:teuthology.orchestra.run.trial023:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/ceph.monmap 2026-02-22T22:33:45.399 INFO:teuthology.orchestra.run.trial023.stderr:2026-02-22T22:33:45.396+0000 7fd64d83d8c0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-2/keyring: can't open /var/lib/ceph/osd/ceph-2/keyring: (2) No such file or directory 2026-02-22T22:33:45.399 INFO:teuthology.orchestra.run.trial023.stderr:2026-02-22T22:33:45.396+0000 7fd64d83d8c0 -1 created new key in keyring /var/lib/ceph/osd/ceph-2/keyring 2026-02-22T22:33:45.400 INFO:teuthology.orchestra.run.trial023.stderr:2026-02-22T22:33:45.396+0000 7fd64d83d8c0 -1 bdev(0x55775fda7800 /var/lib/ceph/osd/ceph-2/block) open stat got: (1) Operation not permitted 2026-02-22T22:33:45.400 INFO:teuthology.orchestra.run.trial023.stderr:2026-02-22T22:33:45.396+0000 7fd64d83d8c0 -1 bluestore(/var/lib/ceph/osd/ceph-2) _read_fsid unparsable uuid 2026-02-22T22:33:48.512 DEBUG:teuthology.orchestra.run.trial023:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2026-02-22T22:33:48.562 INFO:tasks.ceph:Reading keys from all nodes... 2026-02-22T22:33:48.562 DEBUG:teuthology.orchestra.run.trial023:> set -ex 2026-02-22T22:33:48.562 DEBUG:teuthology.orchestra.run.trial023:> sudo dd if=/var/lib/ceph/mgr/ceph-x/keyring of=/dev/stdout 2026-02-22T22:33:48.614 DEBUG:teuthology.orchestra.run.trial023:> set -ex 2026-02-22T22:33:48.614 DEBUG:teuthology.orchestra.run.trial023:> sudo dd if=/var/lib/ceph/osd/ceph-0/keyring of=/dev/stdout 2026-02-22T22:33:48.666 DEBUG:teuthology.orchestra.run.trial023:> set -ex 2026-02-22T22:33:48.666 DEBUG:teuthology.orchestra.run.trial023:> sudo dd if=/var/lib/ceph/osd/ceph-1/keyring of=/dev/stdout 2026-02-22T22:33:48.718 DEBUG:teuthology.orchestra.run.trial023:> set -ex 2026-02-22T22:33:48.718 DEBUG:teuthology.orchestra.run.trial023:> sudo dd if=/var/lib/ceph/osd/ceph-2/keyring of=/dev/stdout 2026-02-22T22:33:48.770 DEBUG:teuthology.orchestra.run.trial023:> set -ex 2026-02-22T22:33:48.770 DEBUG:teuthology.orchestra.run.trial023:> dd if=/etc/ceph/ceph.client.0.keyring of=/dev/stdout 2026-02-22T22:33:48.817 INFO:tasks.ceph:Adding keys to all mons... 2026-02-22T22:33:48.817 DEBUG:teuthology.orchestra.run.trial023:> sudo tee -a /etc/ceph/ceph.keyring 2026-02-22T22:33:48.864 INFO:teuthology.orchestra.run.trial023.stdout:[mgr.x] 2026-02-22T22:33:48.864 INFO:teuthology.orchestra.run.trial023.stdout: key = AQBAhJtp8p7zHBAA9dOPoc865FewilZxaAlp3g== 2026-02-22T22:33:48.864 INFO:teuthology.orchestra.run.trial023.stdout:[osd.0] 2026-02-22T22:33:48.864 INFO:teuthology.orchestra.run.trial023.stdout: key = AQBChJtpjLOHORAAvkjY/GrAk7aPGDSAbURRgQ== 2026-02-22T22:33:48.864 INFO:teuthology.orchestra.run.trial023.stdout:[osd.1] 2026-02-22T22:33:48.864 INFO:teuthology.orchestra.run.trial023.stdout: key = AQBGhJtpvYgECxAAKLNLDNQwbcFFiJ7q0nvvpQ== 2026-02-22T22:33:48.864 INFO:teuthology.orchestra.run.trial023.stdout:[osd.2] 2026-02-22T22:33:48.864 INFO:teuthology.orchestra.run.trial023.stdout: key = AQBJhJtpRFrhFxAAlc3+ueyf816Kt4pX15e01Q== 2026-02-22T22:33:48.865 INFO:teuthology.orchestra.run.trial023.stdout:[client.0] 2026-02-22T22:33:48.865 INFO:teuthology.orchestra.run.trial023.stdout: key = AQBAhJtphSq0IBAAqjUMVmeT8RMwGr++VkA39g== 2026-02-22T22:33:48.865 DEBUG:teuthology.orchestra.run.trial023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2026-02-22T22:33:48.931 DEBUG:teuthology.orchestra.run.trial023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2026-02-22T22:33:48.994 DEBUG:teuthology.orchestra.run.trial023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2026-02-22T22:33:49.058 DEBUG:teuthology.orchestra.run.trial023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2026-02-22T22:33:49.121 DEBUG:teuthology.orchestra.run.trial023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2026-02-22T22:33:49.183 INFO:tasks.ceph:Running mkfs on mon nodes... 2026-02-22T22:33:49.183 DEBUG:teuthology.orchestra.run.trial023:> sudo mkdir -p /var/lib/ceph/mon/ceph-a 2026-02-22T22:33:49.233 DEBUG:teuthology.orchestra.run.trial023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i a --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2026-02-22T22:33:49.298 DEBUG:teuthology.orchestra.run.trial023:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-a 2026-02-22T22:33:49.349 DEBUG:teuthology.orchestra.run.trial023:> sudo mkdir -p /var/lib/ceph/mon/ceph-b 2026-02-22T22:33:49.397 DEBUG:teuthology.orchestra.run.trial023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i b --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2026-02-22T22:33:49.463 DEBUG:teuthology.orchestra.run.trial023:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-b 2026-02-22T22:33:49.513 DEBUG:teuthology.orchestra.run.trial023:> sudo mkdir -p /var/lib/ceph/mon/ceph-c 2026-02-22T22:33:49.561 DEBUG:teuthology.orchestra.run.trial023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i c --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2026-02-22T22:33:49.626 DEBUG:teuthology.orchestra.run.trial023:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-c 2026-02-22T22:33:49.677 DEBUG:teuthology.orchestra.run.trial023:> rm -- /home/ubuntu/cephtest/ceph.monmap 2026-02-22T22:33:49.720 INFO:tasks.ceph:Starting mon daemons in cluster ceph... 2026-02-22T22:33:49.721 INFO:tasks.ceph.mon.a:Restarting daemon 2026-02-22T22:33:49.721 DEBUG:teuthology.orchestra.run.trial023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2026-02-22T22:33:49.762 INFO:tasks.ceph.mon.a:Started 2026-02-22T22:33:49.762 INFO:tasks.ceph.mon.b:Restarting daemon 2026-02-22T22:33:49.763 DEBUG:teuthology.orchestra.run.trial023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i b 2026-02-22T22:33:49.764 INFO:tasks.ceph.mon.b:Started 2026-02-22T22:33:49.764 INFO:tasks.ceph.mon.c:Restarting daemon 2026-02-22T22:33:49.764 DEBUG:teuthology.orchestra.run.trial023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i c 2026-02-22T22:33:49.765 INFO:tasks.ceph.mon.c:Started 2026-02-22T22:33:49.765 DEBUG:teuthology.orchestra.run.trial023:> sudo ceph config set mgr mgr_pool false --force 2026-02-22T22:33:50.150 INFO:tasks.ceph:Starting mgr daemons in cluster ceph... 2026-02-22T22:33:50.150 INFO:tasks.ceph.mgr.x:Restarting daemon 2026-02-22T22:33:50.151 DEBUG:teuthology.orchestra.run.trial023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i x 2026-02-22T22:33:50.152 INFO:tasks.ceph.mgr.x:Started 2026-02-22T22:33:50.152 INFO:tasks.ceph:Setting crush tunables to default 2026-02-22T22:33:50.152 DEBUG:teuthology.orchestra.run.trial023:> sudo ceph --cluster ceph osd crush tunables default 2026-02-22T22:33:50.254 INFO:teuthology.orchestra.run.trial023.stderr:adjusted tunables profile to default 2026-02-22T22:33:50.257 INFO:tasks.ceph.mgr.x.trial023.stderr:2026-02-22T22:33:50.252+0000 7f3ee9b98280 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2026-02-22T22:33:50.264 INFO:tasks.ceph:check_enable_crimson: False 2026-02-22T22:33:50.264 INFO:tasks.ceph:Starting osd daemons in cluster ceph... 2026-02-22T22:33:50.264 DEBUG:teuthology.orchestra.run.trial023:> set -ex 2026-02-22T22:33:50.264 DEBUG:teuthology.orchestra.run.trial023:> sudo dd if=/var/lib/ceph/osd/ceph-0/fsid of=/dev/stdout 2026-02-22T22:33:50.275 DEBUG:teuthology.orchestra.run.trial023:> set -ex 2026-02-22T22:33:50.275 DEBUG:teuthology.orchestra.run.trial023:> sudo dd if=/var/lib/ceph/osd/ceph-1/fsid of=/dev/stdout 2026-02-22T22:33:50.283 INFO:tasks.ceph.mgr.x.trial023.stderr:2026-02-22T22:33:50.280+0000 7f3ee9b98280 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2026-02-22T22:33:50.291 DEBUG:teuthology.orchestra.run.trial023:> set -ex 2026-02-22T22:33:50.291 DEBUG:teuthology.orchestra.run.trial023:> sudo dd if=/var/lib/ceph/osd/ceph-2/fsid of=/dev/stdout 2026-02-22T22:33:50.321 INFO:tasks.ceph.mgr.x.trial023.stderr:2026-02-22T22:33:50.316+0000 7f3ee9b98280 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2026-02-22T22:33:50.328 DEBUG:teuthology.orchestra.run.trial023:> sudo ceph --cluster ceph osd new 634275c9-6a87-4a66-8d94-04b38d5640bf 0 2026-02-22T22:33:50.407 INFO:tasks.ceph.mgr.x.trial023.stderr:2026-02-22T22:33:50.404+0000 7f3ee9b98280 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2026-02-22T22:33:50.482 INFO:teuthology.orchestra.run.trial023.stdout:0 2026-02-22T22:33:50.486 INFO:tasks.ceph.mgr.x.trial023.stderr:2026-02-22T22:33:50.484+0000 7f3ee9b98280 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2026-02-22T22:33:50.492 DEBUG:teuthology.orchestra.run.trial023:> sudo ceph --cluster ceph osd new f73635a0-7d9b-4967-9505-3f230f1a9463 1 2026-02-22T22:33:50.523 INFO:tasks.ceph.mgr.x.trial023.stderr:2026-02-22T22:33:50.520+0000 7f3ee9b98280 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2026-02-22T22:33:50.598 INFO:tasks.ceph.mgr.x.trial023.stderr:2026-02-22T22:33:50.596+0000 7f3ee9b98280 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2026-02-22T22:33:50.623 INFO:teuthology.orchestra.run.trial023.stdout:1 2026-02-22T22:33:50.633 DEBUG:teuthology.orchestra.run.trial023:> sudo ceph --cluster ceph osd new 23a9de65-37ff-46ca-83ab-c20c149a2276 2 2026-02-22T22:33:50.733 INFO:teuthology.orchestra.run.trial023.stdout:2 2026-02-22T22:33:50.742 INFO:tasks.ceph.osd.0:Restarting daemon 2026-02-22T22:33:50.742 DEBUG:teuthology.orchestra.run.trial023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 0 2026-02-22T22:33:50.744 INFO:tasks.ceph.osd.0:Started 2026-02-22T22:33:50.744 INFO:tasks.ceph.osd.1:Restarting daemon 2026-02-22T22:33:50.744 DEBUG:teuthology.orchestra.run.trial023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 1 2026-02-22T22:33:50.745 INFO:tasks.ceph.osd.1:Started 2026-02-22T22:33:50.745 INFO:tasks.ceph.osd.2:Restarting daemon 2026-02-22T22:33:50.745 DEBUG:teuthology.orchestra.run.trial023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 2 2026-02-22T22:33:50.747 INFO:tasks.ceph.osd.2:Started 2026-02-22T22:33:50.747 DEBUG:teuthology.orchestra.run.trial023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2026-02-22T22:33:50.775 INFO:tasks.ceph.mgr.x.trial023.stderr:2026-02-22T22:33:50.772+0000 7f3ee9b98280 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2026-02-22T22:33:50.840 INFO:teuthology.orchestra.run.trial023.stdout: 2026-02-22T22:33:50.840 INFO:teuthology.orchestra.run.trial023.stdout:{"epoch":5,"fsid":"a296be36-532d-4148-b28e-0183b3968057","created":"2026-02-22T22:33:50.079561+0000","modified":"2026-02-22T22:33:50.733447+0000","last_up_change":"0.000000","last_in_change":"2026-02-22T22:33:50.733447+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"634275c9-6a87-4a66-8d94-04b38d5640bf","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"f73635a0-7d9b-4967-9505-3f230f1a9463","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"23a9de65-37ff-46ca-83ab-c20c149a2276","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-22T22:33:50.849 INFO:tasks.ceph.ceph_manager.ceph:[] 2026-02-22T22:33:50.849 INFO:tasks.ceph:Waiting for OSDs to come up 2026-02-22T22:33:51.083 INFO:tasks.ceph.mgr.x.trial023.stderr:2026-02-22T22:33:51.080+0000 7f3ee9b98280 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2026-02-22T22:33:51.150 DEBUG:teuthology.orchestra.run.trial023:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2026-02-22T22:33:51.176 INFO:tasks.ceph.mgr.x.trial023.stderr:2026-02-22T22:33:51.172+0000 7f3ee9b98280 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2026-02-22T22:33:51.204 INFO:tasks.ceph.mgr.x.trial023.stderr:2026-02-22T22:33:51.200+0000 7f3ee9b98280 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2026-02-22T22:33:51.236 INFO:tasks.ceph.mgr.x.trial023.stderr:2026-02-22T22:33:51.232+0000 7f3ee9b98280 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2026-02-22T22:33:51.236 INFO:teuthology.misc.health.trial023.stdout: 2026-02-22T22:33:51.237 INFO:teuthology.misc.health.trial023.stdout:{"epoch":5,"fsid":"a296be36-532d-4148-b28e-0183b3968057","created":"2026-02-22T22:33:50.079561+0000","modified":"2026-02-22T22:33:50.733447+0000","last_up_change":"0.000000","last_in_change":"2026-02-22T22:33:50.733447+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"634275c9-6a87-4a66-8d94-04b38d5640bf","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"f73635a0-7d9b-4967-9505-3f230f1a9463","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"23a9de65-37ff-46ca-83ab-c20c149a2276","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-22T22:33:51.244 DEBUG:teuthology.misc:0 of 3 OSDs are up 2026-02-22T22:33:51.280 INFO:tasks.ceph.mgr.x.trial023.stderr:2026-02-22T22:33:51.276+0000 7f3ee9b98280 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2026-02-22T22:33:51.383 INFO:tasks.ceph.mgr.x.trial023.stderr:2026-02-22T22:33:51.380+0000 7f3ee9b98280 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2026-02-22T22:33:51.579 INFO:tasks.ceph.mgr.x.trial023.stderr:ERROR:root:Module 'xmltodict' is not installed. 2026-02-22T22:33:51.754 INFO:tasks.ceph.mgr.x.trial023.stderr:2026-02-22T22:33:51.752+0000 7f3ee9b98280 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2026-02-22T22:33:51.787 INFO:tasks.ceph.mgr.x.trial023.stderr:2026-02-22T22:33:51.784+0000 7f3ee9b98280 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2026-02-22T22:33:51.881 INFO:tasks.ceph.mgr.x.trial023.stderr:2026-02-22T22:33:51.876+0000 7f3ee9b98280 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2026-02-22T22:33:51.906 INFO:tasks.ceph.mgr.x.trial023.stderr:2026-02-22T22:33:51.904+0000 7f3ee9b98280 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2026-02-22T22:33:51.934 INFO:tasks.ceph.mgr.x.trial023.stderr:2026-02-22T22:33:51.932+0000 7f3ee9b98280 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2026-02-22T22:33:51.961 INFO:tasks.ceph.mgr.x.trial023.stderr:2026-02-22T22:33:51.956+0000 7f3ee9b98280 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2026-02-22T22:33:52.082 INFO:tasks.ceph.mgr.x.trial023.stderr:/usr/lib/python3/dist-packages/scipy/__init__.py:67: UserWarning: NumPy was imported from a Python sub-interpreter but NumPy does not properly support sub-interpreters. This will likely work for most users but might cause hard to track down issues or subtle bugs. A common user of the rare sub-interpreter feature is wsgi which also allows single-interpreter mode. 2026-02-22T22:33:52.082 INFO:tasks.ceph.mgr.x.trial023.stderr:Improvements in the case of bugs are welcome, but is not on the NumPy roadmap, and full support may require significant effort to achieve. 2026-02-22T22:33:52.082 INFO:tasks.ceph.mgr.x.trial023.stderr: from numpy import show_config as show_numpy_config 2026-02-22T22:33:52.089 INFO:tasks.ceph.mgr.x.trial023.stderr:2026-02-22T22:33:52.084+0000 7f3ee9b98280 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2026-02-22T22:33:52.116 INFO:tasks.ceph.mgr.x.trial023.stderr:2026-02-22T22:33:52.112+0000 7f3ee9b98280 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2026-02-22T22:33:52.155 INFO:tasks.ceph.mgr.x.trial023.stderr:2026-02-22T22:33:52.152+0000 7f3ee9b98280 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2026-02-22T22:33:52.551 INFO:tasks.ceph.osd.1.trial023.stderr:2026-02-22T22:33:52.548+0000 7f42d42358c0 -1 Falling back to public interface 2026-02-22T22:33:52.795 INFO:tasks.ceph.osd.2.trial023.stderr:2026-02-22T22:33:52.792+0000 7fc9c85888c0 -1 Falling back to public interface 2026-02-22T22:33:52.802 INFO:tasks.ceph.osd.0.trial023.stderr:2026-02-22T22:33:52.800+0000 7f12bf5508c0 -1 Falling back to public interface 2026-02-22T22:33:53.182 INFO:tasks.ceph.mgr.x.trial023.stderr:2026-02-22T22:33:53.180+0000 7f3e083e9640 -1 mgr.server handle_report got status from non-daemon mon.b 2026-02-22T22:33:53.386 INFO:tasks.ceph.mgr.x.trial023.stderr:2026-02-22T22:33:53.384+0000 7f3e083e9640 -1 mgr.server handle_report got status from non-daemon mon.c 2026-02-22T22:33:56.410 INFO:tasks.ceph.osd.1.trial023.stderr:2026-02-22T22:33:56.408+0000 7f42d42358c0 -1 osd.1 0 log_to_monitors true 2026-02-22T22:33:56.723 INFO:tasks.ceph.osd.0.trial023.stderr:2026-02-22T22:33:56.720+0000 7f12bf5508c0 -1 osd.0 0 log_to_monitors true 2026-02-22T22:33:56.796 INFO:tasks.ceph.osd.2.trial023.stderr:2026-02-22T22:33:56.792+0000 7fc9c85888c0 -1 osd.2 0 log_to_monitors true 2026-02-22T22:33:57.546 DEBUG:teuthology.orchestra.run.trial023:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2026-02-22T22:33:57.689 INFO:teuthology.misc.health.trial023.stdout: 2026-02-22T22:33:57.689 INFO:teuthology.misc.health.trial023.stdout:{"epoch":6,"fsid":"a296be36-532d-4148-b28e-0183b3968057","created":"2026-02-22T22:33:50.079561+0000","modified":"2026-02-22T22:33:57.184378+0000","last_up_change":"0.000000","last_in_change":"2026-02-22T22:33:50.733447+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":3,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"634275c9-6a87-4a66-8d94-04b38d5640bf","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"f73635a0-7d9b-4967-9505-3f230f1a9463","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"23a9de65-37ff-46ca-83ab-c20c149a2276","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-22T22:33:57.698 DEBUG:teuthology.misc:0 of 3 OSDs are up 2026-02-22T22:33:58.192 INFO:tasks.ceph.osd.0.trial023.stderr:2026-02-22T22:33:58.188+0000 7f12bb4f9640 -1 osd.0 0 waiting for initial osdmap 2026-02-22T22:33:58.193 INFO:tasks.ceph.osd.2.trial023.stderr:2026-02-22T22:33:58.188+0000 7fc9c4531640 -1 osd.2 0 waiting for initial osdmap 2026-02-22T22:33:58.193 INFO:tasks.ceph.osd.1.trial023.stderr:2026-02-22T22:33:58.188+0000 7f42d01de640 -1 osd.1 0 waiting for initial osdmap 2026-02-22T22:33:58.207 INFO:tasks.ceph.osd.0.trial023.stderr:2026-02-22T22:33:58.204+0000 7f12b332c640 -1 osd.0 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2026-02-22T22:33:58.207 INFO:tasks.ceph.osd.1.trial023.stderr:2026-02-22T22:33:58.204+0000 7f42cc019640 -1 osd.1 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2026-02-22T22:33:58.207 INFO:tasks.ceph.osd.2.trial023.stderr:2026-02-22T22:33:58.204+0000 7fc9bc364640 -1 osd.2 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2026-02-22T22:34:04.000 DEBUG:teuthology.orchestra.run.trial023:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2026-02-22T22:34:04.139 INFO:teuthology.misc.health.trial023.stdout: 2026-02-22T22:34:04.139 INFO:teuthology.misc.health.trial023.stdout:{"epoch":8,"fsid":"a296be36-532d-4148-b28e-0183b3968057","created":"2026-02-22T22:33:50.079561+0000","modified":"2026-02-22T22:33:59.193101+0000","last_up_change":"2026-02-22T22:33:59.193101+0000","last_in_change":"2026-02-22T22:33:50.733447+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"634275c9-6a87-4a66-8d94-04b38d5640bf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.23:6818","nonce":2573691967},{"type":"v1","addr":"10.20.193.23:6819","nonce":2573691967}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.23:6820","nonce":2573691967},{"type":"v1","addr":"10.20.193.23:6821","nonce":2573691967}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.23:6824","nonce":2573691967},{"type":"v1","addr":"10.20.193.23:6825","nonce":2573691967}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.23:6822","nonce":2573691967},{"type":"v1","addr":"10.20.193.23:6823","nonce":2573691967}]},"public_addr":"10.20.193.23:6819/2573691967","cluster_addr":"10.20.193.23:6821/2573691967","heartbeat_back_addr":"10.20.193.23:6825/2573691967","heartbeat_front_addr":"10.20.193.23:6823/2573691967","state":["exists","up"]},{"osd":1,"uuid":"f73635a0-7d9b-4967-9505-3f230f1a9463","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.23:6802","nonce":3012239351},{"type":"v1","addr":"10.20.193.23:6803","nonce":3012239351}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.23:6804","nonce":3012239351},{"type":"v1","addr":"10.20.193.23:6805","nonce":3012239351}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.23:6808","nonce":3012239351},{"type":"v1","addr":"10.20.193.23:6809","nonce":3012239351}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.23:6806","nonce":3012239351},{"type":"v1","addr":"10.20.193.23:6807","nonce":3012239351}]},"public_addr":"10.20.193.23:6803/3012239351","cluster_addr":"10.20.193.23:6805/3012239351","heartbeat_back_addr":"10.20.193.23:6809/3012239351","heartbeat_front_addr":"10.20.193.23:6807/3012239351","state":["exists","up"]},{"osd":2,"uuid":"23a9de65-37ff-46ca-83ab-c20c149a2276","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.23:6810","nonce":1246129704},{"type":"v1","addr":"10.20.193.23:6811","nonce":1246129704}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.23:6812","nonce":1246129704},{"type":"v1","addr":"10.20.193.23:6813","nonce":1246129704}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.23:6816","nonce":1246129704},{"type":"v1","addr":"10.20.193.23:6817","nonce":1246129704}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.23:6814","nonce":1246129704},{"type":"v1","addr":"10.20.193.23:6815","nonce":1246129704}]},"public_addr":"10.20.193.23:6811/1246129704","cluster_addr":"10.20.193.23:6813/1246129704","heartbeat_back_addr":"10.20.193.23:6817/1246129704","heartbeat_front_addr":"10.20.193.23:6815/1246129704","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-22T22:34:04.148 DEBUG:teuthology.misc:3 of 3 OSDs are up 2026-02-22T22:34:04.148 INFO:tasks.ceph:Creating RBD pool 2026-02-22T22:34:04.148 DEBUG:teuthology.orchestra.run.trial023:> sudo ceph --cluster ceph osd pool create rbd 8 2026-02-22T22:34:05.232 INFO:teuthology.orchestra.run.trial023.stderr:pool 'rbd' created 2026-02-22T22:34:05.244 DEBUG:teuthology.orchestra.run.trial023:> rbd --cluster ceph pool init rbd 2026-02-22T22:34:08.249 INFO:tasks.ceph:Starting mds daemons in cluster ceph... 2026-02-22T22:34:08.250 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2026-02-22T22:34:08.250 INFO:tasks.ceph.ceph_manager.ceph:waiting for mgr available 2026-02-22T22:34:08.250 DEBUG:teuthology.orchestra.run.trial023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr dump --format=json 2026-02-22T22:34:08.250 INFO:tasks.daemonwatchdog.daemon_watchdog:watchdog starting 2026-02-22T22:34:08.409 INFO:teuthology.orchestra.run.trial023.stdout: 2026-02-22T22:34:08.418 INFO:teuthology.orchestra.run.trial023.stdout:{"epoch":4,"flags":0,"active_gid":4102,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.23:6800","nonce":1130938466},{"type":"v1","addr":"10.20.193.23:6801","nonce":1130938466}]},"active_addr":"10.20.193.23:6801/1130938466","active_change":"2026-02-22T22:33:52.177024+0000","active_mgr_features":4540701547738038271,"available":true,"standbys":[],"modules":["iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"update_pg_upmap_activity":{"name":"update_pg_upmap_activity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Updates pg_upmap activity stats to be used in `balancer status detail`","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:10.4.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/loki:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.7.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.2.5","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.51.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/promtail:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba":{"name":"container_image_samba","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-server:devbuilds-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"quay.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"location of dashboards to include in grafana deployments","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"ssh_keepalive_count_max":{"name":"ssh_keepalive_count_max","type":"int","level":"advanced","flags":0,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"How many times ssh connections can fail liveness checks before the host is marked offline","long_desc":"","tags":[],"see_also":[]},"ssh_keepalive_interval":{"name":"ssh_keepalive_interval","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"How often ssh connections are checked for liveness","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_requests":{"name":"max_requests","type":"int","level":"advanced","flags":0,"default_value":"500","min":"","max":"","enum_allowed":[],"desc":"Maximum number of requests to keep in memory. When new request comes in, the oldest request will be removed if the number of requests exceeds the max request number. if un-finished request is removed, error message will be logged in the ceph-mgr log.","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"squid":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"force_disabled_modules":{},"last_failure_osd_epoch":0,"active_clients":[{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"10.20.193.23:0","nonce":38892817}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"10.20.193.23:0","nonce":538099720}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"10.20.193.23:0","nonce":3351396642}]}]} 2026-02-22T22:34:08.421 INFO:tasks.ceph.ceph_manager.ceph:mgr available! 2026-02-22T22:34:08.421 INFO:tasks.ceph.ceph_manager.ceph:waiting for all up 2026-02-22T22:34:08.421 DEBUG:teuthology.orchestra.run.trial023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2026-02-22T22:34:08.566 INFO:teuthology.orchestra.run.trial023.stdout: 2026-02-22T22:34:08.566 INFO:teuthology.orchestra.run.trial023.stdout:{"epoch":12,"fsid":"a296be36-532d-4148-b28e-0183b3968057","created":"2026-02-22T22:33:50.079561+0000","modified":"2026-02-22T22:34:08.241040+0000","last_up_change":"2026-02-22T22:33:59.193101+0000","last_in_change":"2026-02-22T22:33:50.733447+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":"rbd","create_time":"2026-02-22T22:34:04.294257+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"12","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":2,"snap_epoch":12,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"634275c9-6a87-4a66-8d94-04b38d5640bf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":9,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.23:6818","nonce":2573691967},{"type":"v1","addr":"10.20.193.23:6819","nonce":2573691967}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.23:6820","nonce":2573691967},{"type":"v1","addr":"10.20.193.23:6821","nonce":2573691967}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.23:6824","nonce":2573691967},{"type":"v1","addr":"10.20.193.23:6825","nonce":2573691967}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.23:6822","nonce":2573691967},{"type":"v1","addr":"10.20.193.23:6823","nonce":2573691967}]},"public_addr":"10.20.193.23:6819/2573691967","cluster_addr":"10.20.193.23:6821/2573691967","heartbeat_back_addr":"10.20.193.23:6825/2573691967","heartbeat_front_addr":"10.20.193.23:6823/2573691967","state":["exists","up"]},{"osd":1,"uuid":"f73635a0-7d9b-4967-9505-3f230f1a9463","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":9,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.23:6802","nonce":3012239351},{"type":"v1","addr":"10.20.193.23:6803","nonce":3012239351}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.23:6804","nonce":3012239351},{"type":"v1","addr":"10.20.193.23:6805","nonce":3012239351}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.23:6808","nonce":3012239351},{"type":"v1","addr":"10.20.193.23:6809","nonce":3012239351}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.23:6806","nonce":3012239351},{"type":"v1","addr":"10.20.193.23:6807","nonce":3012239351}]},"public_addr":"10.20.193.23:6803/3012239351","cluster_addr":"10.20.193.23:6805/3012239351","heartbeat_back_addr":"10.20.193.23:6809/3012239351","heartbeat_front_addr":"10.20.193.23:6807/3012239351","state":["exists","up"]},{"osd":2,"uuid":"23a9de65-37ff-46ca-83ab-c20c149a2276","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":9,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.23:6810","nonce":1246129704},{"type":"v1","addr":"10.20.193.23:6811","nonce":1246129704}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.23:6812","nonce":1246129704},{"type":"v1","addr":"10.20.193.23:6813","nonce":1246129704}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.23:6816","nonce":1246129704},{"type":"v1","addr":"10.20.193.23:6817","nonce":1246129704}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.23:6814","nonce":1246129704},{"type":"v1","addr":"10.20.193.23:6815","nonce":1246129704}]},"public_addr":"10.20.193.23:6811/1246129704","cluster_addr":"10.20.193.23:6813/1246129704","heartbeat_back_addr":"10.20.193.23:6817/1246129704","heartbeat_front_addr":"10.20.193.23:6815/1246129704","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":1,"snaps":[{"begin":2,"length":1}]}],"new_removed_snaps":[{"pool":1,"snaps":[{"begin":2,"length":1}]}],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-22T22:34:08.578 INFO:tasks.ceph.ceph_manager.ceph:all up! 2026-02-22T22:34:08.578 DEBUG:teuthology.orchestra.run.trial023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2026-02-22T22:34:08.732 INFO:teuthology.orchestra.run.trial023.stdout: 2026-02-22T22:34:08.732 INFO:teuthology.orchestra.run.trial023.stdout:{"epoch":12,"fsid":"a296be36-532d-4148-b28e-0183b3968057","created":"2026-02-22T22:33:50.079561+0000","modified":"2026-02-22T22:34:08.241040+0000","last_up_change":"2026-02-22T22:33:59.193101+0000","last_in_change":"2026-02-22T22:33:50.733447+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":"rbd","create_time":"2026-02-22T22:34:04.294257+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"12","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":2,"snap_epoch":12,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"634275c9-6a87-4a66-8d94-04b38d5640bf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":9,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.23:6818","nonce":2573691967},{"type":"v1","addr":"10.20.193.23:6819","nonce":2573691967}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.23:6820","nonce":2573691967},{"type":"v1","addr":"10.20.193.23:6821","nonce":2573691967}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.23:6824","nonce":2573691967},{"type":"v1","addr":"10.20.193.23:6825","nonce":2573691967}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.23:6822","nonce":2573691967},{"type":"v1","addr":"10.20.193.23:6823","nonce":2573691967}]},"public_addr":"10.20.193.23:6819/2573691967","cluster_addr":"10.20.193.23:6821/2573691967","heartbeat_back_addr":"10.20.193.23:6825/2573691967","heartbeat_front_addr":"10.20.193.23:6823/2573691967","state":["exists","up"]},{"osd":1,"uuid":"f73635a0-7d9b-4967-9505-3f230f1a9463","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":9,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.23:6802","nonce":3012239351},{"type":"v1","addr":"10.20.193.23:6803","nonce":3012239351}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.23:6804","nonce":3012239351},{"type":"v1","addr":"10.20.193.23:6805","nonce":3012239351}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.23:6808","nonce":3012239351},{"type":"v1","addr":"10.20.193.23:6809","nonce":3012239351}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.23:6806","nonce":3012239351},{"type":"v1","addr":"10.20.193.23:6807","nonce":3012239351}]},"public_addr":"10.20.193.23:6803/3012239351","cluster_addr":"10.20.193.23:6805/3012239351","heartbeat_back_addr":"10.20.193.23:6809/3012239351","heartbeat_front_addr":"10.20.193.23:6807/3012239351","state":["exists","up"]},{"osd":2,"uuid":"23a9de65-37ff-46ca-83ab-c20c149a2276","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":9,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.23:6810","nonce":1246129704},{"type":"v1","addr":"10.20.193.23:6811","nonce":1246129704}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.23:6812","nonce":1246129704},{"type":"v1","addr":"10.20.193.23:6813","nonce":1246129704}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.23:6816","nonce":1246129704},{"type":"v1","addr":"10.20.193.23:6817","nonce":1246129704}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.23:6814","nonce":1246129704},{"type":"v1","addr":"10.20.193.23:6815","nonce":1246129704}]},"public_addr":"10.20.193.23:6811/1246129704","cluster_addr":"10.20.193.23:6813/1246129704","heartbeat_back_addr":"10.20.193.23:6817/1246129704","heartbeat_front_addr":"10.20.193.23:6815/1246129704","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":1,"snaps":[{"begin":2,"length":1}]}],"new_removed_snaps":[{"pool":1,"snaps":[{"begin":2,"length":1}]}],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-22T22:34:08.743 DEBUG:teuthology.orchestra.run.trial023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2026-02-22T22:34:08.743 DEBUG:teuthology.orchestra.run.trial023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2026-02-22T22:34:08.744 DEBUG:teuthology.orchestra.run.trial023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2026-02-22T22:34:08.827 INFO:teuthology.orchestra.run.trial023.stdout:34359738371 2026-02-22T22:34:08.828 DEBUG:teuthology.orchestra.run.trial023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2026-02-22T22:34:08.829 INFO:teuthology.orchestra.run.trial023.stdout:34359738371 2026-02-22T22:34:08.829 DEBUG:teuthology.orchestra.run.trial023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2026-02-22T22:34:08.835 INFO:teuthology.orchestra.run.trial023.stdout:34359738371 2026-02-22T22:34:08.836 DEBUG:teuthology.orchestra.run.trial023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2026-02-22T22:34:08.984 INFO:teuthology.orchestra.run.trial023.stdout:34359738370 2026-02-22T22:34:08.995 INFO:tasks.ceph.ceph_manager.ceph:need seq 34359738371 got 34359738370 for osd.2 2026-02-22T22:34:09.019 INFO:teuthology.orchestra.run.trial023.stdout:34359738370 2026-02-22T22:34:09.024 INFO:teuthology.orchestra.run.trial023.stdout:34359738370 2026-02-22T22:34:09.030 INFO:tasks.ceph.ceph_manager.ceph:need seq 34359738371 got 34359738370 for osd.1 2026-02-22T22:34:09.035 INFO:tasks.ceph.ceph_manager.ceph:need seq 34359738371 got 34359738370 for osd.0 2026-02-22T22:34:09.996 DEBUG:teuthology.orchestra.run.trial023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2026-02-22T22:34:10.031 DEBUG:teuthology.orchestra.run.trial023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2026-02-22T22:34:10.036 DEBUG:teuthology.orchestra.run.trial023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2026-02-22T22:34:10.142 INFO:teuthology.orchestra.run.trial023.stdout:34359738370 2026-02-22T22:34:10.153 INFO:tasks.ceph.ceph_manager.ceph:need seq 34359738371 got 34359738370 for osd.2 2026-02-22T22:34:10.183 INFO:teuthology.orchestra.run.trial023.stdout:34359738370 2026-02-22T22:34:10.189 INFO:teuthology.orchestra.run.trial023.stdout:34359738370 2026-02-22T22:34:10.195 INFO:tasks.ceph.ceph_manager.ceph:need seq 34359738371 got 34359738370 for osd.1 2026-02-22T22:34:10.200 INFO:tasks.ceph.ceph_manager.ceph:need seq 34359738371 got 34359738370 for osd.0 2026-02-22T22:34:11.154 DEBUG:teuthology.orchestra.run.trial023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2026-02-22T22:34:11.195 DEBUG:teuthology.orchestra.run.trial023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2026-02-22T22:34:11.201 DEBUG:teuthology.orchestra.run.trial023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2026-02-22T22:34:11.314 INFO:teuthology.orchestra.run.trial023.stdout:34359738371 2026-02-22T22:34:11.326 INFO:tasks.ceph.ceph_manager.ceph:need seq 34359738371 got 34359738371 for osd.2 2026-02-22T22:34:11.327 DEBUG:teuthology.parallel:result is None 2026-02-22T22:34:11.338 INFO:teuthology.orchestra.run.trial023.stdout:34359738371 2026-02-22T22:34:11.350 INFO:tasks.ceph.ceph_manager.ceph:need seq 34359738371 got 34359738371 for osd.1 2026-02-22T22:34:11.350 DEBUG:teuthology.parallel:result is None 2026-02-22T22:34:11.386 INFO:teuthology.orchestra.run.trial023.stdout:34359738371 2026-02-22T22:34:11.399 INFO:tasks.ceph.ceph_manager.ceph:need seq 34359738371 got 34359738371 for osd.0 2026-02-22T22:34:11.400 DEBUG:teuthology.parallel:result is None 2026-02-22T22:34:11.400 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2026-02-22T22:34:11.400 DEBUG:teuthology.orchestra.run.trial023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-22T22:34:11.583 INFO:teuthology.orchestra.run.trial023.stdout: 2026-02-22T22:34:11.583 INFO:teuthology.orchestra.run.trial023.stderr:dumped all 2026-02-22T22:34:11.595 INFO:teuthology.orchestra.run.trial023.stdout:{"pg_ready":true,"pg_map":{"version":15,"stamp":"2026-02-22T22:34:10.182279+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":12,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":81200,"kb_used_data":296,"kb_used_omap":4,"kb_used_meta":80827,"kb_avail":283034320,"statfs":{"total":289910292480,"available":289827143680,"internally_reserved":0,"allocated":303104,"data_stored":72171,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4781,"internal_metadata":82767187},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"0.000000"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":19,"reported_epoch":12,"state":"active+clean","last_fresh":"2026-02-22T22:34:08.245258+0000","last_change":"2026-02-22T22:34:08.245258+0000","last_active":"2026-02-22T22:34:08.245258+0000","last_peered":"2026-02-22T22:34:08.245258+0000","last_clean":"2026-02-22T22:34:08.245258+0000","last_became_active":"2026-02-22T22:34:06.240275+0000","last_became_peered":"2026-02-22T22:34:06.240275+0000","last_unstale":"2026-02-22T22:34:08.245258+0000","last_undegraded":"2026-02-22T22:34:08.245258+0000","last_fullsized":"2026-02-22T22:34:08.245258+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-22T22:34:05.232680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-22T22:34:05.232680+0000","last_clean_scrub_stamp":"2026-02-22T22:34:05.232680+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-24T06:53:55.503419+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00045264099999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"1.6","version":"0'0","reported_seq":19,"reported_epoch":12,"state":"active+clean","last_fresh":"2026-02-22T22:34:08.245426+0000","last_change":"2026-02-22T22:34:08.245426+0000","last_active":"2026-02-22T22:34:08.245426+0000","last_peered":"2026-02-22T22:34:08.245426+0000","last_clean":"2026-02-22T22:34:08.245426+0000","last_became_active":"2026-02-22T22:34:06.240348+0000","last_became_peered":"2026-02-22T22:34:06.240348+0000","last_unstale":"2026-02-22T22:34:08.245426+0000","last_undegraded":"2026-02-22T22:34:08.245426+0000","last_fullsized":"2026-02-22T22:34:08.245426+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-22T22:34:05.232680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-22T22:34:05.232680+0000","last_clean_scrub_stamp":"2026-02-22T22:34:05.232680+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-24T10:29:54.383830+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00060176900000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"1.5","version":"0'0","reported_seq":17,"reported_epoch":12,"state":"active+clean","last_fresh":"2026-02-22T22:34:08.784642+0000","last_change":"2026-02-22T22:34:08.784642+0000","last_active":"2026-02-22T22:34:08.784642+0000","last_peered":"2026-02-22T22:34:08.784642+0000","last_clean":"2026-02-22T22:34:08.784642+0000","last_became_active":"2026-02-22T22:34:06.240413+0000","last_became_peered":"2026-02-22T22:34:06.240413+0000","last_unstale":"2026-02-22T22:34:08.784642+0000","last_undegraded":"2026-02-22T22:34:08.784642+0000","last_fullsized":"2026-02-22T22:34:08.784642+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-22T22:34:05.232680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-22T22:34:05.232680+0000","last_clean_scrub_stamp":"2026-02-22T22:34:05.232680+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-24T02:21:57.682647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00032151800000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"1.4","version":"0'0","reported_seq":19,"reported_epoch":12,"state":"active+clean","last_fresh":"2026-02-22T22:34:08.245098+0000","last_change":"2026-02-22T22:34:08.245098+0000","last_active":"2026-02-22T22:34:08.245098+0000","last_peered":"2026-02-22T22:34:08.245098+0000","last_clean":"2026-02-22T22:34:08.245098+0000","last_became_active":"2026-02-22T22:34:06.240718+0000","last_became_peered":"2026-02-22T22:34:06.240718+0000","last_unstale":"2026-02-22T22:34:08.245098+0000","last_undegraded":"2026-02-22T22:34:08.245098+0000","last_fullsized":"2026-02-22T22:34:08.245098+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-22T22:34:05.232680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-22T22:34:05.232680+0000","last_clean_scrub_stamp":"2026-02-22T22:34:05.232680+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-24T09:45:44.284805+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000308413,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"1.3","version":"10'1","reported_seq":20,"reported_epoch":12,"state":"active+clean","last_fresh":"2026-02-22T22:34:08.245251+0000","last_change":"2026-02-22T22:34:08.245251+0000","last_active":"2026-02-22T22:34:08.245251+0000","last_peered":"2026-02-22T22:34:08.245251+0000","last_clean":"2026-02-22T22:34:08.245251+0000","last_became_active":"2026-02-22T22:34:06.240211+0000","last_became_peered":"2026-02-22T22:34:06.240211+0000","last_unstale":"2026-02-22T22:34:08.245251+0000","last_undegraded":"2026-02-22T22:34:08.245251+0000","last_fullsized":"2026-02-22T22:34:08.245251+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-22T22:34:05.232680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-22T22:34:05.232680+0000","last_clean_scrub_stamp":"2026-02-22T22:34:05.232680+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-24T10:28:10.379787+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00044206200000000001,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"1.2","version":"12'2","reported_seq":21,"reported_epoch":12,"state":"active+clean","last_fresh":"2026-02-22T22:34:08.246941+0000","last_change":"2026-02-22T22:34:08.246631+0000","last_active":"2026-02-22T22:34:08.246941+0000","last_peered":"2026-02-22T22:34:08.246941+0000","last_clean":"2026-02-22T22:34:08.246941+0000","last_became_active":"2026-02-22T22:34:06.239616+0000","last_became_peered":"2026-02-22T22:34:06.239616+0000","last_unstale":"2026-02-22T22:34:08.246941+0000","last_undegraded":"2026-02-22T22:34:08.246941+0000","last_fullsized":"2026-02-22T22:34:08.246941+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-22T22:34:05.232680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-22T22:34:05.232680+0000","last_clean_scrub_stamp":"2026-02-22T22:34:05.232680+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-24T08:37:35.943249+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00064906600000000001,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"1.1","version":"0'0","reported_seq":17,"reported_epoch":12,"state":"active+clean","last_fresh":"2026-02-22T22:34:08.784594+0000","last_change":"2026-02-22T22:34:08.784594+0000","last_active":"2026-02-22T22:34:08.784594+0000","last_peered":"2026-02-22T22:34:08.784594+0000","last_clean":"2026-02-22T22:34:08.784594+0000","last_became_active":"2026-02-22T22:34:06.240408+0000","last_became_peered":"2026-02-22T22:34:06.240408+0000","last_unstale":"2026-02-22T22:34:08.784594+0000","last_undegraded":"2026-02-22T22:34:08.784594+0000","last_fullsized":"2026-02-22T22:34:08.784594+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-22T22:34:05.232680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-22T22:34:05.232680+0000","last_clean_scrub_stamp":"2026-02-22T22:34:05.232680+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-24T01:24:51.802052+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00027554199999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"1.0","version":"0'0","reported_seq":19,"reported_epoch":12,"state":"active+clean","last_fresh":"2026-02-22T22:34:08.245065+0000","last_change":"2026-02-22T22:34:08.245065+0000","last_active":"2026-02-22T22:34:08.245065+0000","last_peered":"2026-02-22T22:34:08.245065+0000","last_clean":"2026-02-22T22:34:08.245065+0000","last_became_active":"2026-02-22T22:34:06.239799+0000","last_became_peered":"2026-02-22T22:34:06.239799+0000","last_unstale":"2026-02-22T22:34:08.245065+0000","last_undegraded":"2026-02-22T22:34:08.245065+0000","last_fullsized":"2026-02-22T22:34:08.245065+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-22T22:34:05.232680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-22T22:34:05.232680+0000","last_clean_scrub_stamp":"2026-02-22T22:34:05.232680+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-24T10:02:16.339895+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00028225400000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738371,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27052,"kb_used_data":88,"kb_used_omap":1,"kb_used_meta":26942,"kb_avail":94344788,"statfs":{"total":96636764160,"available":96609062912,"internally_reserved":0,"allocated":90112,"data_stored":20930,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27589063},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":8,"seq":34359738371,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27068,"kb_used_data":100,"kb_used_omap":1,"kb_used_meta":26942,"kb_avail":94344772,"statfs":{"total":96636764160,"available":96609046528,"internally_reserved":0,"allocated":102400,"data_stored":24028,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27589062},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738371,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27080,"kb_used_data":108,"kb_used_omap":1,"kb_used_meta":26942,"kb_avail":94344760,"statfs":{"total":96636764160,"available":96609034240,"internally_reserved":0,"allocated":110592,"data_stored":27213,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27589062},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-22T22:34:11.595 DEBUG:teuthology.orchestra.run.trial023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-22T22:34:11.751 INFO:teuthology.orchestra.run.trial023.stdout: 2026-02-22T22:34:11.751 INFO:teuthology.orchestra.run.trial023.stderr:dumped all 2026-02-22T22:34:11.761 INFO:teuthology.orchestra.run.trial023.stdout:{"pg_ready":true,"pg_map":{"version":15,"stamp":"2026-02-22T22:34:10.182279+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":12,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":81200,"kb_used_data":296,"kb_used_omap":4,"kb_used_meta":80827,"kb_avail":283034320,"statfs":{"total":289910292480,"available":289827143680,"internally_reserved":0,"allocated":303104,"data_stored":72171,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4781,"internal_metadata":82767187},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"0.000000"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":19,"reported_epoch":12,"state":"active+clean","last_fresh":"2026-02-22T22:34:08.245258+0000","last_change":"2026-02-22T22:34:08.245258+0000","last_active":"2026-02-22T22:34:08.245258+0000","last_peered":"2026-02-22T22:34:08.245258+0000","last_clean":"2026-02-22T22:34:08.245258+0000","last_became_active":"2026-02-22T22:34:06.240275+0000","last_became_peered":"2026-02-22T22:34:06.240275+0000","last_unstale":"2026-02-22T22:34:08.245258+0000","last_undegraded":"2026-02-22T22:34:08.245258+0000","last_fullsized":"2026-02-22T22:34:08.245258+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-22T22:34:05.232680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-22T22:34:05.232680+0000","last_clean_scrub_stamp":"2026-02-22T22:34:05.232680+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-24T06:53:55.503419+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00045264099999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"1.6","version":"0'0","reported_seq":19,"reported_epoch":12,"state":"active+clean","last_fresh":"2026-02-22T22:34:08.245426+0000","last_change":"2026-02-22T22:34:08.245426+0000","last_active":"2026-02-22T22:34:08.245426+0000","last_peered":"2026-02-22T22:34:08.245426+0000","last_clean":"2026-02-22T22:34:08.245426+0000","last_became_active":"2026-02-22T22:34:06.240348+0000","last_became_peered":"2026-02-22T22:34:06.240348+0000","last_unstale":"2026-02-22T22:34:08.245426+0000","last_undegraded":"2026-02-22T22:34:08.245426+0000","last_fullsized":"2026-02-22T22:34:08.245426+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-22T22:34:05.232680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-22T22:34:05.232680+0000","last_clean_scrub_stamp":"2026-02-22T22:34:05.232680+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-24T10:29:54.383830+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00060176900000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"1.5","version":"0'0","reported_seq":17,"reported_epoch":12,"state":"active+clean","last_fresh":"2026-02-22T22:34:08.784642+0000","last_change":"2026-02-22T22:34:08.784642+0000","last_active":"2026-02-22T22:34:08.784642+0000","last_peered":"2026-02-22T22:34:08.784642+0000","last_clean":"2026-02-22T22:34:08.784642+0000","last_became_active":"2026-02-22T22:34:06.240413+0000","last_became_peered":"2026-02-22T22:34:06.240413+0000","last_unstale":"2026-02-22T22:34:08.784642+0000","last_undegraded":"2026-02-22T22:34:08.784642+0000","last_fullsized":"2026-02-22T22:34:08.784642+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-22T22:34:05.232680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-22T22:34:05.232680+0000","last_clean_scrub_stamp":"2026-02-22T22:34:05.232680+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-24T02:21:57.682647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00032151800000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"1.4","version":"0'0","reported_seq":19,"reported_epoch":12,"state":"active+clean","last_fresh":"2026-02-22T22:34:08.245098+0000","last_change":"2026-02-22T22:34:08.245098+0000","last_active":"2026-02-22T22:34:08.245098+0000","last_peered":"2026-02-22T22:34:08.245098+0000","last_clean":"2026-02-22T22:34:08.245098+0000","last_became_active":"2026-02-22T22:34:06.240718+0000","last_became_peered":"2026-02-22T22:34:06.240718+0000","last_unstale":"2026-02-22T22:34:08.245098+0000","last_undegraded":"2026-02-22T22:34:08.245098+0000","last_fullsized":"2026-02-22T22:34:08.245098+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-22T22:34:05.232680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-22T22:34:05.232680+0000","last_clean_scrub_stamp":"2026-02-22T22:34:05.232680+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-24T09:45:44.284805+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000308413,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"1.3","version":"10'1","reported_seq":20,"reported_epoch":12,"state":"active+clean","last_fresh":"2026-02-22T22:34:08.245251+0000","last_change":"2026-02-22T22:34:08.245251+0000","last_active":"2026-02-22T22:34:08.245251+0000","last_peered":"2026-02-22T22:34:08.245251+0000","last_clean":"2026-02-22T22:34:08.245251+0000","last_became_active":"2026-02-22T22:34:06.240211+0000","last_became_peered":"2026-02-22T22:34:06.240211+0000","last_unstale":"2026-02-22T22:34:08.245251+0000","last_undegraded":"2026-02-22T22:34:08.245251+0000","last_fullsized":"2026-02-22T22:34:08.245251+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-22T22:34:05.232680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-22T22:34:05.232680+0000","last_clean_scrub_stamp":"2026-02-22T22:34:05.232680+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-24T10:28:10.379787+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00044206200000000001,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"1.2","version":"12'2","reported_seq":21,"reported_epoch":12,"state":"active+clean","last_fresh":"2026-02-22T22:34:08.246941+0000","last_change":"2026-02-22T22:34:08.246631+0000","last_active":"2026-02-22T22:34:08.246941+0000","last_peered":"2026-02-22T22:34:08.246941+0000","last_clean":"2026-02-22T22:34:08.246941+0000","last_became_active":"2026-02-22T22:34:06.239616+0000","last_became_peered":"2026-02-22T22:34:06.239616+0000","last_unstale":"2026-02-22T22:34:08.246941+0000","last_undegraded":"2026-02-22T22:34:08.246941+0000","last_fullsized":"2026-02-22T22:34:08.246941+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-22T22:34:05.232680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-22T22:34:05.232680+0000","last_clean_scrub_stamp":"2026-02-22T22:34:05.232680+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-24T08:37:35.943249+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00064906600000000001,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"1.1","version":"0'0","reported_seq":17,"reported_epoch":12,"state":"active+clean","last_fresh":"2026-02-22T22:34:08.784594+0000","last_change":"2026-02-22T22:34:08.784594+0000","last_active":"2026-02-22T22:34:08.784594+0000","last_peered":"2026-02-22T22:34:08.784594+0000","last_clean":"2026-02-22T22:34:08.784594+0000","last_became_active":"2026-02-22T22:34:06.240408+0000","last_became_peered":"2026-02-22T22:34:06.240408+0000","last_unstale":"2026-02-22T22:34:08.784594+0000","last_undegraded":"2026-02-22T22:34:08.784594+0000","last_fullsized":"2026-02-22T22:34:08.784594+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-22T22:34:05.232680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-22T22:34:05.232680+0000","last_clean_scrub_stamp":"2026-02-22T22:34:05.232680+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-24T01:24:51.802052+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00027554199999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"1.0","version":"0'0","reported_seq":19,"reported_epoch":12,"state":"active+clean","last_fresh":"2026-02-22T22:34:08.245065+0000","last_change":"2026-02-22T22:34:08.245065+0000","last_active":"2026-02-22T22:34:08.245065+0000","last_peered":"2026-02-22T22:34:08.245065+0000","last_clean":"2026-02-22T22:34:08.245065+0000","last_became_active":"2026-02-22T22:34:06.239799+0000","last_became_peered":"2026-02-22T22:34:06.239799+0000","last_unstale":"2026-02-22T22:34:08.245065+0000","last_undegraded":"2026-02-22T22:34:08.245065+0000","last_fullsized":"2026-02-22T22:34:08.245065+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-22T22:34:05.232680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-22T22:34:05.232680+0000","last_clean_scrub_stamp":"2026-02-22T22:34:05.232680+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-24T10:02:16.339895+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00028225400000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738371,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27052,"kb_used_data":88,"kb_used_omap":1,"kb_used_meta":26942,"kb_avail":94344788,"statfs":{"total":96636764160,"available":96609062912,"internally_reserved":0,"allocated":90112,"data_stored":20930,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27589063},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":8,"seq":34359738371,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27068,"kb_used_data":100,"kb_used_omap":1,"kb_used_meta":26942,"kb_avail":94344772,"statfs":{"total":96636764160,"available":96609046528,"internally_reserved":0,"allocated":102400,"data_stored":24028,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27589062},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738371,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27080,"kb_used_data":108,"kb_used_omap":1,"kb_used_meta":26942,"kb_avail":94344760,"statfs":{"total":96636764160,"available":96609034240,"internally_reserved":0,"allocated":110592,"data_stored":27213,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27589062},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-22T22:34:11.762 INFO:tasks.ceph.ceph_manager.ceph:clean! 2026-02-22T22:34:11.762 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2026-02-22T22:34:11.762 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy 2026-02-22T22:34:11.762 DEBUG:teuthology.orchestra.run.trial023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2026-02-22T22:34:11.919 INFO:teuthology.orchestra.run.trial023.stdout: 2026-02-22T22:34:11.919 INFO:teuthology.orchestra.run.trial023.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2026-02-22T22:34:11.929 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy done 2026-02-22T22:34:11.929 INFO:teuthology.run_tasks:Running task full_sequential... 2026-02-22T22:34:11.941 INFO:teuthology.task.full_sequential:In full_sequential, running task radosbench... 2026-02-22T22:34:11.943 INFO:tasks.radosbench:Beginning radosbench... 2026-02-22T22:34:11.943 INFO:tasks.ceph.ceph_manager.ceph:creating pool_name unique_pool_0 2026-02-22T22:34:11.943 DEBUG:teuthology.orchestra.run.trial023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create unique_pool_0 16 2026-02-22T22:34:12.370 INFO:teuthology.orchestra.run.trial023.stderr:pool 'unique_pool_0' created 2026-02-22T22:34:12.382 DEBUG:teuthology.orchestra.run.trial023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool application enable unique_pool_0 rados --yes-i-really-mean-it || true 2026-02-22T22:34:13.374 INFO:teuthology.orchestra.run.trial023.stderr:enabled application 'rados' on pool 'unique_pool_0' 2026-02-22T22:34:14.389 DEBUG:teuthology.orchestra.run.trial023:> /bin/sh -c 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --no-log-to-stderr --name client.0 -b 65536 --object-size 65536 -p unique_pool_0 bench 30 write' 2026-02-22T22:34:14.390 INFO:tasks.radosbench:joining radosbench (timing out after 1200s) 2026-02-22T22:34:14.390 INFO:teuthology.orchestra.run:waiting for 1200 2026-02-22T22:34:14.414 INFO:tasks.radosbench.radosbench.0.trial023.stdout:hints = 1 2026-02-22T22:34:14.414 INFO:tasks.radosbench.radosbench.0.trial023.stdout:Maintaining 16 concurrent writes of 65536 bytes to objects of size 65536 for up to 30 seconds or 0 objects 2026-02-22T22:34:14.414 INFO:tasks.radosbench.radosbench.0.trial023.stdout:Object prefix: benchmark_data_trial023_19642 2026-02-22T22:34:14.414 INFO:tasks.radosbench.radosbench.0.trial023.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2026-02-22T22:34:14.414 INFO:tasks.radosbench.radosbench.0.trial023.stdout: 0 0 0 0 0 0 - 0 2026-02-22T22:34:15.414 INFO:tasks.radosbench.radosbench.0.trial023.stdout: 1 16 6452 6436 402.217 402.25 0.00248936 0.0024822 2026-02-22T22:34:16.414 INFO:tasks.radosbench.radosbench.0.trial023.stdout: 2 16 14490 14474 452.277 502.375 0.00190891 0.00220822 2026-02-22T22:34:17.414 INFO:tasks.radosbench.radosbench.0.trial023.stdout: 3 16 21749 21733 452.736 453.688 0.000894672 0.00218264 2026-02-22T22:34:18.415 INFO:tasks.radosbench.radosbench.0.trial023.stdout: 4 16 26032 26016 406.47 267.688 0.00186596 0.00245789 2026-02-22T22:34:19.415 INFO:tasks.radosbench.radosbench.0.trial023.stdout: 5 16 32759 32743 409.257 420.438 0.00192426 0.00244123 2026-02-22T22:34:20.415 INFO:tasks.radosbench.radosbench.0.trial023.stdout: 6 16 39008 38992 406.137 390.562 0.00279035 0.00246001 2026-02-22T22:34:21.415 INFO:tasks.radosbench.radosbench.0.trial023.stdout: 7 16 44865 44849 400.408 366.062 0.0017523 0.00249537 2026-02-22T22:34:22.415 INFO:tasks.radosbench.radosbench.0.trial023.stdout: 8 16 50430 50414 393.83 347.812 0.00328044 0.00253715 2026-02-22T22:34:23.415 INFO:tasks.radosbench.radosbench.0.trial023.stdout: 9 16 55588 55572 385.888 322.375 0.00303687 0.00258913 2026-02-22T22:34:24.415 INFO:tasks.radosbench.radosbench.0.trial023.stdout: 10 16 60568 60552 378.422 311.25 0.00227182 0.00264038 2026-02-22T22:34:25.415 INFO:tasks.radosbench.radosbench.0.trial023.stdout: 11 16 65284 65268 370.814 294.75 0.00327499 0.00269454 2026-02-22T22:34:26.415 INFO:tasks.radosbench.radosbench.0.trial023.stdout: 12 16 69761 69745 363.228 279.812 0.00307723 0.00275116 2026-02-22T22:34:27.415 INFO:tasks.radosbench.radosbench.0.trial023.stdout: 13 16 74051 74035 355.911 268.125 0.00425957 0.00280762 2026-02-22T22:34:28.415 INFO:tasks.radosbench.radosbench.0.trial023.stdout: 14 16 78154 78138 348.805 256.438 0.00290501 0.002865 2026-02-22T22:34:29.415 INFO:tasks.radosbench.radosbench.0.trial023.stdout: 15 16 82087 82071 341.937 245.812 0.00270593 0.00292247 2026-02-22T22:34:30.416 INFO:tasks.radosbench.radosbench.0.trial023.stdout: 16 16 85832 85816 335.194 234.062 0.00180377 0.00298115 2026-02-22T22:34:31.416 INFO:tasks.radosbench.radosbench.0.trial023.stdout: 17 16 89517 89501 329.023 230.312 0.00234354 0.00303739 2026-02-22T22:34:32.415 INFO:tasks.radosbench.radosbench.0.trial023.stdout: 18 16 93018 93002 322.9 218.812 0.00395622 0.0030949 2026-02-22T22:34:33.416 INFO:tasks.radosbench.radosbench.0.trial023.stdout: 19 16 96403 96387 317.039 211.562 0.00525207 0.00315216 2026-02-22T22:34:34.416 INFO:tasks.radosbench.radosbench.0.trial023.stdout:2026-02-22T22:34:34.418170+0000 min lat: 0.000754341 max lat: 0.209148 avg lat: 0.00320872 2026-02-22T22:34:34.416 INFO:tasks.radosbench.radosbench.0.trial023.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2026-02-22T22:34:34.416 INFO:tasks.radosbench.radosbench.0.trial023.stdout: 20 16 99689 99673 311.455 205.375 0.00279193 0.00320872 2026-02-22T22:34:35.416 INFO:tasks.radosbench.radosbench.0.trial023.stdout: 21 16 102786 102770 305.841 193.562 0.00586887 0.00326778 2026-02-22T22:34:36.416 INFO:tasks.radosbench.radosbench.0.trial023.stdout: 22 16 105806 105790 300.518 188.75 0.00359015 0.00332554 2026-02-22T22:34:37.416 INFO:tasks.radosbench.radosbench.0.trial023.stdout: 23 16 108747 108731 295.443 183.812 0.00647965 0.00338266 2026-02-22T22:34:38.416 INFO:tasks.radosbench.radosbench.0.trial023.stdout: 24 16 111556 111540 290.447 175.562 0.00285552 0.0034377 2026-02-22T22:34:39.416 INFO:tasks.radosbench.radosbench.0.trial023.stdout: 25 16 113772 113756 284.369 138.5 0.00352903 0.00351459 2026-02-22T22:34:40.416 INFO:tasks.radosbench.radosbench.0.trial023.stdout: 26 16 116587 116571 280.198 175.938 0.00764073 0.00356687 2026-02-22T22:34:41.416 INFO:tasks.radosbench.radosbench.0.trial023.stdout: 27 16 119321 119305 276.149 170.875 0.00587276 0.00361933 2026-02-22T22:34:42.416 INFO:tasks.radosbench.radosbench.0.trial023.stdout: 28 16 122012 121996 272.293 168.188 0.00342989 0.00367051 2026-02-22T22:34:43.416 INFO:tasks.radosbench.radosbench.0.trial023.stdout: 29 16 124601 124585 268.482 161.812 0.00959488 0.00372254 2026-02-22T22:34:44.416 INFO:tasks.radosbench.radosbench.0.trial023.stdout: 30 8 126585 126577 263.683 124.5 0.00539497 0.0037905 2026-02-22T22:34:45.417 INFO:tasks.radosbench.radosbench.0.trial023.stdout:Total time run: 30.0047 2026-02-22T22:34:45.417 INFO:tasks.radosbench.radosbench.0.trial023.stdout:Total writes made: 126585 2026-02-22T22:34:45.417 INFO:tasks.radosbench.radosbench.0.trial023.stdout:Write size: 65536 2026-02-22T22:34:45.417 INFO:tasks.radosbench.radosbench.0.trial023.stdout:Object size: 65536 2026-02-22T22:34:45.417 INFO:tasks.radosbench.radosbench.0.trial023.stdout:Bandwidth (MB/sec): 263.677 2026-02-22T22:34:45.417 INFO:tasks.radosbench.radosbench.0.trial023.stdout:Stddev Bandwidth: 98.8722 2026-02-22T22:34:45.417 INFO:tasks.radosbench.radosbench.0.trial023.stdout:Max bandwidth (MB/sec): 502.375 2026-02-22T22:34:45.418 INFO:tasks.radosbench.radosbench.0.trial023.stdout:Min bandwidth (MB/sec): 124.5 2026-02-22T22:34:45.418 INFO:tasks.radosbench.radosbench.0.trial023.stdout:Average IOPS: 4218 2026-02-22T22:34:45.418 INFO:tasks.radosbench.radosbench.0.trial023.stdout:Stddev IOPS: 1581.96 2026-02-22T22:34:45.418 INFO:tasks.radosbench.radosbench.0.trial023.stdout:Max IOPS: 8038 2026-02-22T22:34:45.418 INFO:tasks.radosbench.radosbench.0.trial023.stdout:Min IOPS: 1992 2026-02-22T22:34:45.418 INFO:tasks.radosbench.radosbench.0.trial023.stdout:Average Latency(s): 0.00379069 2026-02-22T22:34:45.418 INFO:tasks.radosbench.radosbench.0.trial023.stdout:Stddev Latency(s): 0.00571781 2026-02-22T22:34:45.418 INFO:tasks.radosbench.radosbench.0.trial023.stdout:Max latency(s): 0.25805 2026-02-22T22:34:45.418 INFO:tasks.radosbench.radosbench.0.trial023.stdout:Min latency(s): 0.000754341 2026-02-22T22:34:45.421 INFO:tasks.radosbench.radosbench.0.trial023.stdout:Cleaning up (deleting benchmark objects) 2026-02-22T22:35:41.132 INFO:tasks.radosbench.radosbench.0.trial023.stdout:Removed 126585 objects 2026-02-22T22:35:41.132 INFO:tasks.radosbench.radosbench.0.trial023.stdout:Clean up completed and total clean up time :55.7108 2026-02-22T22:35:45.906 INFO:tasks.ceph.ceph_manager.ceph:removing pool_name unique_pool_0 2026-02-22T22:35:45.906 DEBUG:teuthology.orchestra.run.trial023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm unique_pool_0 unique_pool_0 --yes-i-really-really-mean-it 2026-02-22T22:35:46.107 INFO:teuthology.orchestra.run.trial023.stderr:pool 'unique_pool_0' removed 2026-02-22T22:35:46.122 INFO:teuthology.task.full_sequential:In full_sequential, running task rebuild_mondb... 2026-02-22T22:35:46.127 DEBUG:teuthology.orchestra.run.trial023:> ceph mon getmap -o /tmp/monmap 2026-02-22T22:35:46.297 INFO:teuthology.orchestra.run.trial023.stderr:got monmap epoch 1 2026-02-22T22:35:46.308 DEBUG:teuthology.orchestra.run.trial023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2026-02-22T22:35:46.462 INFO:teuthology.orchestra.run.trial023.stdout: 2026-02-22T22:35:46.462 INFO:teuthology.orchestra.run.trial023.stdout:{"epoch":19,"fsid":"a296be36-532d-4148-b28e-0183b3968057","created":"2026-02-22T22:33:50.079561+0000","modified":"2026-02-22T22:35:46.107691+0000","last_up_change":"2026-02-22T22:33:59.193101+0000","last_in_change":"2026-02-22T22:33:50.733447+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":"rbd","create_time":"2026-02-22T22:34:04.294257+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"12","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":2,"snap_epoch":12,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"634275c9-6a87-4a66-8d94-04b38d5640bf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.23:6818","nonce":2573691967},{"type":"v1","addr":"10.20.193.23:6819","nonce":2573691967}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.23:6820","nonce":2573691967},{"type":"v1","addr":"10.20.193.23:6821","nonce":2573691967}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.23:6824","nonce":2573691967},{"type":"v1","addr":"10.20.193.23:6825","nonce":2573691967}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.23:6822","nonce":2573691967},{"type":"v1","addr":"10.20.193.23:6823","nonce":2573691967}]},"public_addr":"10.20.193.23:6819/2573691967","cluster_addr":"10.20.193.23:6821/2573691967","heartbeat_back_addr":"10.20.193.23:6825/2573691967","heartbeat_front_addr":"10.20.193.23:6823/2573691967","state":["exists","up"]},{"osd":1,"uuid":"f73635a0-7d9b-4967-9505-3f230f1a9463","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.23:6802","nonce":3012239351},{"type":"v1","addr":"10.20.193.23:6803","nonce":3012239351}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.23:6804","nonce":3012239351},{"type":"v1","addr":"10.20.193.23:6805","nonce":3012239351}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.23:6808","nonce":3012239351},{"type":"v1","addr":"10.20.193.23:6809","nonce":3012239351}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.23:6806","nonce":3012239351},{"type":"v1","addr":"10.20.193.23:6807","nonce":3012239351}]},"public_addr":"10.20.193.23:6803/3012239351","cluster_addr":"10.20.193.23:6805/3012239351","heartbeat_back_addr":"10.20.193.23:6809/3012239351","heartbeat_front_addr":"10.20.193.23:6807/3012239351","state":["exists","up"]},{"osd":2,"uuid":"23a9de65-37ff-46ca-83ab-c20c149a2276","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.23:6810","nonce":1246129704},{"type":"v1","addr":"10.20.193.23:6811","nonce":1246129704}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.23:6812","nonce":1246129704},{"type":"v1","addr":"10.20.193.23:6813","nonce":1246129704}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.23:6816","nonce":1246129704},{"type":"v1","addr":"10.20.193.23:6817","nonce":1246129704}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.23:6814","nonce":1246129704},{"type":"v1","addr":"10.20.193.23:6815","nonce":1246129704}]},"public_addr":"10.20.193.23:6811/1246129704","cluster_addr":"10.20.193.23:6813/1246129704","heartbeat_back_addr":"10.20.193.23:6817/1246129704","heartbeat_front_addr":"10.20.193.23:6815/1246129704","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2026-02-22T22:33:57.746206+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2026-02-22T22:33:57.411823+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2026-02-22T22:33:57.826496+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-22T22:35:46.472 INFO:tasks.rebuild_mondb.ceph_manager:[{'pool': 1, 'pool_name': 'rbd', 'create_time': '2026-02-22T22:34:04.294257+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '12', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 2, 'snap_epoch': 12, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2026-02-22T22:35:46.472 DEBUG:teuthology.orchestra.run.trial023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2026-02-22T22:35:46.637 INFO:teuthology.orchestra.run.trial023.stdout:pg_num: 8 2026-02-22T22:35:46.651 INFO:tasks.rebuild_mondb:killing ceph:mon.a 2026-02-22T22:35:46.651 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2026-02-22T22:35:46.651 INFO:teuthology.orchestra.run:waiting for 300 2026-02-22T22:35:46.664 INFO:tasks.ceph.mon.a:Stopped 2026-02-22T22:35:46.665 DEBUG:teuthology.orchestra.run.trial023:> sudo rm -r /var/lib/ceph/mon/ceph-a/store.db 2026-02-22T22:35:46.715 DEBUG:teuthology.orchestra.run.trial023:> sudo rm -r /var/lib/ceph/mon/ceph-a/external_log_to 2026-02-22T22:35:46.769 INFO:tasks.rebuild_mondb:killing ceph:mon.b 2026-02-22T22:35:46.769 DEBUG:tasks.ceph.mon.b:waiting for process to exit 2026-02-22T22:35:46.769 INFO:teuthology.orchestra.run:waiting for 300 2026-02-22T22:35:46.821 INFO:tasks.ceph.mon.b:Stopped 2026-02-22T22:35:46.821 DEBUG:teuthology.orchestra.run.trial023:> sudo rm -r /var/lib/ceph/mon/ceph-b 2026-02-22T22:35:46.873 INFO:tasks.rebuild_mondb:killing ceph:mon.c 2026-02-22T22:35:46.874 DEBUG:tasks.ceph.mon.c:waiting for process to exit 2026-02-22T22:35:46.874 INFO:teuthology.orchestra.run:waiting for 300 2026-02-22T22:35:46.928 INFO:tasks.ceph.mon.c:Stopped 2026-02-22T22:35:46.929 DEBUG:teuthology.orchestra.run.trial023:> sudo rm -r /var/lib/ceph/mon/ceph-c 2026-02-22T22:35:46.982 INFO:tasks.rebuild_mondb:collecting maps from ceph:osd.0 2026-02-22T22:35:46.983 DEBUG:teuthology.orchestra.run.trial023:> sudo mkdir -m o+x -p /home/ubuntu/cephtest/mon-store 2026-02-22T22:35:47.037 DEBUG:teuthology.misc::sh: tar cz -f /tmp/rebuild_mondb-4gsfgklr.tgz -C /tmp/tmpl1nbplf2 -- . 2026-02-22T22:35:47.088 DEBUG:teuthology.orchestra.run.trial023:> sudo tar xz -C /home/ubuntu/cephtest/mon-store -f /tmp/rebuild_mondb-_684reie.tgz 2026-02-22T22:35:47.099 DEBUG:teuthology.orchestra.run.trial023:> sudo rm -fr /tmp/rebuild_mondb-_684reie.tgz 2026-02-22T22:35:47.148 INFO:tasks.rebuild_mondb:rm -rf /tmp/tmpl1nbplf2 2026-02-22T22:35:47.148 INFO:tasks.rebuild_mondb:cot /home/ubuntu/cephtest/mon-store 2026-02-22T22:35:47.149 DEBUG:tasks.ceph.osd.0:waiting for process to exit 2026-02-22T22:35:47.149 INFO:teuthology.orchestra.run:waiting for 300 2026-02-22T22:35:51.226 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~0s 2026-02-22T22:35:53.250 INFO:tasks.ceph.osd.0:Stopped 2026-02-22T22:35:53.251 INFO:tasks.rebuild_mondb.ceph_manager:sudo adjust-ulimits ceph-objectstore-tool --data-path /var/lib/ceph/osd/ceph-0 --journal-path /var/lib/ceph/osd/ceph-0/journal --no-mon-config --op update-mon-db --mon-store-path /home/ubuntu/cephtest/mon-store 2026-02-22T22:35:53.251 DEBUG:teuthology.orchestra.run.trial023:> bash -e -x -c 'sudo adjust-ulimits ceph-objectstore-tool --data-path /var/lib/ceph/osd/ceph-0 --journal-path /var/lib/ceph/osd/ceph-0/journal --no-mon-config --op update-mon-db --mon-store-path /home/ubuntu/cephtest/mon-store ' 2026-02-22T22:35:53.256 INFO:teuthology.orchestra.run.trial023.stderr:+ '[' -z '' ']' 2026-02-22T22:35:53.256 INFO:teuthology.orchestra.run.trial023.stderr:+ return 2026-02-22T22:35:53.256 INFO:teuthology.orchestra.run.trial023.stderr:+ case $- in 2026-02-22T22:35:53.256 INFO:teuthology.orchestra.run.trial023.stderr:+ return 2026-02-22T22:35:53.257 INFO:teuthology.orchestra.run.trial023.stderr:+ sudo adjust-ulimits ceph-objectstore-tool --data-path /var/lib/ceph/osd/ceph-0 --journal-path /var/lib/ceph/osd/ceph-0/journal --no-mon-config --op update-mon-db --mon-store-path /home/ubuntu/cephtest/mon-store 2026-02-22T22:35:56.528 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 has been restored 2026-02-22T22:35:58.554 INFO:teuthology.orchestra.run.trial023.stdout:osd.0 : 0 osdmaps trimmed, 19 osdmaps added. 2026-02-22T22:35:59.104 INFO:tasks.rebuild_mondb:pull dir /home/ubuntu/cephtest/mon-store -> /tmp/tmpl1nbplf2 2026-02-22T22:35:59.104 DEBUG:teuthology.misc:Transferring archived files from trial023:/home/ubuntu/cephtest/mon-store to /tmp/tmp5yy42z5p 2026-02-22T22:35:59.105 DEBUG:teuthology.orchestra.run.trial023:> sudo tar c -f - -C /home/ubuntu/cephtest/mon-store -- . 2026-02-22T22:35:59.158 INFO:tasks.rebuild_mondb:rm -rf osd:/home/ubuntu/cephtest/mon-store 2026-02-22T22:35:59.158 DEBUG:teuthology.orchestra.run.trial023:> sudo rm -fr /home/ubuntu/cephtest/mon-store 2026-02-22T22:35:59.205 INFO:tasks.rebuild_mondb:collecting maps from ceph:osd.1 2026-02-22T22:35:59.206 DEBUG:teuthology.orchestra.run.trial023:> sudo mkdir -m o+x -p /home/ubuntu/cephtest/mon-store 2026-02-22T22:35:59.257 DEBUG:teuthology.misc::sh: tar cz -f /tmp/rebuild_mondb-zakdicc8.tgz -C /tmp/tmp5yy42z5p -- . 2026-02-22T22:35:59.304 DEBUG:teuthology.orchestra.run.trial023:> sudo tar xz -C /home/ubuntu/cephtest/mon-store -f /tmp/rebuild_mondb-e5x040aj.tgz 2026-02-22T22:35:59.315 DEBUG:teuthology.orchestra.run.trial023:> sudo rm -fr /tmp/rebuild_mondb-e5x040aj.tgz 2026-02-22T22:35:59.368 INFO:tasks.rebuild_mondb:rm -rf /tmp/tmp5yy42z5p 2026-02-22T22:35:59.369 INFO:tasks.rebuild_mondb:cot /home/ubuntu/cephtest/mon-store 2026-02-22T22:35:59.369 DEBUG:tasks.ceph.osd.1:waiting for process to exit 2026-02-22T22:35:59.369 INFO:teuthology.orchestra.run:waiting for 300 2026-02-22T22:36:01.730 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~0s 2026-02-22T22:36:05.470 INFO:tasks.ceph.osd.1:Stopped 2026-02-22T22:36:05.470 INFO:tasks.rebuild_mondb.ceph_manager:sudo adjust-ulimits ceph-objectstore-tool --data-path /var/lib/ceph/osd/ceph-1 --journal-path /var/lib/ceph/osd/ceph-1/journal --no-mon-config --op update-mon-db --mon-store-path /home/ubuntu/cephtest/mon-store 2026-02-22T22:36:05.471 DEBUG:teuthology.orchestra.run.trial023:> bash -e -x -c 'sudo adjust-ulimits ceph-objectstore-tool --data-path /var/lib/ceph/osd/ceph-1 --journal-path /var/lib/ceph/osd/ceph-1/journal --no-mon-config --op update-mon-db --mon-store-path /home/ubuntu/cephtest/mon-store ' 2026-02-22T22:36:05.477 INFO:teuthology.orchestra.run.trial023.stderr:+ '[' -z '' ']' 2026-02-22T22:36:05.477 INFO:teuthology.orchestra.run.trial023.stderr:+ return 2026-02-22T22:36:05.477 INFO:teuthology.orchestra.run.trial023.stderr:+ case $- in 2026-02-22T22:36:05.477 INFO:teuthology.orchestra.run.trial023.stderr:+ return 2026-02-22T22:36:05.477 INFO:teuthology.orchestra.run.trial023.stderr:+ sudo adjust-ulimits ceph-objectstore-tool --data-path /var/lib/ceph/osd/ceph-1 --journal-path /var/lib/ceph/osd/ceph-1/journal --no-mon-config --op update-mon-db --mon-store-path /home/ubuntu/cephtest/mon-store 2026-02-22T22:36:06.931 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 has been restored 2026-02-22T22:36:08.080 INFO:tasks.ceph.osd.2.trial023.stderr:2026-02-22T22:36:08.078+0000 7fc9bdb67640 -1 osd.2 19 heartbeat_check: no reply from 10.20.193.23:6822 osd.0 since back 2026-02-22T22:35:46.505602+0000 front 2026-02-22T22:35:46.505550+0000 (oldest deadline 2026-02-22T22:36:07.605442+0000) 2026-02-22T22:36:09.053 INFO:tasks.ceph.osd.2.trial023.stderr:2026-02-22T22:36:09.050+0000 7fc9bdb67640 -1 osd.2 19 heartbeat_check: no reply from 10.20.193.23:6822 osd.0 since back 2026-02-22T22:35:46.505602+0000 front 2026-02-22T22:35:46.505550+0000 (oldest deadline 2026-02-22T22:36:07.605442+0000) 2026-02-22T22:36:10.073 INFO:tasks.ceph.osd.2.trial023.stderr:2026-02-22T22:36:10.074+0000 7fc9bdb67640 -1 osd.2 19 heartbeat_check: no reply from 10.20.193.23:6822 osd.0 since back 2026-02-22T22:35:46.505602+0000 front 2026-02-22T22:35:46.505550+0000 (oldest deadline 2026-02-22T22:36:07.605442+0000) 2026-02-22T22:36:11.026 INFO:tasks.ceph.osd.2.trial023.stderr:2026-02-22T22:36:11.026+0000 7fc9bdb67640 -1 osd.2 19 heartbeat_check: no reply from 10.20.193.23:6822 osd.0 since back 2026-02-22T22:35:46.505602+0000 front 2026-02-22T22:35:46.505550+0000 (oldest deadline 2026-02-22T22:36:07.605442+0000) 2026-02-22T22:36:11.053 INFO:teuthology.orchestra.run.trial023.stdout:osd.1 : 0 osdmaps trimmed, 0 osdmaps added. 2026-02-22T22:36:11.570 INFO:tasks.rebuild_mondb:pull dir /home/ubuntu/cephtest/mon-store -> /tmp/tmp5yy42z5p 2026-02-22T22:36:11.571 DEBUG:teuthology.misc:Transferring archived files from trial023:/home/ubuntu/cephtest/mon-store to /tmp/tmpuqzi__m_ 2026-02-22T22:36:11.571 DEBUG:teuthology.orchestra.run.trial023:> sudo tar c -f - -C /home/ubuntu/cephtest/mon-store -- . 2026-02-22T22:36:11.625 INFO:tasks.rebuild_mondb:rm -rf osd:/home/ubuntu/cephtest/mon-store 2026-02-22T22:36:11.625 DEBUG:teuthology.orchestra.run.trial023:> sudo rm -fr /home/ubuntu/cephtest/mon-store 2026-02-22T22:36:11.673 INFO:tasks.rebuild_mondb:collecting maps from ceph:osd.2 2026-02-22T22:36:11.673 DEBUG:teuthology.orchestra.run.trial023:> sudo mkdir -m o+x -p /home/ubuntu/cephtest/mon-store 2026-02-22T22:36:11.725 DEBUG:teuthology.misc::sh: tar cz -f /tmp/rebuild_mondb-_l9mb6jj.tgz -C /tmp/tmpuqzi__m_ -- . 2026-02-22T22:36:11.779 DEBUG:teuthology.orchestra.run.trial023:> sudo tar xz -C /home/ubuntu/cephtest/mon-store -f /tmp/rebuild_mondb-shgjofby.tgz 2026-02-22T22:36:11.790 DEBUG:teuthology.orchestra.run.trial023:> sudo rm -fr /tmp/rebuild_mondb-shgjofby.tgz 2026-02-22T22:36:11.841 INFO:tasks.rebuild_mondb:rm -rf /tmp/tmpuqzi__m_ 2026-02-22T22:36:11.841 INFO:tasks.rebuild_mondb:cot /home/ubuntu/cephtest/mon-store 2026-02-22T22:36:11.842 DEBUG:tasks.ceph.osd.2:waiting for process to exit 2026-02-22T22:36:11.842 INFO:teuthology.orchestra.run:waiting for 300 2026-02-22T22:36:12.078 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~0s 2026-02-22T22:36:17.179 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~5s 2026-02-22T22:36:17.944 INFO:tasks.ceph.osd.2:Stopped 2026-02-22T22:36:17.944 INFO:tasks.rebuild_mondb.ceph_manager:sudo adjust-ulimits ceph-objectstore-tool --data-path /var/lib/ceph/osd/ceph-2 --journal-path /var/lib/ceph/osd/ceph-2/journal --no-mon-config --op update-mon-db --mon-store-path /home/ubuntu/cephtest/mon-store 2026-02-22T22:36:17.944 DEBUG:teuthology.orchestra.run.trial023:> bash -e -x -c 'sudo adjust-ulimits ceph-objectstore-tool --data-path /var/lib/ceph/osd/ceph-2 --journal-path /var/lib/ceph/osd/ceph-2/journal --no-mon-config --op update-mon-db --mon-store-path /home/ubuntu/cephtest/mon-store ' 2026-02-22T22:36:17.949 INFO:teuthology.orchestra.run.trial023.stderr:+ '[' -z '' ']' 2026-02-22T22:36:17.949 INFO:teuthology.orchestra.run.trial023.stderr:+ return 2026-02-22T22:36:17.949 INFO:teuthology.orchestra.run.trial023.stderr:+ case $- in 2026-02-22T22:36:17.949 INFO:teuthology.orchestra.run.trial023.stderr:+ return 2026-02-22T22:36:17.950 INFO:teuthology.orchestra.run.trial023.stderr:+ sudo adjust-ulimits ceph-objectstore-tool --data-path /var/lib/ceph/osd/ceph-2 --journal-path /var/lib/ceph/osd/ceph-2/journal --no-mon-config --op update-mon-db --mon-store-path /home/ubuntu/cephtest/mon-store 2026-02-22T22:36:21.763 INFO:teuthology.orchestra.run.trial023.stdout:osd.2 : 0 osdmaps trimmed, 0 osdmaps added. 2026-02-22T22:36:22.281 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 has been restored 2026-02-22T22:36:22.290 INFO:tasks.rebuild_mondb:pull dir /home/ubuntu/cephtest/mon-store -> /tmp/tmpuqzi__m_ 2026-02-22T22:36:22.290 DEBUG:teuthology.misc:Transferring archived files from trial023:/home/ubuntu/cephtest/mon-store to /tmp/tmpadwpsq1j 2026-02-22T22:36:22.291 DEBUG:teuthology.orchestra.run.trial023:> sudo tar c -f - -C /home/ubuntu/cephtest/mon-store -- . 2026-02-22T22:36:22.345 INFO:tasks.rebuild_mondb:rm -rf osd:/home/ubuntu/cephtest/mon-store 2026-02-22T22:36:22.346 DEBUG:teuthology.orchestra.run.trial023:> sudo rm -fr /home/ubuntu/cephtest/mon-store 2026-02-22T22:36:22.393 DEBUG:teuthology.misc::sh: tar cz -f /tmp/rebuild_mondb-ysd1fx7w.tgz -C /tmp/tmpadwpsq1j -- . 2026-02-22T22:36:22.443 DEBUG:teuthology.orchestra.run.trial023:> sudo tar xz -C /var/lib/ceph/mon/ceph-a -f /tmp/rebuild_mondb-qcfuekty.tgz 2026-02-22T22:36:22.454 DEBUG:teuthology.orchestra.run.trial023:> sudo rm -fr /tmp/rebuild_mondb-qcfuekty.tgz 2026-02-22T22:36:22.504 DEBUG:teuthology.orchestra.run.trial023:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-a 2026-02-22T22:36:22.553 DEBUG:teuthology.orchestra.run.trial023:> sudo ceph-authtool /etc/ceph/ceph.keyring -n mon. --cap mon 'allow *' 2026-02-22T22:36:22.612 DEBUG:teuthology.orchestra.run.trial023:> sudo ceph-authtool /etc/ceph/ceph.keyring -n client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' 2026-02-22T22:36:22.674 DEBUG:teuthology.orchestra.run.trial023:> sudo -u ceph CEPH_ARGS=--no-mon-config ceph-monstore-tool /var/lib/ceph/mon/ceph-a rebuild -- --keyring /etc/ceph/ceph.keyring --monmap /tmp/monmap 2026-02-22T22:36:22.738 INFO:teuthology.orchestra.run.trial023.stderr:2026-02-22T22:36:22.734+0000 7f2cb1f3f9c0 4 rocksdb: RocksDB version: 7.9.2 2026-02-22T22:36:22.739 INFO:teuthology.orchestra.run.trial023.stderr: 2026-02-22T22:36:22.739 INFO:teuthology.orchestra.run.trial023.stderr:2026-02-22T22:36:22.734+0000 7f2cb1f3f9c0 4 rocksdb: Git sha 0 2026-02-22T22:36:22.739 INFO:teuthology.orchestra.run.trial023.stderr:2026-02-22T22:36:22.734+0000 7f2cb1f3f9c0 4 rocksdb: Compile date 2026-02-22 16:49:20 2026-02-22T22:36:22.739 INFO:teuthology.orchestra.run.trial023.stderr:2026-02-22T22:36:22.734+0000 7f2cb1f3f9c0 4 rocksdb: DB SUMMARY 2026-02-22T22:36:22.739 INFO:teuthology.orchestra.run.trial023.stderr: 2026-02-22T22:36:22.739 INFO:teuthology.orchestra.run.trial023.stderr:2026-02-22T22:36:22.734+0000 7f2cb1f3f9c0 4 rocksdb: DB Session ID: VEJW5RMBCUXBDZ4QAHDS 2026-02-22T22:36:22.739 INFO:teuthology.orchestra.run.trial023.stderr: 2026-02-22T22:36:22.740 INFO:teuthology.orchestra.run.trial023.stderr:2026-02-22T22:36:22.734+0000 7f2cb1f3f9c0 4 rocksdb: CURRENT file: CURRENT 2026-02-22T22:36:22.740 INFO:teuthology.orchestra.run.trial023.stderr: 2026-02-22T22:36:22.740 INFO:teuthology.orchestra.run.trial023.stderr:2026-02-22T22:36:22.734+0000 7f2cb1f3f9c0 4 rocksdb: IDENTITY file: IDENTITY 2026-02-22T22:36:22.740 INFO:teuthology.orchestra.run.trial023.stderr: 2026-02-22T22:36:22.740 INFO:teuthology.orchestra.run.trial023.stderr:2026-02-22T22:36:22.734+0000 7f2cb1f3f9c0 4 rocksdb: MANIFEST file: MANIFEST-000015 size: 287 Bytes 2026-02-22T22:36:22.740 INFO:teuthology.orchestra.run.trial023.stderr: 2026-02-22T22:36:22.740 INFO:teuthology.orchestra.run.trial023.stderr:2026-02-22T22:36:22.734+0000 7f2cb1f3f9c0 4 rocksdb: SST files in /var/lib/ceph/mon/ceph-a/store.db dir, Total Num: 2, files: 000008.sst 000013.sst 2026-02-22T22:36:22.741 INFO:teuthology.orchestra.run.trial023.stderr: 2026-02-22T22:36:22.741 INFO:teuthology.orchestra.run.trial023.stderr:2026-02-22T22:36:22.734+0000 7f2cb1f3f9c0 4 rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-a/store.db: 000014.log size: 243 ; 2026-02-22T22:36:22.741 INFO:teuthology.orchestra.run.trial023.stderr: 2026-02-22T22:36:22.741 INFO:teuthology.orchestra.run.trial023.stderr:2026-02-22T22:36:22.734+0000 7f2cb1f3f9c0 4 rocksdb: Options.error_if_exists: 0 2026-02-22T22:36:22.741 INFO:teuthology.orchestra.run.trial023.stderr:2026-02-22T22:36:22.734+0000 7f2cb1f3f9c0 4 rocksdb: Options.create_if_missing: 0 2026-02-22T22:36:22.741 INFO:teuthology.orchestra.run.trial023.stderr:2026-02-22T22:36:22.734+0000 7f2cb1f3f9c0 4 rocksdb: Options.paranoid_checks: 1 2026-02-22T22:36:22.741 INFO:teuthology.orchestra.run.trial023.stderr:2026-02-22T22:36:22.734+0000 7f2cb1f3f9c0 4 rocksdb: Options.flush_verify_memtable_count: 1 2026-02-22T22:36:22.741 INFO:teuthology.orchestra.run.trial023.stderr:2026-02-22T22:36:22.734+0000 7f2cb1f3f9c0 4 rocksdb: Options.track_and_verify_wals_in_manifest: 0 2026-02-22T22:36:22.741 INFO:teuthology.orchestra.run.trial023.stderr:2026-02-22T22:36:22.734+0000 7f2cb1f3f9c0 4 rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2026-02-22T22:36:22.741 INFO:teuthology.orchestra.run.trial023.stderr:2026-02-22T22:36:22.734+0000 7f2cb1f3f9c0 4 rocksdb: Options.env: 0x56060c930520 2026-02-22T22:36:22.742 INFO:teuthology.orchestra.run.trial023.stderr:2026-02-22T22:36:22.734+0000 7f2cb1f3f9c0 4 rocksdb: Options.fs: PosixFileSystem 2026-02-22T22:36:22.742 INFO:teuthology.orchestra.run.trial023.stderr:2026-02-22T22:36:22.734+0000 7f2cb1f3f9c0 4 rocksdb: Options.info_log: 0x56064b56c000 2026-02-22T22:36:22.742 INFO:teuthology.orchestra.run.trial023.stderr:2026-02-22T22:36:22.734+0000 7f2cb1f3f9c0 4 rocksdb: Options.max_file_opening_threads: 16 2026-02-22T22:36:22.742 INFO:teuthology.orchestra.run.trial023.stderr:2026-02-22T22:36:22.734+0000 7f2cb1f3f9c0 4 rocksdb: Options.statistics: (nil) 2026-02-22T22:36:22.742 INFO:teuthology.orchestra.run.trial023.stderr:2026-02-22T22:36:22.734+0000 7f2cb1f3f9c0 4 rocksdb: Options.use_fsync: 0 2026-02-22T22:36:22.742 INFO:teuthology.orchestra.run.trial023.stderr:2026-02-22T22:36:22.734+0000 7f2cb1f3f9c0 4 rocksdb: Options.max_log_file_size: 0 2026-02-22T22:36:22.742 INFO:teuthology.orchestra.run.trial023.stderr:2026-02-22T22:36:22.734+0000 7f2cb1f3f9c0 4 rocksdb: Options.max_manifest_file_size: 1073741824 2026-02-22T22:36:22.742 INFO:teuthology.orchestra.run.trial023.stderr:2026-02-22T22:36:22.734+0000 7f2cb1f3f9c0 4 rocksdb: Options.log_file_time_to_roll: 0 2026-02-22T22:36:22.743 INFO:teuthology.orchestra.run.trial023.stderr:2026-02-22T22:36:22.734+0000 7f2cb1f3f9c0 4 rocksdb: Options.keep_log_file_num: 1000 2026-02-22T22:36:22.743 INFO:teuthology.orchestra.run.trial023.stderr:2026-02-22T22:36:22.734+0000 7f2cb1f3f9c0 4 rocksdb: Options.recycle_log_file_num: 0 2026-02-22T22:36:22.743 INFO:teuthology.orchestra.run.trial023.stderr:2026-02-22T22:36:22.734+0000 7f2cb1f3f9c0 4 rocksdb: Options.allow_fallocate: 1 2026-02-22T22:36:22.743 INFO:teuthology.orchestra.run.trial023.stderr:2026-02-22T22:36:22.734+0000 7f2cb1f3f9c0 4 rocksdb: Options.allow_mmap_reads: 0 2026-02-22T22:36:22.743 INFO:teuthology.orchestra.run.trial023.stderr:2026-02-22T22:36:22.734+0000 7f2cb1f3f9c0 4 rocksdb: Options.allow_mmap_writes: 0 2026-02-22T22:36:22.743 INFO:teuthology.orchestra.run.trial023.stderr:2026-02-22T22:36:22.734+0000 7f2cb1f3f9c0 4 rocksdb: Options.use_direct_reads: 0 2026-02-22T22:36:22.743 INFO:teuthology.orchestra.run.trial023.stderr:2026-02-22T22:36:22.734+0000 7f2cb1f3f9c0 4 rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2026-02-22T22:36:22.743 INFO:teuthology.orchestra.run.trial023.stderr:2026-02-22T22:36:22.734+0000 7f2cb1f3f9c0 4 rocksdb: Options.create_missing_column_families: 0 2026-02-22T22:36:22.743 INFO:teuthology.orchestra.run.trial023.stderr:2026-02-22T22:36:22.734+0000 7f2cb1f3f9c0 4 rocksdb: Options.db_log_dir: 2026-02-22T22:36:22.743 INFO:teuthology.orchestra.run.trial023.stderr:2026-02-22T22:36:22.734+0000 7f2cb1f3f9c0 4 rocksdb: Options.wal_dir: 2026-02-22T22:36:22.743 INFO:teuthology.orchestra.run.trial023.stderr:2026-02-22T22:36:22.734+0000 7f2cb1f3f9c0 4 rocksdb: Options.table_cache_numshardbits: 6 2026-02-22T22:36:22.743 INFO:teuthology.orchestra.run.trial023.stderr:2026-02-22T22:36:22.734+0000 7f2cb1f3f9c0 4 rocksdb: Options.WAL_ttl_seconds: 0 2026-02-22T22:36:22.743 INFO:teuthology.orchestra.run.trial023.stderr:2026-02-22T22:36:22.734+0000 7f2cb1f3f9c0 4 rocksdb: Options.WAL_size_limit_MB: 0 2026-02-22T22:36:22.744 INFO:teuthology.orchestra.run.trial023.stderr:2026-02-22T22:36:22.734+0000 7f2cb1f3f9c0 4 rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2026-02-22T22:36:22.744 INFO:teuthology.orchestra.run.trial023.stderr:2026-02-22T22:36:22.734+0000 7f2cb1f3f9c0 4 rocksdb: Options.manifest_preallocation_size: 4194304 2026-02-22T22:36:22.744 INFO:teuthology.orchestra.run.trial023.stderr:2026-02-22T22:36:22.734+0000 7f2cb1f3f9c0 4 rocksdb: Options.is_fd_close_on_exec: 1 2026-02-22T22:36:22.744 INFO:teuthology.orchestra.run.trial023.stderr:2026-02-22T22:36:22.734+0000 7f2cb1f3f9c0 4 rocksdb: Options.advise_random_on_open: 1 2026-02-22T22:36:22.744 INFO:teuthology.orchestra.run.trial023.stderr:2026-02-22T22:36:22.734+0000 7f2cb1f3f9c0 4 rocksdb: Options.db_write_buffer_size: 0 2026-02-22T22:36:22.744 INFO:teuthology.orchestra.run.trial023.stderr:2026-02-22T22:36:22.734+0000 7f2cb1f3f9c0 4 rocksdb: Options.write_buffer_manager: 0x56064b483040 2026-02-22T22:36:22.744 INFO:teuthology.orchestra.run.trial023.stderr:2026-02-22T22:36:22.734+0000 7f2cb1f3f9c0 4 rocksdb: Options.access_hint_on_compaction_start: 1 2026-02-22T22:36:22.744 INFO:teuthology.orchestra.run.trial023.stderr:2026-02-22T22:36:22.734+0000 7f2cb1f3f9c0 4 rocksdb: Options.random_access_max_buffer_size: 1048576 2026-02-22T22:36:22.744 INFO:teuthology.orchestra.run.trial023.stderr:2026-02-22T22:36:22.734+0000 7f2cb1f3f9c0 4 rocksdb: Options.use_adaptive_mutex: 0 2026-02-22T22:36:22.744 INFO:teuthology.orchestra.run.trial023.stderr:2026-02-22T22:36:22.734+0000 7f2cb1f3f9c0 4 rocksdb: Options.rate_limiter: (nil) 2026-02-22T22:36:22.744 INFO:teuthology.orchestra.run.trial023.stderr:2026-02-22T22:36:22.734+0000 7f2cb1f3f9c0 4 rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2026-02-22T22:36:22.744 INFO:teuthology.orchestra.run.trial023.stderr:2026-02-22T22:36:22.734+0000 7f2cb1f3f9c0 4 rocksdb: Options.wal_recovery_mode: 2 2026-02-22T22:36:22.744 INFO:teuthology.orchestra.run.trial023.stderr:2026-02-22T22:36:22.734+0000 7f2cb1f3f9c0 4 rocksdb: Options.enable_thread_tracking: 0 2026-02-22T22:36:22.745 INFO:teuthology.orchestra.run.trial023.stderr:2026-02-22T22:36:22.734+0000 7f2cb1f3f9c0 4 rocksdb: Options.enable_pipelined_write: 0 2026-02-22T22:36:22.745 INFO:teuthology.orchestra.run.trial023.stderr:2026-02-22T22:36:22.734+0000 7f2cb1f3f9c0 4 rocksdb: Options.unordered_write: 0 2026-02-22T22:36:22.745 INFO:teuthology.orchestra.run.trial023.stderr:2026-02-22T22:36:22.734+0000 7f2cb1f3f9c0 4 rocksdb: Options.allow_concurrent_memtable_write: 1 2026-02-22T22:36:22.745 INFO:teuthology.orchestra.run.trial023.stderr:2026-02-22T22:36:22.734+0000 7f2cb1f3f9c0 4 rocksdb: Options.enable_write_thread_adaptive_yield: 1 2026-02-22T22:36:22.745 INFO:teuthology.orchestra.run.trial023.stderr:2026-02-22T22:36:22.734+0000 7f2cb1f3f9c0 4 rocksdb: Options.write_thread_max_yield_usec: 100 2026-02-22T22:36:22.745 INFO:teuthology.orchestra.run.trial023.stderr:2026-02-22T22:36:22.734+0000 7f2cb1f3f9c0 4 rocksdb: Options.write_thread_slow_yield_usec: 3 2026-02-22T22:36:22.745 INFO:teuthology.orchestra.run.trial023.stderr:2026-02-22T22:36:22.734+0000 7f2cb1f3f9c0 4 rocksdb: Options.row_cache: None 2026-02-22T22:36:22.745 INFO:teuthology.orchestra.run.trial023.stderr:2026-02-22T22:36:22.734+0000 7f2cb1f3f9c0 4 rocksdb: Options.wal_filter: None 2026-02-22T22:36:22.745 INFO:teuthology.orchestra.run.trial023.stderr:2026-02-22T22:36:22.734+0000 7f2cb1f3f9c0 4 rocksdb: Options.avoid_flush_during_recovery: 0 2026-02-22T22:36:22.745 INFO:teuthology.orchestra.run.trial023.stderr:2026-02-22T22:36:22.734+0000 7f2cb1f3f9c0 4 rocksdb: Options.allow_ingest_behind: 0 2026-02-22T22:36:22.745 INFO:teuthology.orchestra.run.trial023.stderr:2026-02-22T22:36:22.734+0000 7f2cb1f3f9c0 4 rocksdb: Options.two_write_queues: 0 2026-02-22T22:36:22.745 INFO:teuthology.orchestra.run.trial023.stderr:2026-02-22T22:36:22.734+0000 7f2cb1f3f9c0 4 rocksdb: Options.manual_wal_flush: 0 2026-02-22T22:36:22.745 INFO:teuthology.orchestra.run.trial023.stderr:2026-02-22T22:36:22.734+0000 7f2cb1f3f9c0 4 rocksdb: Options.wal_compression: 0 2026-02-22T22:36:22.746 INFO:teuthology.orchestra.run.trial023.stderr:2026-02-22T22:36:22.734+0000 7f2cb1f3f9c0 4 rocksdb: Options.atomic_flush: 0 2026-02-22T22:36:22.746 INFO:teuthology.orchestra.run.trial023.stderr:2026-02-22T22:36:22.734+0000 7f2cb1f3f9c0 4 rocksdb: Options.avoid_unnecessary_blocking_io: 0 2026-02-22T22:36:22.746 INFO:teuthology.orchestra.run.trial023.stderr:2026-02-22T22:36:22.734+0000 7f2cb1f3f9c0 4 rocksdb: Options.persist_stats_to_disk: 0 2026-02-22T22:36:22.746 INFO:teuthology.orchestra.run.trial023.stderr:2026-02-22T22:36:22.734+0000 7f2cb1f3f9c0 4 rocksdb: Options.write_dbid_to_manifest: 0 2026-02-22T22:36:22.746 INFO:teuthology.orchestra.run.trial023.stderr:2026-02-22T22:36:22.734+0000 7f2cb1f3f9c0 4 rocksdb: Options.log_readahead_size: 0 2026-02-22T22:36:22.746 INFO:teuthology.orchestra.run.trial023.stderr:2026-02-22T22:36:22.734+0000 7f2cb1f3f9c0 4 rocksdb: Options.file_checksum_gen_factory: Unknown 2026-02-22T22:36:22.746 INFO:teuthology.orchestra.run.trial023.stderr:2026-02-22T22:36:22.734+0000 7f2cb1f3f9c0 4 rocksdb: Options.best_efforts_recovery: 0 2026-02-22T22:36:22.746 INFO:teuthology.orchestra.run.trial023.stderr:2026-02-22T22:36:22.734+0000 7f2cb1f3f9c0 4 rocksdb: Options.max_bgerror_resume_count: 2147483647 2026-02-22T22:36:22.746 INFO:teuthology.orchestra.run.trial023.stderr:2026-02-22T22:36:22.734+0000 7f2cb1f3f9c0 4 rocksdb: Options.bgerror_resume_retry_interval: 1000000 2026-02-22T22:36:22.746 INFO:teuthology.orchestra.run.trial023.stderr:2026-02-22T22:36:22.734+0000 7f2cb1f3f9c0 4 rocksdb: Options.allow_data_in_errors: 0 2026-02-22T22:36:22.746 INFO:teuthology.orchestra.run.trial023.stderr:2026-02-22T22:36:22.734+0000 7f2cb1f3f9c0 4 rocksdb: Options.db_host_id: __hostname__ 2026-02-22T22:36:22.746 INFO:teuthology.orchestra.run.trial023.stderr:2026-02-22T22:36:22.734+0000 7f2cb1f3f9c0 4 rocksdb: Options.enforce_single_del_contracts: true 2026-02-22T22:36:22.746 INFO:teuthology.orchestra.run.trial023.stderr:2026-02-22T22:36:22.734+0000 7f2cb1f3f9c0 4 rocksdb: Options.max_background_jobs: 2 2026-02-22T22:36:22.746 INFO:teuthology.orchestra.run.trial023.stderr:2026-02-22T22:36:22.734+0000 7f2cb1f3f9c0 4 rocksdb: Options.max_background_compactions: -1 2026-02-22T22:36:22.747 INFO:teuthology.orchestra.run.trial023.stderr:2026-02-22T22:36:22.734+0000 7f2cb1f3f9c0 4 rocksdb: Options.max_subcompactions: 1 2026-02-22T22:36:22.747 INFO:teuthology.orchestra.run.trial023.stderr:2026-02-22T22:36:22.734+0000 7f2cb1f3f9c0 4 rocksdb: Options.avoid_flush_during_shutdown: 0 2026-02-22T22:36:22.747 INFO:teuthology.orchestra.run.trial023.stderr:2026-02-22T22:36:22.734+0000 7f2cb1f3f9c0 4 rocksdb: Options.writable_file_max_buffer_size: 1048576 2026-02-22T22:36:22.747 INFO:teuthology.orchestra.run.trial023.stderr:2026-02-22T22:36:22.734+0000 7f2cb1f3f9c0 4 rocksdb: Options.delayed_write_rate : 16777216 2026-02-22T22:36:22.747 INFO:teuthology.orchestra.run.trial023.stderr:2026-02-22T22:36:22.734+0000 7f2cb1f3f9c0 4 rocksdb: Options.max_total_wal_size: 0 2026-02-22T22:36:22.747 INFO:teuthology.orchestra.run.trial023.stderr:2026-02-22T22:36:22.734+0000 7f2cb1f3f9c0 4 rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2026-02-22T22:36:22.747 INFO:teuthology.orchestra.run.trial023.stderr:2026-02-22T22:36:22.734+0000 7f2cb1f3f9c0 4 rocksdb: Options.stats_dump_period_sec: 600 2026-02-22T22:36:22.747 INFO:teuthology.orchestra.run.trial023.stderr:2026-02-22T22:36:22.734+0000 7f2cb1f3f9c0 4 rocksdb: Options.stats_persist_period_sec: 600 2026-02-22T22:36:22.747 INFO:teuthology.orchestra.run.trial023.stderr:2026-02-22T22:36:22.734+0000 7f2cb1f3f9c0 4 rocksdb: Options.stats_history_buffer_size: 1048576 2026-02-22T22:36:22.747 INFO:teuthology.orchestra.run.trial023.stderr:2026-02-22T22:36:22.734+0000 7f2cb1f3f9c0 4 rocksdb: Options.max_open_files: -1 2026-02-22T22:36:22.747 INFO:teuthology.orchestra.run.trial023.stderr:2026-02-22T22:36:22.734+0000 7f2cb1f3f9c0 4 rocksdb: Options.bytes_per_sync: 0 2026-02-22T22:36:22.747 INFO:teuthology.orchestra.run.trial023.stderr:2026-02-22T22:36:22.734+0000 7f2cb1f3f9c0 4 rocksdb: Options.wal_bytes_per_sync: 0 2026-02-22T22:36:22.748 INFO:teuthology.orchestra.run.trial023.stderr:2026-02-22T22:36:22.734+0000 7f2cb1f3f9c0 4 rocksdb: Options.strict_bytes_per_sync: 0 2026-02-22T22:36:22.748 INFO:teuthology.orchestra.run.trial023.stderr:2026-02-22T22:36:22.734+0000 7f2cb1f3f9c0 4 rocksdb: Options.compaction_readahead_size: 0 2026-02-22T22:36:22.748 INFO:teuthology.orchestra.run.trial023.stderr:2026-02-22T22:36:22.734+0000 7f2cb1f3f9c0 4 rocksdb: Options.max_background_flushes: -1 2026-02-22T22:36:22.748 INFO:teuthology.orchestra.run.trial023.stderr:2026-02-22T22:36:22.734+0000 7f2cb1f3f9c0 4 rocksdb: Compression algorithms supported: 2026-02-22T22:36:22.748 INFO:teuthology.orchestra.run.trial023.stderr:2026-02-22T22:36:22.734+0000 7f2cb1f3f9c0 4 rocksdb: kZSTD supported: 0 2026-02-22T22:36:22.748 INFO:teuthology.orchestra.run.trial023.stderr:2026-02-22T22:36:22.734+0000 7f2cb1f3f9c0 4 rocksdb: kXpressCompression supported: 0 2026-02-22T22:36:22.748 INFO:teuthology.orchestra.run.trial023.stderr:2026-02-22T22:36:22.734+0000 7f2cb1f3f9c0 4 rocksdb: kBZip2Compression supported: 0 2026-02-22T22:36:22.748 INFO:teuthology.orchestra.run.trial023.stderr:2026-02-22T22:36:22.734+0000 7f2cb1f3f9c0 4 rocksdb: kZSTDNotFinalCompression supported: 0 2026-02-22T22:36:22.748 INFO:teuthology.orchestra.run.trial023.stderr:2026-02-22T22:36:22.734+0000 7f2cb1f3f9c0 4 rocksdb: kLZ4Compression supported: 1 2026-02-22T22:36:22.748 INFO:teuthology.orchestra.run.trial023.stderr:2026-02-22T22:36:22.734+0000 7f2cb1f3f9c0 4 rocksdb: kZlibCompression supported: 1 2026-02-22T22:36:22.748 INFO:teuthology.orchestra.run.trial023.stderr:2026-02-22T22:36:22.734+0000 7f2cb1f3f9c0 4 rocksdb: kLZ4HCCompression supported: 1 2026-02-22T22:36:22.748 INFO:teuthology.orchestra.run.trial023.stderr:2026-02-22T22:36:22.734+0000 7f2cb1f3f9c0 4 rocksdb: kSnappyCompression supported: 1 2026-02-22T22:36:22.748 INFO:teuthology.orchestra.run.trial023.stderr:2026-02-22T22:36:22.734+0000 7f2cb1f3f9c0 4 rocksdb: Fast CRC32 supported: Supported on x86 2026-02-22T22:36:22.749 INFO:teuthology.orchestra.run.trial023.stderr:2026-02-22T22:36:22.734+0000 7f2cb1f3f9c0 4 rocksdb: DMutex implementation: pthread_mutex_t 2026-02-22T22:36:22.749 INFO:teuthology.orchestra.run.trial023.stderr:2026-02-22T22:36:22.734+0000 7f2cb1f3f9c0 4 rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000015 2026-02-22T22:36:22.749 INFO:teuthology.orchestra.run.trial023.stderr: 2026-02-22T22:36:22.749 INFO:teuthology.orchestra.run.trial023.stderr:2026-02-22T22:36:22.734+0000 7f2cb1f3f9c0 4 rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2026-02-22T22:36:22.749 INFO:teuthology.orchestra.run.trial023.stderr: 2026-02-22T22:36:22.749 INFO:teuthology.orchestra.run.trial023.stderr:2026-02-22T22:36:22.734+0000 7f2cb1f3f9c0 4 rocksdb: Options.comparator: leveldb.BytewiseComparator 2026-02-22T22:36:22.749 INFO:teuthology.orchestra.run.trial023.stderr:2026-02-22T22:36:22.734+0000 7f2cb1f3f9c0 4 rocksdb: Options.merge_operator: 2026-02-22T22:36:22.749 INFO:teuthology.orchestra.run.trial023.stderr:2026-02-22T22:36:22.734+0000 7f2cb1f3f9c0 4 rocksdb: Options.compaction_filter: None 2026-02-22T22:36:22.749 INFO:teuthology.orchestra.run.trial023.stderr:2026-02-22T22:36:22.734+0000 7f2cb1f3f9c0 4 rocksdb: Options.compaction_filter_factory: None 2026-02-22T22:36:22.749 INFO:teuthology.orchestra.run.trial023.stderr:2026-02-22T22:36:22.734+0000 7f2cb1f3f9c0 4 rocksdb: Options.sst_partitioner_factory: None 2026-02-22T22:36:22.749 INFO:teuthology.orchestra.run.trial023.stderr:2026-02-22T22:36:22.734+0000 7f2cb1f3f9c0 4 rocksdb: Options.memtable_factory: SkipListFactory 2026-02-22T22:36:22.749 INFO:teuthology.orchestra.run.trial023.stderr:2026-02-22T22:36:22.734+0000 7f2cb1f3f9c0 4 rocksdb: Options.table_factory: BlockBasedTable 2026-02-22T22:36:22.750 INFO:teuthology.orchestra.run.trial023.stderr:2026-02-22T22:36:22.734+0000 7f2cb1f3f9c0 4 rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x56064b56c260) 2026-02-22T22:36:22.750 INFO:teuthology.orchestra.run.trial023.stderr: cache_index_and_filter_blocks: 1 2026-02-22T22:36:22.750 INFO:teuthology.orchestra.run.trial023.stderr: cache_index_and_filter_blocks_with_high_priority: 0 2026-02-22T22:36:22.750 INFO:teuthology.orchestra.run.trial023.stderr: pin_l0_filter_and_index_blocks_in_cache: 0 2026-02-22T22:36:22.750 INFO:teuthology.orchestra.run.trial023.stderr: pin_top_level_index_and_filter: 1 2026-02-22T22:36:22.750 INFO:teuthology.orchestra.run.trial023.stderr: index_type: 0 2026-02-22T22:36:22.750 INFO:teuthology.orchestra.run.trial023.stderr: data_block_index_type: 0 2026-02-22T22:36:22.750 INFO:teuthology.orchestra.run.trial023.stderr: index_shortening: 1 2026-02-22T22:36:22.750 INFO:teuthology.orchestra.run.trial023.stderr: data_block_hash_table_util_ratio: 0.750000 2026-02-22T22:36:22.750 INFO:teuthology.orchestra.run.trial023.stderr: checksum: 4 2026-02-22T22:36:22.750 INFO:teuthology.orchestra.run.trial023.stderr: no_block_cache: 0 2026-02-22T22:36:22.750 INFO:teuthology.orchestra.run.trial023.stderr: block_cache: 0x56064b4a5770 2026-02-22T22:36:22.751 INFO:teuthology.orchestra.run.trial023.stderr: block_cache_name: BinnedLRUCache 2026-02-22T22:36:22.751 INFO:teuthology.orchestra.run.trial023.stderr: block_cache_options: 2026-02-22T22:36:22.751 INFO:teuthology.orchestra.run.trial023.stderr: capacity : 536870912 2026-02-22T22:36:22.751 INFO:teuthology.orchestra.run.trial023.stderr: num_shard_bits : 4 2026-02-22T22:36:22.751 INFO:teuthology.orchestra.run.trial023.stderr: strict_capacity_limit : 0 2026-02-22T22:36:22.751 INFO:teuthology.orchestra.run.trial023.stderr: high_pri_pool_ratio: 0.000 2026-02-22T22:36:22.751 INFO:teuthology.orchestra.run.trial023.stderr: block_cache_compressed: (nil) 2026-02-22T22:36:22.751 INFO:teuthology.orchestra.run.trial023.stderr: persistent_cache: (nil) 2026-02-22T22:36:22.751 INFO:teuthology.orchestra.run.trial023.stderr: block_size: 4096 2026-02-22T22:36:22.751 INFO:teuthology.orchestra.run.trial023.stderr: block_size_deviation: 10 2026-02-22T22:36:22.751 INFO:teuthology.orchestra.run.trial023.stderr: block_restart_interval: 16 2026-02-22T22:36:22.751 INFO:teuthology.orchestra.run.trial023.stderr: index_block_restart_interval: 1 2026-02-22T22:36:22.751 INFO:teuthology.orchestra.run.trial023.stderr: metadata_block_size: 4096 2026-02-22T22:36:22.752 INFO:teuthology.orchestra.run.trial023.stderr: partition_filters: 0 2026-02-22T22:36:22.752 INFO:teuthology.orchestra.run.trial023.stderr: use_delta_encoding: 1 2026-02-22T22:36:22.752 INFO:teuthology.orchestra.run.trial023.stderr: filter_policy: bloomfilter 2026-02-22T22:36:22.752 INFO:teuthology.orchestra.run.trial023.stderr: whole_key_filtering: 1 2026-02-22T22:36:22.752 INFO:teuthology.orchestra.run.trial023.stderr: verify_compression: 0 2026-02-22T22:36:22.752 INFO:teuthology.orchestra.run.trial023.stderr: read_amp_bytes_per_bit: 0 2026-02-22T22:36:22.752 INFO:teuthology.orchestra.run.trial023.stderr: format_version: 5 2026-02-22T22:36:22.752 INFO:teuthology.orchestra.run.trial023.stderr: enable_index_compression: 1 2026-02-22T22:36:22.752 INFO:teuthology.orchestra.run.trial023.stderr: block_align: 0 2026-02-22T22:36:22.752 INFO:teuthology.orchestra.run.trial023.stderr: max_auto_readahead_size: 262144 2026-02-22T22:36:22.752 INFO:teuthology.orchestra.run.trial023.stderr: prepopulate_block_cache: 0 2026-02-22T22:36:22.752 INFO:teuthology.orchestra.run.trial023.stderr: initial_auto_readahead_size: 8192 2026-02-22T22:36:22.753 INFO:teuthology.orchestra.run.trial023.stderr: num_file_reads_for_auto_readahead: 2 2026-02-22T22:36:22.753 INFO:teuthology.orchestra.run.trial023.stderr: 2026-02-22T22:36:22.753 INFO:teuthology.orchestra.run.trial023.stderr:2026-02-22T22:36:22.734+0000 7f2cb1f3f9c0 4 rocksdb: Options.write_buffer_size: 33554432 2026-02-22T22:36:22.753 INFO:teuthology.orchestra.run.trial023.stderr:2026-02-22T22:36:22.734+0000 7f2cb1f3f9c0 4 rocksdb: Options.max_write_buffer_number: 2 2026-02-22T22:36:22.753 INFO:teuthology.orchestra.run.trial023.stderr:2026-02-22T22:36:22.734+0000 7f2cb1f3f9c0 4 rocksdb: Options.compression: NoCompression 2026-02-22T22:36:22.753 INFO:teuthology.orchestra.run.trial023.stderr:2026-02-22T22:36:22.734+0000 7f2cb1f3f9c0 4 rocksdb: Options.bottommost_compression: Disabled 2026-02-22T22:36:22.753 INFO:teuthology.orchestra.run.trial023.stderr:2026-02-22T22:36:22.734+0000 7f2cb1f3f9c0 4 rocksdb: Options.prefix_extractor: nullptr 2026-02-22T22:36:22.753 INFO:teuthology.orchestra.run.trial023.stderr:2026-02-22T22:36:22.734+0000 7f2cb1f3f9c0 4 rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2026-02-22T22:36:22.753 INFO:teuthology.orchestra.run.trial023.stderr:2026-02-22T22:36:22.734+0000 7f2cb1f3f9c0 4 rocksdb: Options.num_levels: 7 2026-02-22T22:36:22.753 INFO:teuthology.orchestra.run.trial023.stderr:2026-02-22T22:36:22.734+0000 7f2cb1f3f9c0 4 rocksdb: Options.min_write_buffer_number_to_merge: 1 2026-02-22T22:36:22.753 INFO:teuthology.orchestra.run.trial023.stderr:2026-02-22T22:36:22.734+0000 7f2cb1f3f9c0 4 rocksdb: Options.max_write_buffer_number_to_maintain: 0 2026-02-22T22:36:22.753 INFO:teuthology.orchestra.run.trial023.stderr:2026-02-22T22:36:22.734+0000 7f2cb1f3f9c0 4 rocksdb: Options.max_write_buffer_size_to_maintain: 0 2026-02-22T22:36:22.753 INFO:teuthology.orchestra.run.trial023.stderr:2026-02-22T22:36:22.734+0000 7f2cb1f3f9c0 4 rocksdb: Options.bottommost_compression_opts.window_bits: -14 2026-02-22T22:36:22.754 INFO:teuthology.orchestra.run.trial023.stderr:2026-02-22T22:36:22.734+0000 7f2cb1f3f9c0 4 rocksdb: Options.bottommost_compression_opts.level: 32767 2026-02-22T22:36:22.754 INFO:teuthology.orchestra.run.trial023.stderr:2026-02-22T22:36:22.734+0000 7f2cb1f3f9c0 4 rocksdb: Options.bottommost_compression_opts.strategy: 0 2026-02-22T22:36:22.754 INFO:teuthology.orchestra.run.trial023.stderr:2026-02-22T22:36:22.734+0000 7f2cb1f3f9c0 4 rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2026-02-22T22:36:22.754 INFO:teuthology.orchestra.run.trial023.stderr:2026-02-22T22:36:22.734+0000 7f2cb1f3f9c0 4 rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2026-02-22T22:36:22.754 INFO:teuthology.orchestra.run.trial023.stderr:2026-02-22T22:36:22.734+0000 7f2cb1f3f9c0 4 rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2026-02-22T22:36:22.754 INFO:teuthology.orchestra.run.trial023.stderr:2026-02-22T22:36:22.734+0000 7f2cb1f3f9c0 4 rocksdb: Options.bottommost_compression_opts.enabled: false 2026-02-22T22:36:22.754 INFO:teuthology.orchestra.run.trial023.stderr:2026-02-22T22:36:22.734+0000 7f2cb1f3f9c0 4 rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2026-02-22T22:36:22.754 INFO:teuthology.orchestra.run.trial023.stderr:2026-02-22T22:36:22.734+0000 7f2cb1f3f9c0 4 rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2026-02-22T22:36:22.754 INFO:teuthology.orchestra.run.trial023.stderr:2026-02-22T22:36:22.734+0000 7f2cb1f3f9c0 4 rocksdb: Options.compression_opts.window_bits: -14 2026-02-22T22:36:22.754 INFO:teuthology.orchestra.run.trial023.stderr:2026-02-22T22:36:22.734+0000 7f2cb1f3f9c0 4 rocksdb: Options.compression_opts.level: 32767 2026-02-22T22:36:22.754 INFO:teuthology.orchestra.run.trial023.stderr:2026-02-22T22:36:22.734+0000 7f2cb1f3f9c0 4 rocksdb: Options.compression_opts.strategy: 0 2026-02-22T22:36:22.754 INFO:teuthology.orchestra.run.trial023.stderr:2026-02-22T22:36:22.734+0000 7f2cb1f3f9c0 4 rocksdb: Options.compression_opts.max_dict_bytes: 0 2026-02-22T22:36:22.754 INFO:teuthology.orchestra.run.trial023.stderr:2026-02-22T22:36:22.734+0000 7f2cb1f3f9c0 4 rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2026-02-22T22:36:22.755 INFO:teuthology.orchestra.run.trial023.stderr:2026-02-22T22:36:22.734+0000 7f2cb1f3f9c0 4 rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2026-02-22T22:36:22.755 INFO:teuthology.orchestra.run.trial023.stderr:2026-02-22T22:36:22.734+0000 7f2cb1f3f9c0 4 rocksdb: Options.compression_opts.parallel_threads: 1 2026-02-22T22:36:22.755 INFO:teuthology.orchestra.run.trial023.stderr:2026-02-22T22:36:22.734+0000 7f2cb1f3f9c0 4 rocksdb: Options.compression_opts.enabled: false 2026-02-22T22:36:22.755 INFO:teuthology.orchestra.run.trial023.stderr:2026-02-22T22:36:22.734+0000 7f2cb1f3f9c0 4 rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2026-02-22T22:36:22.755 INFO:teuthology.orchestra.run.trial023.stderr:2026-02-22T22:36:22.734+0000 7f2cb1f3f9c0 4 rocksdb: Options.level0_file_num_compaction_trigger: 4 2026-02-22T22:36:22.755 INFO:teuthology.orchestra.run.trial023.stderr:2026-02-22T22:36:22.734+0000 7f2cb1f3f9c0 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2026-02-22T22:36:22.755 INFO:teuthology.orchestra.run.trial023.stderr:2026-02-22T22:36:22.734+0000 7f2cb1f3f9c0 4 rocksdb: Options.level0_stop_writes_trigger: 36 2026-02-22T22:36:22.755 INFO:teuthology.orchestra.run.trial023.stderr:2026-02-22T22:36:22.734+0000 7f2cb1f3f9c0 4 rocksdb: Options.target_file_size_base: 67108864 2026-02-22T22:36:22.755 INFO:teuthology.orchestra.run.trial023.stderr:2026-02-22T22:36:22.734+0000 7f2cb1f3f9c0 4 rocksdb: Options.target_file_size_multiplier: 1 2026-02-22T22:36:22.755 INFO:teuthology.orchestra.run.trial023.stderr:2026-02-22T22:36:22.734+0000 7f2cb1f3f9c0 4 rocksdb: Options.max_bytes_for_level_base: 268435456 2026-02-22T22:36:22.755 INFO:teuthology.orchestra.run.trial023.stderr:2026-02-22T22:36:22.734+0000 7f2cb1f3f9c0 4 rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2026-02-22T22:36:22.755 INFO:teuthology.orchestra.run.trial023.stderr:2026-02-22T22:36:22.734+0000 7f2cb1f3f9c0 4 rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2026-02-22T22:36:22.756 INFO:teuthology.orchestra.run.trial023.stderr:2026-02-22T22:36:22.734+0000 7f2cb1f3f9c0 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2026-02-22T22:36:22.756 INFO:teuthology.orchestra.run.trial023.stderr:2026-02-22T22:36:22.734+0000 7f2cb1f3f9c0 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2026-02-22T22:36:22.756 INFO:teuthology.orchestra.run.trial023.stderr:2026-02-22T22:36:22.734+0000 7f2cb1f3f9c0 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2026-02-22T22:36:22.756 INFO:teuthology.orchestra.run.trial023.stderr:2026-02-22T22:36:22.734+0000 7f2cb1f3f9c0 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2026-02-22T22:36:22.756 INFO:teuthology.orchestra.run.trial023.stderr:2026-02-22T22:36:22.734+0000 7f2cb1f3f9c0 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2026-02-22T22:36:22.756 INFO:teuthology.orchestra.run.trial023.stderr:2026-02-22T22:36:22.734+0000 7f2cb1f3f9c0 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2026-02-22T22:36:22.756 INFO:teuthology.orchestra.run.trial023.stderr:2026-02-22T22:36:22.734+0000 7f2cb1f3f9c0 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2026-02-22T22:36:22.756 INFO:teuthology.orchestra.run.trial023.stderr:2026-02-22T22:36:22.734+0000 7f2cb1f3f9c0 4 rocksdb: Options.max_sequential_skip_in_iterations: 8 2026-02-22T22:36:22.756 INFO:teuthology.orchestra.run.trial023.stderr:2026-02-22T22:36:22.734+0000 7f2cb1f3f9c0 4 rocksdb: Options.max_compaction_bytes: 1677721600 2026-02-22T22:36:22.756 INFO:teuthology.orchestra.run.trial023.stderr:2026-02-22T22:36:22.734+0000 7f2cb1f3f9c0 4 rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2026-02-22T22:36:22.756 INFO:teuthology.orchestra.run.trial023.stderr:2026-02-22T22:36:22.734+0000 7f2cb1f3f9c0 4 rocksdb: Options.arena_block_size: 1048576 2026-02-22T22:36:22.756 INFO:teuthology.orchestra.run.trial023.stderr:2026-02-22T22:36:22.734+0000 7f2cb1f3f9c0 4 rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2026-02-22T22:36:22.756 INFO:teuthology.orchestra.run.trial023.stderr:2026-02-22T22:36:22.734+0000 7f2cb1f3f9c0 4 rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2026-02-22T22:36:22.757 INFO:teuthology.orchestra.run.trial023.stderr:2026-02-22T22:36:22.734+0000 7f2cb1f3f9c0 4 rocksdb: Options.disable_auto_compactions: 0 2026-02-22T22:36:22.757 INFO:teuthology.orchestra.run.trial023.stderr:2026-02-22T22:36:22.734+0000 7f2cb1f3f9c0 4 rocksdb: Options.compaction_style: kCompactionStyleLevel 2026-02-22T22:36:22.757 INFO:teuthology.orchestra.run.trial023.stderr:2026-02-22T22:36:22.734+0000 7f2cb1f3f9c0 4 rocksdb: Options.compaction_pri: kMinOverlappingRatio 2026-02-22T22:36:22.757 INFO:teuthology.orchestra.run.trial023.stderr:2026-02-22T22:36:22.734+0000 7f2cb1f3f9c0 4 rocksdb: Options.compaction_options_universal.size_ratio: 1 2026-02-22T22:36:22.757 INFO:teuthology.orchestra.run.trial023.stderr:2026-02-22T22:36:22.734+0000 7f2cb1f3f9c0 4 rocksdb: Options.compaction_options_universal.min_merge_width: 2 2026-02-22T22:36:22.757 INFO:teuthology.orchestra.run.trial023.stderr:2026-02-22T22:36:22.734+0000 7f2cb1f3f9c0 4 rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2026-02-22T22:36:22.757 INFO:teuthology.orchestra.run.trial023.stderr:2026-02-22T22:36:22.734+0000 7f2cb1f3f9c0 4 rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2026-02-22T22:36:22.757 INFO:teuthology.orchestra.run.trial023.stderr:2026-02-22T22:36:22.734+0000 7f2cb1f3f9c0 4 rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2026-02-22T22:36:22.757 INFO:teuthology.orchestra.run.trial023.stderr:2026-02-22T22:36:22.734+0000 7f2cb1f3f9c0 4 rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2026-02-22T22:36:22.757 INFO:teuthology.orchestra.run.trial023.stderr:2026-02-22T22:36:22.734+0000 7f2cb1f3f9c0 4 rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2026-02-22T22:36:22.757 INFO:teuthology.orchestra.run.trial023.stderr:2026-02-22T22:36:22.734+0000 7f2cb1f3f9c0 4 rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2026-02-22T22:36:22.757 INFO:teuthology.orchestra.run.trial023.stderr:2026-02-22T22:36:22.734+0000 7f2cb1f3f9c0 4 rocksdb: Options.table_properties_collectors: CompactOnDeletionCollector (Sliding window size = 32768 Deletion trigger = 16384 Deletion ratio = 0); 2026-02-22T22:36:22.757 INFO:teuthology.orchestra.run.trial023.stderr:2026-02-22T22:36:22.734+0000 7f2cb1f3f9c0 4 rocksdb: Options.inplace_update_support: 0 2026-02-22T22:36:22.758 INFO:teuthology.orchestra.run.trial023.stderr:2026-02-22T22:36:22.734+0000 7f2cb1f3f9c0 4 rocksdb: Options.inplace_update_num_locks: 10000 2026-02-22T22:36:22.758 INFO:teuthology.orchestra.run.trial023.stderr:2026-02-22T22:36:22.734+0000 7f2cb1f3f9c0 4 rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2026-02-22T22:36:22.758 INFO:teuthology.orchestra.run.trial023.stderr:2026-02-22T22:36:22.734+0000 7f2cb1f3f9c0 4 rocksdb: Options.memtable_whole_key_filtering: 0 2026-02-22T22:36:22.758 INFO:teuthology.orchestra.run.trial023.stderr:2026-02-22T22:36:22.734+0000 7f2cb1f3f9c0 4 rocksdb: Options.memtable_huge_page_size: 0 2026-02-22T22:36:22.758 INFO:teuthology.orchestra.run.trial023.stderr:2026-02-22T22:36:22.734+0000 7f2cb1f3f9c0 4 rocksdb: Options.bloom_locality: 0 2026-02-22T22:36:22.758 INFO:teuthology.orchestra.run.trial023.stderr:2026-02-22T22:36:22.734+0000 7f2cb1f3f9c0 4 rocksdb: Options.max_successive_merges: 0 2026-02-22T22:36:22.758 INFO:teuthology.orchestra.run.trial023.stderr:2026-02-22T22:36:22.734+0000 7f2cb1f3f9c0 4 rocksdb: Options.optimize_filters_for_hits: 0 2026-02-22T22:36:22.758 INFO:teuthology.orchestra.run.trial023.stderr:2026-02-22T22:36:22.734+0000 7f2cb1f3f9c0 4 rocksdb: Options.paranoid_file_checks: 0 2026-02-22T22:36:22.758 INFO:teuthology.orchestra.run.trial023.stderr:2026-02-22T22:36:22.734+0000 7f2cb1f3f9c0 4 rocksdb: Options.force_consistency_checks: 1 2026-02-22T22:36:22.758 INFO:teuthology.orchestra.run.trial023.stderr:2026-02-22T22:36:22.734+0000 7f2cb1f3f9c0 4 rocksdb: Options.report_bg_io_stats: 0 2026-02-22T22:36:22.758 INFO:teuthology.orchestra.run.trial023.stderr:2026-02-22T22:36:22.734+0000 7f2cb1f3f9c0 4 rocksdb: Options.ttl: 2592000 2026-02-22T22:36:22.758 INFO:teuthology.orchestra.run.trial023.stderr:2026-02-22T22:36:22.734+0000 7f2cb1f3f9c0 4 rocksdb: Options.periodic_compaction_seconds: 0 2026-02-22T22:36:22.758 INFO:teuthology.orchestra.run.trial023.stderr:2026-02-22T22:36:22.734+0000 7f2cb1f3f9c0 4 rocksdb: Options.preclude_last_level_data_seconds: 0 2026-02-22T22:36:22.758 INFO:teuthology.orchestra.run.trial023.stderr:2026-02-22T22:36:22.734+0000 7f2cb1f3f9c0 4 rocksdb: Options.preserve_internal_time_seconds: 0 2026-02-22T22:36:22.759 INFO:teuthology.orchestra.run.trial023.stderr:2026-02-22T22:36:22.734+0000 7f2cb1f3f9c0 4 rocksdb: Options.enable_blob_files: false 2026-02-22T22:36:22.759 INFO:teuthology.orchestra.run.trial023.stderr:2026-02-22T22:36:22.734+0000 7f2cb1f3f9c0 4 rocksdb: Options.min_blob_size: 0 2026-02-22T22:36:22.759 INFO:teuthology.orchestra.run.trial023.stderr:2026-02-22T22:36:22.734+0000 7f2cb1f3f9c0 4 rocksdb: Options.blob_file_size: 268435456 2026-02-22T22:36:22.759 INFO:teuthology.orchestra.run.trial023.stderr:2026-02-22T22:36:22.734+0000 7f2cb1f3f9c0 4 rocksdb: Options.blob_compression_type: NoCompression 2026-02-22T22:36:22.759 INFO:teuthology.orchestra.run.trial023.stderr:2026-02-22T22:36:22.734+0000 7f2cb1f3f9c0 4 rocksdb: Options.enable_blob_garbage_collection: false 2026-02-22T22:36:22.759 INFO:teuthology.orchestra.run.trial023.stderr:2026-02-22T22:36:22.734+0000 7f2cb1f3f9c0 4 rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2026-02-22T22:36:22.759 INFO:teuthology.orchestra.run.trial023.stderr:2026-02-22T22:36:22.734+0000 7f2cb1f3f9c0 4 rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2026-02-22T22:36:22.759 INFO:teuthology.orchestra.run.trial023.stderr:2026-02-22T22:36:22.734+0000 7f2cb1f3f9c0 4 rocksdb: Options.blob_compaction_readahead_size: 0 2026-02-22T22:36:22.759 INFO:teuthology.orchestra.run.trial023.stderr:2026-02-22T22:36:22.734+0000 7f2cb1f3f9c0 4 rocksdb: Options.blob_file_starting_level: 0 2026-02-22T22:36:22.759 INFO:teuthology.orchestra.run.trial023.stderr:2026-02-22T22:36:22.734+0000 7f2cb1f3f9c0 4 rocksdb: Options.experimental_mempurge_threshold: 0.000000 2026-02-22T22:36:22.759 INFO:teuthology.orchestra.run.trial023.stderr:2026-02-22T22:36:22.734+0000 7f2cb1f3f9c0 4 rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000015 succeeded,manifest_file_number is 15, next_file_number is 17, last_sequence is 84, log_number is 10,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 10 2026-02-22T22:36:22.759 INFO:teuthology.orchestra.run.trial023.stderr: 2026-02-22T22:36:22.759 INFO:teuthology.orchestra.run.trial023.stderr:2026-02-22T22:36:22.734+0000 7f2cb1f3f9c0 4 rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 10 2026-02-22T22:36:22.760 INFO:teuthology.orchestra.run.trial023.stderr: 2026-02-22T22:36:22.760 INFO:teuthology.orchestra.run.trial023.stderr:2026-02-22T22:36:22.734+0000 7f2cb1f3f9c0 4 rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: 66ebef0a-faa5-447d-85b0-98882949d3b5 2026-02-22T22:36:22.760 INFO:teuthology.orchestra.run.trial023.stderr: 2026-02-22T22:36:22.760 INFO:teuthology.orchestra.run.trial023.stderr:2026-02-22T22:36:22.734+0000 7f2cb1f3f9c0 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1771799782738549, "job": 1, "event": "recovery_started", "wal_files": [14]} 2026-02-22T22:36:22.760 INFO:teuthology.orchestra.run.trial023.stderr:2026-02-22T22:36:22.734+0000 7f2cb1f3f9c0 4 rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #14 mode 2 2026-02-22T22:36:22.760 INFO:teuthology.orchestra.run.trial023.stderr:2026-02-22T22:36:22.734+0000 7f2cb1f3f9c0 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1771799782738837, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 18, "file_size": 1370, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 85, "largest_seqno": 87, "table_properties": {"data_size": 237, "index_size": 40, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 69, "raw_key_size": 71, "raw_average_key_size": 23, "raw_value_size": 148, "raw_average_value_size": 49, "num_data_blocks": 1, "num_entries": 3, "num_filter_entries": 3, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[CompactOnDeletionCollector]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1771799782, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "66ebef0a-faa5-447d-85b0-98882949d3b5", "db_session_id": "VEJW5RMBCUXBDZ4QAHDS", "orig_file_number": 18, "seqno_to_time_mapping": "N/A"}} 2026-02-22T22:36:22.760 INFO:teuthology.orchestra.run.trial023.stderr:2026-02-22T22:36:22.734+0000 7f2cb1f3f9c0 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1771799782738923, "job": 1, "event": "recovery_finished"} 2026-02-22T22:36:22.760 INFO:teuthology.orchestra.run.trial023.stderr:2026-02-22T22:36:22.734+0000 7f2cb1f3f9c0 4 rocksdb: [db/version_set.cc:5047] Creating manifest 20 2026-02-22T22:36:22.760 INFO:teuthology.orchestra.run.trial023.stderr: 2026-02-22T22:36:22.760 INFO:teuthology.orchestra.run.trial023.stderr:2026-02-22T22:36:22.734+0000 7f2cb1f3f9c0 4 rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-a/store.db/000014.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2026-02-22T22:36:22.760 INFO:teuthology.orchestra.run.trial023.stderr:2026-02-22T22:36:22.734+0000 7f2cb1f3f9c0 4 rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x56064b4a6e00 2026-02-22T22:36:22.761 INFO:teuthology.orchestra.run.trial023.stderr:2026-02-22T22:36:22.734+0000 7f2cb1f3f9c0 4 rocksdb: DB pointer 0x56064b584000 2026-02-22T22:36:22.761 INFO:teuthology.orchestra.run.trial023.stderr:2026-02-22T22:36:22.734+0000 7f2ca8ef0640 4 rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2026-02-22T22:36:22.761 INFO:teuthology.orchestra.run.trial023.stderr:2026-02-22T22:36:22.734+0000 7f2ca8ef0640 4 rocksdb: [db/db_impl/db_impl.cc:1111] 2026-02-22T22:36:22.761 INFO:teuthology.orchestra.run.trial023.stderr:** DB Stats ** 2026-02-22T22:36:22.761 INFO:teuthology.orchestra.run.trial023.stderr:Uptime(secs): 0.0 total, 0.0 interval 2026-02-22T22:36:22.761 INFO:teuthology.orchestra.run.trial023.stderr:Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2026-02-22T22:36:22.761 INFO:teuthology.orchestra.run.trial023.stderr:Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2026-02-22T22:36:22.761 INFO:teuthology.orchestra.run.trial023.stderr:Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2026-02-22T22:36:22.761 INFO:teuthology.orchestra.run.trial023.stderr:Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2026-02-22T22:36:22.761 INFO:teuthology.orchestra.run.trial023.stderr:Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2026-02-22T22:36:22.761 INFO:teuthology.orchestra.run.trial023.stderr:Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2026-02-22T22:36:22.761 INFO:teuthology.orchestra.run.trial023.stderr: 2026-02-22T22:36:22.761 INFO:teuthology.orchestra.run.trial023.stderr:** Compaction Stats [default] ** 2026-02-22T22:36:22.762 INFO:teuthology.orchestra.run.trial023.stderr:Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2026-02-22T22:36:22.762 INFO:teuthology.orchestra.run.trial023.stderr:------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2026-02-22T22:36:22.762 INFO:teuthology.orchestra.run.trial023.stderr: L0 3/0 53.38 KB 0.8 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 4.7 0.00 0.00 1 0.000 0 0 0.0 0.0 2026-02-22T22:36:22.762 INFO:teuthology.orchestra.run.trial023.stderr: Sum 3/0 53.38 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 4.7 0.00 0.00 1 0.000 0 0 0.0 0.0 2026-02-22T22:36:22.762 INFO:teuthology.orchestra.run.trial023.stderr: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 4.7 0.00 0.00 1 0.000 0 0 0.0 0.0 2026-02-22T22:36:22.762 INFO:teuthology.orchestra.run.trial023.stderr: 2026-02-22T22:36:22.762 INFO:teuthology.orchestra.run.trial023.stderr:** Compaction Stats [default] ** 2026-02-22T22:36:22.762 INFO:teuthology.orchestra.run.trial023.stderr:Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2026-02-22T22:36:22.762 INFO:teuthology.orchestra.run.trial023.stderr:--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2026-02-22T22:36:22.762 INFO:teuthology.orchestra.run.trial023.stderr:User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 4.7 0.00 0.00 1 0.000 0 0 0.0 0.0 2026-02-22T22:36:22.762 INFO:teuthology.orchestra.run.trial023.stderr: 2026-02-22T22:36:22.762 INFO:teuthology.orchestra.run.trial023.stderr:Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2026-02-22T22:36:22.763 INFO:teuthology.orchestra.run.trial023.stderr: 2026-02-22T22:36:22.763 INFO:teuthology.orchestra.run.trial023.stderr:Uptime(secs): 0.0 total, 0.0 interval 2026-02-22T22:36:22.763 INFO:teuthology.orchestra.run.trial023.stderr:Flush(GB): cumulative 0.000, interval 0.000 2026-02-22T22:36:22.763 INFO:teuthology.orchestra.run.trial023.stderr:AddFile(GB): cumulative 0.000, interval 0.000 2026-02-22T22:36:22.763 INFO:teuthology.orchestra.run.trial023.stderr:AddFile(Total Files): cumulative 0, interval 0 2026-02-22T22:36:22.763 INFO:teuthology.orchestra.run.trial023.stderr:AddFile(L0 Files): cumulative 0, interval 0 2026-02-22T22:36:22.763 INFO:teuthology.orchestra.run.trial023.stderr:AddFile(Keys): cumulative 0, interval 0 2026-02-22T22:36:22.763 INFO:teuthology.orchestra.run.trial023.stderr:Cumulative compaction: 0.00 GB write, 0.72 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2026-02-22T22:36:22.763 INFO:teuthology.orchestra.run.trial023.stderr:Interval compaction: 0.00 GB write, 0.72 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2026-02-22T22:36:22.763 INFO:teuthology.orchestra.run.trial023.stderr:Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2026-02-22T22:36:22.763 INFO:teuthology.orchestra.run.trial023.stderr:Block cache BinnedLRUCache@0x56064b4a5770#22120 capacity: 512.00 MB usage: 1.02 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 5e-06 secs_since: 0 2026-02-22T22:36:22.763 INFO:teuthology.orchestra.run.trial023.stderr:Block cache entry stats(count,size,portion): FilterBlock(3,0.41 KB,7.89762e-05%) IndexBlock(3,0.61 KB,0.000116229%) Misc(1,0.00 KB,0%) 2026-02-22T22:36:22.763 INFO:teuthology.orchestra.run.trial023.stderr: 2026-02-22T22:36:22.764 INFO:teuthology.orchestra.run.trial023.stderr:** File Read Latency Histogram By Level [default] ** 2026-02-22T22:36:22.764 INFO:teuthology.orchestra.run.trial023.stderr: 2026-02-22T22:36:22.764 INFO:teuthology.orchestra.run.trial023.stderr:2026-02-22T22:36:22.734+0000 7f2cb1f3f9c0 2 auth: KeyRing::load: loaded key file /etc/ceph/ceph.keyring 2026-02-22T22:36:22.764 INFO:teuthology.orchestra.run.trial023.stdout:adding auth for 'mon.': auth(key=AQA/hJtpYEb+MxAA54slIDpAMLONgad2qad7Ww==) with caps({mon=allow *}) 2026-02-22T22:36:22.764 INFO:teuthology.orchestra.run.trial023.stdout:adding auth for 'osd.0': auth(key=AQBChJtpjLOHORAAvkjY/GrAk7aPGDSAbURRgQ==) with caps({mgr=allow profile osd,mon=allow profile osd,osd=allow *}) 2026-02-22T22:36:22.764 INFO:teuthology.orchestra.run.trial023.stdout:adding auth for 'osd.1': auth(key=AQBGhJtpvYgECxAAKLNLDNQwbcFFiJ7q0nvvpQ==) with caps({mgr=allow profile osd,mon=allow profile osd,osd=allow *}) 2026-02-22T22:36:22.765 INFO:teuthology.orchestra.run.trial023.stdout:adding auth for 'osd.2': auth(key=AQBJhJtpRFrhFxAAlc3+ueyf816Kt4pX15e01Q==) with caps({mgr=allow profile osd,mon=allow profile osd,osd=allow *}) 2026-02-22T22:36:22.765 INFO:teuthology.orchestra.run.trial023.stdout:adding auth for 'client.0': auth(key=AQBAhJtphSq0IBAAqjUMVmeT8RMwGr++VkA39g==) with caps({mds=allow,mgr=allow r,mon=allow rw,osd=allow rwx}) 2026-02-22T22:36:22.765 INFO:teuthology.orchestra.run.trial023.stdout:adding auth for 'client.admin': auth(key=AQBAhJtpSeoSDRAAaHCDERToCIs3fcJoA6aZwg==) with caps({mds=allow *,mgr=allow *,mon=allow *,osd=allow *}) 2026-02-22T22:36:22.765 INFO:teuthology.orchestra.run.trial023.stdout:adding auth for 'mgr.x': auth(key=AQBAhJtp8p7zHBAA9dOPoc865FewilZxaAlp3g==) with caps({mds=allow *,mon=allow profile mgr,osd=allow *}) 2026-02-22T22:36:22.766 INFO:teuthology.orchestra.run.trial023.stderr:2026-02-22T22:36:22.734+0000 7f2cb1f3f9c0 20 cephx keyserver: prepare_rotating_update before: data.rotating_ver=0 2026-02-22T22:36:22.766 INFO:teuthology.orchestra.run.trial023.stderr:2026-02-22T22:36:22.734+0000 7f2cb1f3f9c0 10 cephx keyserver: _rotate_secret adding auth 2026-02-22T22:36:22.766 INFO:teuthology.orchestra.run.trial023.stderr:2026-02-22T22:36:22.734+0000 7f2cb1f3f9c0 30 cephx keyserver: _rotate_secret adding auth id 1 AQDmhJtpajoaLBAAsdJQV5G74o8/I/PMAJVzZw== expires 2026-02-22T22:47:22.739915+0000 2026-02-22T22:36:22.766 INFO:teuthology.orchestra.run.trial023.stderr:2026-02-22T22:36:22.734+0000 7f2cb1f3f9c0 10 cephx keyserver: _rotate_secret adding auth 2026-02-22T22:36:22.766 INFO:teuthology.orchestra.run.trial023.stderr:2026-02-22T22:36:22.734+0000 7f2cb1f3f9c0 30 cephx keyserver: _rotate_secret adding auth id 2 AQDmhJtpjlQaLBAALwiKYP6qPcuYKBMlIcAS+A== expires 2026-02-22T22:58:22.739915+0000 2026-02-22T22:36:22.766 INFO:teuthology.orchestra.run.trial023.stderr:2026-02-22T22:36:22.734+0000 7f2cb1f3f9c0 10 cephx keyserver: _rotate_secret adding auth 2026-02-22T22:36:22.766 INFO:teuthology.orchestra.run.trial023.stderr:2026-02-22T22:36:22.734+0000 7f2cb1f3f9c0 30 cephx keyserver: _rotate_secret adding auth id 3 AQDmhJtp810aLBAAD0+seaRm8TWVcKCG3w/UrQ== expires 2026-02-22T23:09:22.739915+0000 2026-02-22T22:36:22.766 INFO:teuthology.orchestra.run.trial023.stderr:2026-02-22T22:36:22.734+0000 7f2cb1f3f9c0 10 cephx keyserver: _rotate_secret adding mon 2026-02-22T22:36:22.767 INFO:teuthology.orchestra.run.trial023.stderr:2026-02-22T22:36:22.734+0000 7f2cb1f3f9c0 30 cephx keyserver: _rotate_secret adding mon id 1 AQDmhJtpnmcaLBAAMbZpaQzfu2HpGGXiLzWYeA== expires 2026-02-22T22:40:22.739927+0000 2026-02-22T22:36:22.767 INFO:teuthology.orchestra.run.trial023.stderr:2026-02-22T22:36:22.734+0000 7f2cb1f3f9c0 10 cephx keyserver: _rotate_secret adding mon 2026-02-22T22:36:22.767 INFO:teuthology.orchestra.run.trial023.stderr:2026-02-22T22:36:22.734+0000 7f2cb1f3f9c0 30 cephx keyserver: _rotate_secret adding mon id 2 AQDmhJtp+HAaLBAAyYdbGUlfggjMCgRD4AhDSw== expires 2026-02-22T22:44:22.739927+0000 2026-02-22T22:36:22.767 INFO:teuthology.orchestra.run.trial023.stderr:2026-02-22T22:36:22.734+0000 7f2cb1f3f9c0 10 cephx keyserver: _rotate_secret adding mon 2026-02-22T22:36:22.767 INFO:teuthology.orchestra.run.trial023.stderr:2026-02-22T22:36:22.734+0000 7f2cb1f3f9c0 30 cephx keyserver: _rotate_secret adding mon id 3 AQDmhJtpD38aLBAAeZtqf+ppFf2r53PD0NT2Vg== expires 2026-02-22T22:48:22.739927+0000 2026-02-22T22:36:22.767 INFO:teuthology.orchestra.run.trial023.stderr:2026-02-22T22:36:22.734+0000 7f2cb1f3f9c0 10 cephx keyserver: _rotate_secret adding osd 2026-02-22T22:36:22.767 INFO:teuthology.orchestra.run.trial023.stderr:2026-02-22T22:36:22.734+0000 7f2cb1f3f9c0 30 cephx keyserver: _rotate_secret adding osd id 1 AQDmhJtptYcaLBAA4KPMnKO/akge2JF+GenTFQ== expires 2026-02-22T22:40:22.739935+0000 2026-02-22T22:36:22.767 INFO:teuthology.orchestra.run.trial023.stderr:2026-02-22T22:36:22.734+0000 7f2cb1f3f9c0 10 cephx keyserver: _rotate_secret adding osd 2026-02-22T22:36:22.767 INFO:teuthology.orchestra.run.trial023.stderr:2026-02-22T22:36:22.734+0000 7f2cb1f3f9c0 30 cephx keyserver: _rotate_secret adding osd id 2 AQDmhJtpPJIaLBAAAnzXl99DusgiwWewUGs9Vg== expires 2026-02-22T22:44:22.739935+0000 2026-02-22T22:36:22.767 INFO:teuthology.orchestra.run.trial023.stderr:2026-02-22T22:36:22.734+0000 7f2cb1f3f9c0 10 cephx keyserver: _rotate_secret adding osd 2026-02-22T22:36:22.767 INFO:teuthology.orchestra.run.trial023.stderr:2026-02-22T22:36:22.734+0000 7f2cb1f3f9c0 30 cephx keyserver: _rotate_secret adding osd id 3 AQDmhJtpdJoaLBAA0JRedBocjujUXxgX7apw/Q== expires 2026-02-22T22:48:22.739935+0000 2026-02-22T22:36:22.767 INFO:teuthology.orchestra.run.trial023.stderr:2026-02-22T22:36:22.734+0000 7f2cb1f3f9c0 10 cephx keyserver: _rotate_secret adding mds 2026-02-22T22:36:22.768 INFO:teuthology.orchestra.run.trial023.stderr:2026-02-22T22:36:22.734+0000 7f2cb1f3f9c0 30 cephx keyserver: _rotate_secret adding mds id 1 AQDmhJtptqIaLBAAO31e598FZ3+V7oPaldnLZg== expires 2026-02-22T22:40:22.739942+0000 2026-02-22T22:36:22.768 INFO:teuthology.orchestra.run.trial023.stderr:2026-02-22T22:36:22.734+0000 7f2cb1f3f9c0 10 cephx keyserver: _rotate_secret adding mds 2026-02-22T22:36:22.768 INFO:teuthology.orchestra.run.trial023.stderr:2026-02-22T22:36:22.734+0000 7f2cb1f3f9c0 30 cephx keyserver: _rotate_secret adding mds id 2 AQDmhJtpdKwaLBAAXnJhQdA+iNbOb0nXPqu32w== expires 2026-02-22T22:44:22.739942+0000 2026-02-22T22:36:22.768 INFO:teuthology.orchestra.run.trial023.stderr:2026-02-22T22:36:22.734+0000 7f2cb1f3f9c0 10 cephx keyserver: _rotate_secret adding mds 2026-02-22T22:36:22.768 INFO:teuthology.orchestra.run.trial023.stderr:2026-02-22T22:36:22.734+0000 7f2cb1f3f9c0 30 cephx keyserver: _rotate_secret adding mds id 3 AQDmhJtpdbUaLBAAIvnzIO/BHthQBTjJnP7RmA== expires 2026-02-22T22:48:22.739942+0000 2026-02-22T22:36:22.768 INFO:teuthology.orchestra.run.trial023.stderr:2026-02-22T22:36:22.734+0000 7f2cb1f3f9c0 10 cephx keyserver: _rotate_secret adding mgr 2026-02-22T22:36:22.768 INFO:teuthology.orchestra.run.trial023.stderr:2026-02-22T22:36:22.734+0000 7f2cb1f3f9c0 30 cephx keyserver: _rotate_secret adding mgr id 1 AQDmhJtp370aLBAALabYwlnS1yZuwL6B4rR23w== expires 2026-02-22T22:40:22.739949+0000 2026-02-22T22:36:22.768 INFO:teuthology.orchestra.run.trial023.stderr:2026-02-22T22:36:22.734+0000 7f2cb1f3f9c0 10 cephx keyserver: _rotate_secret adding mgr 2026-02-22T22:36:22.768 INFO:teuthology.orchestra.run.trial023.stderr:2026-02-22T22:36:22.734+0000 7f2cb1f3f9c0 30 cephx keyserver: _rotate_secret adding mgr id 2 AQDmhJtpo8YaLBAAYbx9jYDYlvuSYqikJpAlqw== expires 2026-02-22T22:44:22.739949+0000 2026-02-22T22:36:22.768 INFO:teuthology.orchestra.run.trial023.stderr:2026-02-22T22:36:22.734+0000 7f2cb1f3f9c0 10 cephx keyserver: _rotate_secret adding mgr 2026-02-22T22:36:22.768 INFO:teuthology.orchestra.run.trial023.stderr:2026-02-22T22:36:22.734+0000 7f2cb1f3f9c0 30 cephx keyserver: _rotate_secret adding mgr id 3 AQDmhJtp79caLBAAMaDLvPtVNUpx/Qqp34VNmw== expires 2026-02-22T22:48:22.739949+0000 2026-02-22T22:36:22.769 INFO:teuthology.orchestra.run.trial023.stderr:2026-02-22T22:36:22.734+0000 7f2cb1f3f9c0 20 cephx keyserver: prepare_rotating_update after: pending_data.rotating_ver=1 2026-02-22T22:36:22.769 INFO:teuthology.orchestra.run.trial023.stdout:update_mkfs pulling initial monmap from /tmp/monmap 2026-02-22T22:36:22.769 INFO:teuthology.orchestra.run.trial023.stderr:2026-02-22T22:36:22.738+0000 7f2cb1f3f9c0 4 rocksdb: [db/db_impl/db_impl.cc:496] Shutdown: canceling all background work 2026-02-22T22:36:22.770 INFO:teuthology.orchestra.run.trial023.stderr:2026-02-22T22:36:22.738+0000 7f2cb1f3f9c0 4 rocksdb: [db/db_impl/db_impl.cc:704] Shutdown complete 2026-02-22T22:36:22.770 INFO:teuthology.orchestra.run.trial023.stdout:epoch 1 2026-02-22T22:36:22.771 INFO:teuthology.orchestra.run.trial023.stdout:fsid a296be36-532d-4148-b28e-0183b3968057 2026-02-22T22:36:22.771 INFO:teuthology.orchestra.run.trial023.stdout:last_changed 2026-02-22T22:33:36.033736+0000 2026-02-22T22:36:22.771 INFO:teuthology.orchestra.run.trial023.stdout:created 2026-02-22T22:33:36.033736+0000 2026-02-22T22:36:22.771 INFO:teuthology.orchestra.run.trial023.stdout:min_mon_release 19 (squid) 2026-02-22T22:36:22.771 INFO:teuthology.orchestra.run.trial023.stdout:election_strategy: 3 2026-02-22T22:36:22.771 INFO:teuthology.orchestra.run.trial023.stdout:0: [v2:10.20.193.23:3300/0,v1:10.20.193.23:6789/0] mon.a 2026-02-22T22:36:22.771 INFO:teuthology.orchestra.run.trial023.stdout:1: [v2:10.20.193.23:3301/0,v1:10.20.193.23:6790/0] mon.b 2026-02-22T22:36:22.771 INFO:teuthology.orchestra.run.trial023.stdout:2: [v2:10.20.193.23:3302/0,v1:10.20.193.23:6791/0] mon.c 2026-02-22T22:36:22.771 INFO:tasks.rebuild_mondb:reviving mon.a 2026-02-22T22:36:22.772 INFO:tasks.ceph.mon.a:Restarting daemon 2026-02-22T22:36:22.772 DEBUG:teuthology.orchestra.run.trial023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2026-02-22T22:36:22.814 INFO:tasks.ceph.mon.a:Started 2026-02-22T22:36:22.814 INFO:tasks.rebuild_mondb:running mkfs on ceph:mon.b 2026-02-22T22:36:22.815 DEBUG:teuthology.orchestra.run.trial023:> sudo ceph-mon --cluster ceph --mkfs -i b --keyring /etc/ceph/ceph.keyring --monmap /tmp/monmap 2026-02-22T22:36:22.841 INFO:tasks.rebuild_mondb:reviving mon.b 2026-02-22T22:36:22.841 INFO:tasks.ceph.mon.b:Restarting daemon 2026-02-22T22:36:22.841 DEBUG:teuthology.orchestra.run.trial023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i b 2026-02-22T22:36:22.882 INFO:tasks.ceph.mon.b:Started 2026-02-22T22:36:22.882 INFO:tasks.rebuild_mondb:running mkfs on ceph:mon.c 2026-02-22T22:36:22.883 DEBUG:teuthology.orchestra.run.trial023:> sudo ceph-mon --cluster ceph --mkfs -i c --keyring /etc/ceph/ceph.keyring --monmap /tmp/monmap 2026-02-22T22:36:22.916 INFO:tasks.rebuild_mondb:reviving mon.c 2026-02-22T22:36:22.916 INFO:tasks.ceph.mon.c:Restarting daemon 2026-02-22T22:36:22.916 DEBUG:teuthology.orchestra.run.trial023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i c 2026-02-22T22:36:22.962 INFO:tasks.ceph.mon.c:Started 2026-02-22T22:36:22.963 INFO:tasks.rebuild_mondb.ceph_manager:waiting for quorum size 3 2026-02-22T22:36:22.963 DEBUG:teuthology.orchestra.run.trial023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2026-02-22T22:36:25.474 INFO:tasks.ceph.mgr.x.trial023.stderr:2026-02-22T22:36:25.474+0000 7f3ee9358640 -1 monclient(hunting): handle_auth_bad_method server allowed_methods [2] but i only support [2] 2026-02-22T22:36:25.474 INFO:tasks.ceph.mgr.x.trial023.stderr:2026-02-22T22:36:25.474+0000 7f3ee7354640 -1 monclient(hunting): handle_auth_bad_method server allowed_methods [2] but i only support [2] 2026-02-22T22:36:25.474 INFO:tasks.ceph.mgr.x.trial023.stderr:2026-02-22T22:36:25.474+0000 7f3ee8b57640 -1 monclient(hunting): handle_auth_bad_method server allowed_methods [2] but i only support [2] 2026-02-22T22:36:25.486 INFO:tasks.ceph.mgr.x.trial023.stderr:2026-02-22T22:36:25.486+0000 7f3ee8b57640 -1 monclient(hunting): handle_auth_bad_method server allowed_methods [2] but i only support [2] 2026-02-22T22:36:25.486 INFO:tasks.ceph.mgr.x.trial023.stderr:2026-02-22T22:36:25.486+0000 7f3ee7354640 -1 monclient(hunting): handle_auth_bad_method server allowed_methods [2] but i only support [2] 2026-02-22T22:36:25.486 INFO:tasks.ceph.mgr.x.trial023.stderr:2026-02-22T22:36:25.486+0000 7f3ee7354640 -1 monclient(hunting): handle_auth_bad_method server allowed_methods [2] but i only support [2] 2026-02-22T22:36:25.502 INFO:tasks.ceph.mgr.x.trial023.stderr:2026-02-22T22:36:25.502+0000 7f3ee9358640 -1 monclient(hunting): handle_auth_bad_method server allowed_methods [2] but i only support [2] 2026-02-22T22:36:25.502 INFO:tasks.ceph.mgr.x.trial023.stderr:2026-02-22T22:36:25.502+0000 7f3ee7354640 -1 monclient(hunting): handle_auth_bad_method server allowed_methods [2] but i only support [2] 2026-02-22T22:36:25.502 INFO:tasks.ceph.mgr.x.trial023.stderr:2026-02-22T22:36:25.502+0000 7f3ee7354640 -1 monclient(hunting): handle_auth_bad_method server allowed_methods [2] but i only support [2] 2026-02-22T22:36:25.506 INFO:tasks.ceph.mgr.x.trial023.stderr:2026-02-22T22:36:25.506+0000 7f3ee7354640 -1 monclient(hunting): handle_auth_bad_method server allowed_methods [2] but i only support [2] 2026-02-22T22:36:25.506 INFO:tasks.ceph.mgr.x.trial023.stderr:2026-02-22T22:36:25.506+0000 7f3ee9358640 -1 monclient(hunting): handle_auth_bad_method server allowed_methods [2] but i only support [2] 2026-02-22T22:36:25.506 INFO:tasks.ceph.mgr.x.trial023.stderr:2026-02-22T22:36:25.506+0000 7f3ee8b57640 -1 monclient(hunting): handle_auth_bad_method server allowed_methods [2] but i only support [2] 2026-02-22T22:36:33.470 INFO:teuthology.orchestra.run.trial023.stdout:{"election_epoch":2,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":0,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"a296be36-532d-4148-b28e-0183b3968057","modified":"2026-02-22T22:33:36.033736Z","created":"2026-02-22T22:33:36.033736Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":3,"disallowed_leaders":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.23:3300","nonce":0},{"type":"v1","addr":"10.20.193.23:6789","nonce":0}]},"addr":"10.20.193.23:6789/0","public_addr":"10.20.193.23:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.23:3301","nonce":0},{"type":"v1","addr":"10.20.193.23:6790","nonce":0}]},"addr":"10.20.193.23:6790/0","public_addr":"10.20.193.23:6790/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.23:3302","nonce":0},{"type":"v1","addr":"10.20.193.23:6791","nonce":0}]},"addr":"10.20.193.23:6791/0","public_addr":"10.20.193.23:6791/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2026-02-22T22:36:33.481 INFO:tasks.rebuild_mondb.ceph_manager:quorum is size 3 2026-02-22T22:36:33.481 INFO:tasks.rebuild_mondb:reviving mgr.x 2026-02-22T22:36:33.481 INFO:tasks.ceph.mgr.x:Restarting daemon 2026-02-22T22:36:33.481 INFO:tasks.ceph.mgr.x:Stopping old one... 2026-02-22T22:36:33.481 DEBUG:tasks.ceph.mgr.x:waiting for process to exit 2026-02-22T22:36:33.481 INFO:teuthology.orchestra.run:waiting for 300 2026-02-22T22:36:33.507 INFO:tasks.ceph.mgr.x:Stopped 2026-02-22T22:36:33.507 DEBUG:teuthology.orchestra.run.trial023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i x 2026-02-22T22:36:33.551 INFO:tasks.ceph.mgr.x:Started 2026-02-22T22:36:33.551 INFO:tasks.rebuild_mondb:reviving osd.0 2026-02-22T22:36:33.551 INFO:tasks.ceph.osd.0:Restarting daemon 2026-02-22T22:36:33.551 DEBUG:teuthology.orchestra.run.trial023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 0 2026-02-22T22:36:33.553 INFO:tasks.ceph.osd.0:Started 2026-02-22T22:36:33.553 DEBUG:teuthology.orchestra.run.trial023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2026-02-22T22:36:33.613 INFO:teuthology.orchestra.run.trial023.stderr:no valid command found; 10 closest matches: 2026-02-22T22:36:33.613 INFO:teuthology.orchestra.run.trial023.stderr:0 2026-02-22T22:36:33.613 INFO:teuthology.orchestra.run.trial023.stderr:1 2026-02-22T22:36:33.613 INFO:teuthology.orchestra.run.trial023.stderr:2 2026-02-22T22:36:33.613 INFO:teuthology.orchestra.run.trial023.stderr:abort 2026-02-22T22:36:33.613 INFO:teuthology.orchestra.run.trial023.stderr:assert 2026-02-22T22:36:33.614 INFO:teuthology.orchestra.run.trial023.stderr:config diff 2026-02-22T22:36:33.614 INFO:teuthology.orchestra.run.trial023.stderr:config diff get 2026-02-22T22:36:33.614 INFO:teuthology.orchestra.run.trial023.stderr:config get 2026-02-22T22:36:33.614 INFO:teuthology.orchestra.run.trial023.stderr:config help [] 2026-02-22T22:36:33.614 INFO:teuthology.orchestra.run.trial023.stderr:config set ... 2026-02-22T22:36:33.614 INFO:teuthology.orchestra.run.trial023.stderr:admin_socket: invalid command 2026-02-22T22:36:33.615 DEBUG:teuthology.orchestra.run:got remote process result: 22 2026-02-22T22:36:33.616 INFO:tasks.rebuild_mondb.ceph_manager:waiting on admin_socket for osd-0, ['dump_ops_in_flight'] 2026-02-22T22:36:33.628 INFO:tasks.ceph.mgr.x.trial023.stderr:2026-02-22T22:36:33.625+0000 7fa616e66280 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2026-02-22T22:36:33.653 INFO:tasks.ceph.mgr.x.trial023.stderr:2026-02-22T22:36:33.653+0000 7fa616e66280 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2026-02-22T22:36:33.691 INFO:tasks.ceph.mgr.x.trial023.stderr:2026-02-22T22:36:33.689+0000 7fa616e66280 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2026-02-22T22:36:33.755 INFO:tasks.ceph.mgr.x.trial023.stderr:2026-02-22T22:36:33.753+0000 7fa616e66280 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2026-02-22T22:36:33.821 INFO:tasks.ceph.mgr.x.trial023.stderr:2026-02-22T22:36:33.817+0000 7fa616e66280 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2026-02-22T22:36:33.858 INFO:tasks.ceph.mgr.x.trial023.stderr:2026-02-22T22:36:33.857+0000 7fa616e66280 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2026-02-22T22:36:33.926 INFO:tasks.ceph.mgr.x.trial023.stderr:2026-02-22T22:36:33.925+0000 7fa616e66280 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2026-02-22T22:36:34.088 INFO:tasks.ceph.mgr.x.trial023.stderr:2026-02-22T22:36:34.085+0000 7fa616e66280 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2026-02-22T22:36:34.328 INFO:tasks.ceph.mgr.x.trial023.stderr:2026-02-22T22:36:34.325+0000 7fa616e66280 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2026-02-22T22:36:34.416 INFO:tasks.ceph.mgr.x.trial023.stderr:2026-02-22T22:36:34.413+0000 7fa616e66280 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2026-02-22T22:36:34.442 INFO:tasks.ceph.mgr.x.trial023.stderr:2026-02-22T22:36:34.441+0000 7fa616e66280 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2026-02-22T22:36:34.469 INFO:tasks.ceph.mgr.x.trial023.stderr:2026-02-22T22:36:34.465+0000 7fa616e66280 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2026-02-22T22:36:34.499 INFO:tasks.ceph.mgr.x.trial023.stderr:2026-02-22T22:36:34.497+0000 7fa616e66280 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2026-02-22T22:36:34.594 INFO:tasks.ceph.mgr.x.trial023.stderr:2026-02-22T22:36:34.593+0000 7fa616e66280 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2026-02-22T22:36:34.760 INFO:tasks.ceph.mgr.x.trial023.stderr:ERROR:root:Module 'xmltodict' is not installed. 2026-02-22T22:36:34.909 INFO:tasks.ceph.mgr.x.trial023.stderr:2026-02-22T22:36:34.905+0000 7fa616e66280 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2026-02-22T22:36:34.940 INFO:tasks.ceph.mgr.x.trial023.stderr:2026-02-22T22:36:34.937+0000 7fa616e66280 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2026-02-22T22:36:35.012 INFO:tasks.ceph.mgr.x.trial023.stderr:2026-02-22T22:36:35.009+0000 7fa616e66280 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2026-02-22T22:36:35.037 INFO:tasks.ceph.mgr.x.trial023.stderr:2026-02-22T22:36:35.033+0000 7fa616e66280 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2026-02-22T22:36:35.062 INFO:tasks.ceph.mgr.x.trial023.stderr:2026-02-22T22:36:35.061+0000 7fa616e66280 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2026-02-22T22:36:35.087 INFO:tasks.ceph.mgr.x.trial023.stderr:2026-02-22T22:36:35.085+0000 7fa616e66280 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2026-02-22T22:36:35.103 INFO:tasks.ceph.osd.0.trial023.stderr:2026-02-22T22:36:35.101+0000 7fee548a28c0 -1 Falling back to public interface 2026-02-22T22:36:35.165 INFO:tasks.ceph.mgr.x.trial023.stderr:/usr/lib/python3/dist-packages/scipy/__init__.py:67: UserWarning: NumPy was imported from a Python sub-interpreter but NumPy does not properly support sub-interpreters. This will likely work for most users but might cause hard to track down issues or subtle bugs. A common user of the rare sub-interpreter feature is wsgi which also allows single-interpreter mode. 2026-02-22T22:36:35.165 INFO:tasks.ceph.mgr.x.trial023.stderr:Improvements in the case of bugs are welcome, but is not on the NumPy roadmap, and full support may require significant effort to achieve. 2026-02-22T22:36:35.165 INFO:tasks.ceph.mgr.x.trial023.stderr: from numpy import show_config as show_numpy_config 2026-02-22T22:36:35.171 INFO:tasks.ceph.mgr.x.trial023.stderr:2026-02-22T22:36:35.169+0000 7fa616e66280 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2026-02-22T22:36:35.204 INFO:tasks.ceph.mgr.x.trial023.stderr:2026-02-22T22:36:35.201+0000 7fa616e66280 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2026-02-22T22:36:35.242 INFO:tasks.ceph.mgr.x.trial023.stderr:2026-02-22T22:36:35.241+0000 7fa616e66280 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2026-02-22T22:36:36.329 INFO:tasks.ceph.mgr.x.trial023.stderr:2026-02-22T22:36:36.325+0000 7fa5356b9640 -1 mgr.server handle_report got status from non-daemon mon.b 2026-02-22T22:36:36.329 INFO:tasks.ceph.mgr.x.trial023.stderr:2026-02-22T22:36:36.325+0000 7fa5356b9640 -1 mgr.server handle_report got status from non-daemon mon.c 2026-02-22T22:36:38.617 DEBUG:teuthology.orchestra.run.trial023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2026-02-22T22:36:38.685 INFO:teuthology.orchestra.run.trial023.stderr:no valid command found; 10 closest matches: 2026-02-22T22:36:38.685 INFO:teuthology.orchestra.run.trial023.stderr:0 2026-02-22T22:36:38.685 INFO:teuthology.orchestra.run.trial023.stderr:1 2026-02-22T22:36:38.686 INFO:teuthology.orchestra.run.trial023.stderr:2 2026-02-22T22:36:38.686 INFO:teuthology.orchestra.run.trial023.stderr:abort 2026-02-22T22:36:38.686 INFO:teuthology.orchestra.run.trial023.stderr:assert 2026-02-22T22:36:38.686 INFO:teuthology.orchestra.run.trial023.stderr:bluestore allocator dump block 2026-02-22T22:36:38.686 INFO:teuthology.orchestra.run.trial023.stderr:bluestore allocator fragmentation block 2026-02-22T22:36:38.686 INFO:teuthology.orchestra.run.trial023.stderr:bluestore allocator fragmentation histogram block [] [] 2026-02-22T22:36:38.686 INFO:teuthology.orchestra.run.trial023.stderr:bluestore allocator score block 2026-02-22T22:36:38.686 INFO:teuthology.orchestra.run.trial023.stderr:config diff 2026-02-22T22:36:38.686 INFO:teuthology.orchestra.run.trial023.stderr:admin_socket: invalid command 2026-02-22T22:36:38.687 DEBUG:teuthology.orchestra.run:got remote process result: 22 2026-02-22T22:36:38.688 INFO:tasks.rebuild_mondb.ceph_manager:waiting on admin_socket for osd-0, ['dump_ops_in_flight'] 2026-02-22T22:36:38.971 INFO:tasks.ceph.osd.0.trial023.stderr:2026-02-22T22:36:38.969+0000 7fee548a28c0 -1 osd.0 19 log_to_monitors true 2026-02-22T22:36:39.370 INFO:tasks.ceph.osd.0.trial023.stderr:2026-02-22T22:36:39.369+0000 7fee4c686640 -1 osd.0 19 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2026-02-22T22:36:43.689 DEBUG:teuthology.orchestra.run.trial023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2026-02-22T22:36:43.752 INFO:teuthology.orchestra.run.trial023.stdout:{ 2026-02-22T22:36:43.752 INFO:teuthology.orchestra.run.trial023.stdout: "ops": [], 2026-02-22T22:36:43.752 INFO:teuthology.orchestra.run.trial023.stdout: "num_ops": 0 2026-02-22T22:36:43.752 INFO:teuthology.orchestra.run.trial023.stdout:} 2026-02-22T22:36:43.759 INFO:tasks.rebuild_mondb:reviving osd.1 2026-02-22T22:36:43.759 INFO:tasks.ceph.osd.1:Restarting daemon 2026-02-22T22:36:43.760 DEBUG:teuthology.orchestra.run.trial023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 1 2026-02-22T22:36:43.761 INFO:tasks.ceph.osd.1:Started 2026-02-22T22:36:43.761 DEBUG:teuthology.orchestra.run.trial023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2026-02-22T22:36:43.820 INFO:teuthology.orchestra.run.trial023.stderr:no valid command found; 10 closest matches: 2026-02-22T22:36:43.821 INFO:teuthology.orchestra.run.trial023.stderr:0 2026-02-22T22:36:43.821 INFO:teuthology.orchestra.run.trial023.stderr:1 2026-02-22T22:36:43.821 INFO:teuthology.orchestra.run.trial023.stderr:2 2026-02-22T22:36:43.821 INFO:teuthology.orchestra.run.trial023.stderr:abort 2026-02-22T22:36:43.821 INFO:teuthology.orchestra.run.trial023.stderr:assert 2026-02-22T22:36:43.821 INFO:teuthology.orchestra.run.trial023.stderr:config diff 2026-02-22T22:36:43.821 INFO:teuthology.orchestra.run.trial023.stderr:config diff get 2026-02-22T22:36:43.821 INFO:teuthology.orchestra.run.trial023.stderr:config get 2026-02-22T22:36:43.821 INFO:teuthology.orchestra.run.trial023.stderr:config help [] 2026-02-22T22:36:43.821 INFO:teuthology.orchestra.run.trial023.stderr:config set ... 2026-02-22T22:36:43.822 INFO:teuthology.orchestra.run.trial023.stderr:admin_socket: invalid command 2026-02-22T22:36:43.822 DEBUG:teuthology.orchestra.run:got remote process result: 22 2026-02-22T22:36:43.823 INFO:tasks.rebuild_mondb.ceph_manager:waiting on admin_socket for osd-1, ['dump_ops_in_flight'] 2026-02-22T22:36:45.307 INFO:tasks.ceph.osd.1.trial023.stderr:2026-02-22T22:36:45.304+0000 7fdc4f13f8c0 -1 Falling back to public interface 2026-02-22T22:36:48.824 DEBUG:teuthology.orchestra.run.trial023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2026-02-22T22:36:48.893 INFO:teuthology.orchestra.run.trial023.stderr:no valid command found; 10 closest matches: 2026-02-22T22:36:48.893 INFO:teuthology.orchestra.run.trial023.stderr:0 2026-02-22T22:36:48.893 INFO:teuthology.orchestra.run.trial023.stderr:1 2026-02-22T22:36:48.894 INFO:teuthology.orchestra.run.trial023.stderr:2 2026-02-22T22:36:48.894 INFO:teuthology.orchestra.run.trial023.stderr:abort 2026-02-22T22:36:48.894 INFO:teuthology.orchestra.run.trial023.stderr:assert 2026-02-22T22:36:48.894 INFO:teuthology.orchestra.run.trial023.stderr:config diff 2026-02-22T22:36:48.894 INFO:teuthology.orchestra.run.trial023.stderr:config diff get 2026-02-22T22:36:48.894 INFO:teuthology.orchestra.run.trial023.stderr:config get 2026-02-22T22:36:48.894 INFO:teuthology.orchestra.run.trial023.stderr:config help [] 2026-02-22T22:36:48.894 INFO:teuthology.orchestra.run.trial023.stderr:config set ... 2026-02-22T22:36:48.894 INFO:teuthology.orchestra.run.trial023.stderr:admin_socket: invalid command 2026-02-22T22:36:48.895 DEBUG:teuthology.orchestra.run:got remote process result: 22 2026-02-22T22:36:48.896 INFO:tasks.rebuild_mondb.ceph_manager:waiting on admin_socket for osd-1, ['dump_ops_in_flight'] 2026-02-22T22:36:49.275 INFO:tasks.ceph.osd.1.trial023.stderr:2026-02-22T22:36:49.272+0000 7fdc4f13f8c0 -1 osd.1 19 log_to_monitors true 2026-02-22T22:36:49.351 INFO:tasks.ceph.osd.1.trial023.stderr:2026-02-22T22:36:49.348+0000 7fdc42f1b640 -1 osd.1 19 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2026-02-22T22:36:53.897 DEBUG:teuthology.orchestra.run.trial023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2026-02-22T22:36:53.968 INFO:teuthology.orchestra.run.trial023.stdout:{ 2026-02-22T22:36:53.968 INFO:teuthology.orchestra.run.trial023.stdout: "ops": [], 2026-02-22T22:36:53.969 INFO:teuthology.orchestra.run.trial023.stdout: "num_ops": 0 2026-02-22T22:36:53.969 INFO:teuthology.orchestra.run.trial023.stdout:} 2026-02-22T22:36:53.975 INFO:tasks.rebuild_mondb:reviving osd.2 2026-02-22T22:36:53.975 INFO:tasks.ceph.osd.2:Restarting daemon 2026-02-22T22:36:53.976 DEBUG:teuthology.orchestra.run.trial023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 2 2026-02-22T22:36:53.985 INFO:tasks.ceph.osd.2:Started 2026-02-22T22:36:53.985 DEBUG:teuthology.orchestra.run.trial023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2026-02-22T22:36:54.049 INFO:teuthology.orchestra.run.trial023.stderr:no valid command found; 10 closest matches: 2026-02-22T22:36:54.049 INFO:teuthology.orchestra.run.trial023.stderr:0 2026-02-22T22:36:54.049 INFO:teuthology.orchestra.run.trial023.stderr:1 2026-02-22T22:36:54.049 INFO:teuthology.orchestra.run.trial023.stderr:2 2026-02-22T22:36:54.049 INFO:teuthology.orchestra.run.trial023.stderr:abort 2026-02-22T22:36:54.049 INFO:teuthology.orchestra.run.trial023.stderr:assert 2026-02-22T22:36:54.049 INFO:teuthology.orchestra.run.trial023.stderr:config diff 2026-02-22T22:36:54.049 INFO:teuthology.orchestra.run.trial023.stderr:config diff get 2026-02-22T22:36:54.049 INFO:teuthology.orchestra.run.trial023.stderr:config get 2026-02-22T22:36:54.049 INFO:teuthology.orchestra.run.trial023.stderr:config help [] 2026-02-22T22:36:54.049 INFO:teuthology.orchestra.run.trial023.stderr:config set ... 2026-02-22T22:36:54.049 INFO:teuthology.orchestra.run.trial023.stderr:admin_socket: invalid command 2026-02-22T22:36:54.051 DEBUG:teuthology.orchestra.run:got remote process result: 22 2026-02-22T22:36:54.051 INFO:tasks.rebuild_mondb.ceph_manager:waiting on admin_socket for osd-2, ['dump_ops_in_flight'] 2026-02-22T22:36:55.292 INFO:tasks.ceph.osd.2.trial023.stderr:2026-02-22T22:36:55.288+0000 7f5ff456d8c0 -1 Falling back to public interface 2026-02-22T22:36:59.052 DEBUG:teuthology.orchestra.run.trial023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2026-02-22T22:36:59.116 INFO:teuthology.orchestra.run.trial023.stderr:no valid command found; 10 closest matches: 2026-02-22T22:36:59.116 INFO:teuthology.orchestra.run.trial023.stderr:0 2026-02-22T22:36:59.116 INFO:teuthology.orchestra.run.trial023.stderr:1 2026-02-22T22:36:59.116 INFO:teuthology.orchestra.run.trial023.stderr:2 2026-02-22T22:36:59.116 INFO:teuthology.orchestra.run.trial023.stderr:abort 2026-02-22T22:36:59.117 INFO:teuthology.orchestra.run.trial023.stderr:assert 2026-02-22T22:36:59.117 INFO:teuthology.orchestra.run.trial023.stderr:bluestore allocator dump block 2026-02-22T22:36:59.117 INFO:teuthology.orchestra.run.trial023.stderr:bluestore allocator fragmentation block 2026-02-22T22:36:59.117 INFO:teuthology.orchestra.run.trial023.stderr:bluestore allocator fragmentation histogram block [] [] 2026-02-22T22:36:59.117 INFO:teuthology.orchestra.run.trial023.stderr:bluestore allocator score block 2026-02-22T22:36:59.117 INFO:teuthology.orchestra.run.trial023.stderr:config diff 2026-02-22T22:36:59.117 INFO:teuthology.orchestra.run.trial023.stderr:admin_socket: invalid command 2026-02-22T22:36:59.118 DEBUG:teuthology.orchestra.run:got remote process result: 22 2026-02-22T22:36:59.119 INFO:tasks.rebuild_mondb.ceph_manager:waiting on admin_socket for osd-2, ['dump_ops_in_flight'] 2026-02-22T22:36:59.138 INFO:tasks.ceph.osd.2.trial023.stderr:2026-02-22T22:36:59.132+0000 7f5ff456d8c0 -1 osd.2 19 log_to_monitors true 2026-02-22T22:36:59.455 INFO:tasks.ceph.osd.2.trial023.stderr:2026-02-22T22:36:59.452+0000 7f5fec351640 -1 osd.2 19 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2026-02-22T22:37:04.120 DEBUG:teuthology.orchestra.run.trial023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2026-02-22T22:37:04.184 INFO:teuthology.orchestra.run.trial023.stdout:{ 2026-02-22T22:37:04.184 INFO:teuthology.orchestra.run.trial023.stdout: "ops": [], 2026-02-22T22:37:04.184 INFO:teuthology.orchestra.run.trial023.stdout: "num_ops": 0 2026-02-22T22:37:04.184 INFO:teuthology.orchestra.run.trial023.stdout:} 2026-02-22T22:37:04.191 INFO:teuthology.task.full_sequential:In full_sequential, running task radosbench... 2026-02-22T22:37:04.192 INFO:tasks.radosbench:Beginning radosbench... 2026-02-22T22:37:04.192 INFO:tasks.ceph.ceph_manager.ceph:creating pool_name unique_pool_1 2026-02-22T22:37:04.192 DEBUG:teuthology.orchestra.run.trial023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create unique_pool_1 16 2026-02-22T22:37:04.484 INFO:teuthology.orchestra.run.trial023.stderr:pool 'unique_pool_1' created 2026-02-22T22:37:04.498 DEBUG:teuthology.orchestra.run.trial023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool application enable unique_pool_1 rados --yes-i-really-mean-it || true 2026-02-22T22:37:05.488 INFO:teuthology.orchestra.run.trial023.stderr:enabled application 'rados' on pool 'unique_pool_1' 2026-02-22T22:37:06.505 DEBUG:teuthology.orchestra.run.trial023:> /bin/sh -c 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --no-log-to-stderr --name client.0 -b 65536 --object-size 65536 -p unique_pool_1 bench 30 write' 2026-02-22T22:37:06.507 INFO:tasks.radosbench:joining radosbench (timing out after 1200s) 2026-02-22T22:37:06.507 INFO:teuthology.orchestra.run:waiting for 1200 2026-02-22T22:37:06.530 INFO:tasks.radosbench.radosbench.0.trial023.stdout:hints = 1 2026-02-22T22:37:06.530 INFO:tasks.radosbench.radosbench.0.trial023.stdout:Maintaining 16 concurrent writes of 65536 bytes to objects of size 65536 for up to 30 seconds or 0 objects 2026-02-22T22:37:06.530 INFO:tasks.radosbench.radosbench.0.trial023.stdout:Object prefix: benchmark_data_trial023_25083 2026-02-22T22:37:06.531 INFO:tasks.radosbench.radosbench.0.trial023.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2026-02-22T22:37:06.531 INFO:tasks.radosbench.radosbench.0.trial023.stdout: 0 0 0 0 0 0 - 0 2026-02-22T22:37:07.530 INFO:tasks.radosbench.radosbench.0.trial023.stdout: 1 16 8829 8813 550.774 550.812 0.00152349 0.00181251 2026-02-22T22:37:08.531 INFO:tasks.radosbench.radosbench.0.trial023.stdout: 2 16 17291 17275 539.804 528.875 0.00135365 0.00184976 2026-02-22T22:37:09.531 INFO:tasks.radosbench.radosbench.0.trial023.stdout: 3 16 23474 23458 488.672 386.438 0.00218477 0.00204377 2026-02-22T22:37:10.530 INFO:tasks.radosbench.radosbench.0.trial023.stdout: 4 16 30938 30922 483.12 466.5 0.00175296 0.00206772 2026-02-22T22:37:11.531 INFO:tasks.radosbench.radosbench.0.trial023.stdout: 5 16 37979 37963 474.502 440.062 0.00154784 0.00210534 2026-02-22T22:37:12.531 INFO:tasks.radosbench.radosbench.0.trial023.stdout: 6 16 43048 43032 448.215 316.812 0.00297951 0.00222913 2026-02-22T22:37:13.531 INFO:tasks.radosbench.radosbench.0.trial023.stdout: 7 16 49022 49006 437.52 373.375 0.0023688 0.00228369 2026-02-22T22:37:14.531 INFO:tasks.radosbench.radosbench.0.trial023.stdout: 8 16 54556 54540 426.061 345.875 0.00153311 0.00234501 2026-02-22T22:37:15.531 INFO:tasks.radosbench.radosbench.0.trial023.stdout: 9 16 60015 59999 416.628 341.188 0.00294725 0.00239799 2026-02-22T22:37:16.531 INFO:tasks.radosbench.radosbench.0.trial023.stdout: 10 16 65122 65106 406.881 319.188 0.00409534 0.00245576 2026-02-22T22:37:17.531 INFO:tasks.radosbench.radosbench.0.trial023.stdout: 11 16 70024 70008 397.742 306.375 0.00744213 0.00251232 2026-02-22T22:37:18.531 INFO:tasks.radosbench.radosbench.0.trial023.stdout: 12 16 74580 74564 388.325 284.75 0.00368668 0.00257316 2026-02-22T22:37:19.532 INFO:tasks.radosbench.radosbench.0.trial023.stdout: 13 16 78987 78971 379.639 275.438 0.00280279 0.00263215 2026-02-22T22:37:20.531 INFO:tasks.radosbench.radosbench.0.trial023.stdout: 14 16 83135 83119 371.039 259.25 0.00524279 0.00269319 2026-02-22T22:37:21.532 INFO:tasks.radosbench.radosbench.0.trial023.stdout: 15 16 87125 87109 362.927 249.375 0.00249608 0.00275336 2026-02-22T22:37:22.532 INFO:tasks.radosbench.radosbench.0.trial023.stdout: 16 16 90961 90945 355.227 239.75 0.00605893 0.00281315 2026-02-22T22:37:23.532 INFO:tasks.radosbench.radosbench.0.trial023.stdout: 17 16 94693 94677 348.051 233.25 0.0028562 0.00287129 2026-02-22T22:37:24.533 INFO:tasks.radosbench.radosbench.0.trial023.stdout: 18 16 98267 98251 341.124 223.375 0.00563711 0.00292952 2026-02-22T22:37:25.532 INFO:tasks.radosbench.radosbench.0.trial023.stdout: 19 16 101686 101670 334.416 213.688 0.00470555 0.00298832 2026-02-22T22:37:26.532 INFO:tasks.radosbench.radosbench.0.trial023.stdout:2026-02-22T22:37:26.533101+0000 min lat: 0.000682823 max lat: 0.206879 avg lat: 0.00304676 2026-02-22T22:37:26.532 INFO:tasks.radosbench.radosbench.0.trial023.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2026-02-22T22:37:26.533 INFO:tasks.radosbench.radosbench.0.trial023.stdout: 20 16 104981 104965 327.991 205.938 0.00912532 0.00304676 2026-02-22T22:37:27.532 INFO:tasks.radosbench.radosbench.0.trial023.stdout: 21 16 108220 108204 322.012 202.438 0.00364417 0.00310342 2026-02-22T22:37:28.532 INFO:tasks.radosbench.radosbench.0.trial023.stdout: 22 16 111320 111304 316.181 193.75 0.00438074 0.0031608 2026-02-22T22:37:29.532 INFO:tasks.radosbench.radosbench.0.trial023.stdout: 23 16 114317 114301 310.577 187.312 0.0031232 0.00321778 2026-02-22T22:37:30.532 INFO:tasks.radosbench.radosbench.0.trial023.stdout: 24 16 117248 117232 305.269 183.188 0.00230211 0.00327393 2026-02-22T22:37:31.532 INFO:tasks.radosbench.radosbench.0.trial023.stdout: 25 16 120113 120097 300.22 179.062 0.00569121 0.00332882 2026-02-22T22:37:32.533 INFO:tasks.radosbench.radosbench.0.trial023.stdout: 26 16 122924 122908 295.43 175.688 0.0113942 0.0033829 2026-02-22T22:37:33.533 INFO:tasks.radosbench.radosbench.0.trial023.stdout: 27 16 125656 125640 290.812 170.75 0.00665842 0.00343668 2026-02-22T22:37:34.533 INFO:tasks.radosbench.radosbench.0.trial023.stdout: 28 16 128338 128322 286.412 167.625 0.00532541 0.00348933 2026-02-22T22:37:35.533 INFO:tasks.radosbench.radosbench.0.trial023.stdout: 29 16 130337 130321 280.843 124.938 0.00655151 0.00355884 2026-02-22T22:37:36.533 INFO:tasks.radosbench.radosbench.0.trial023.stdout: 30 8 132867 132859 276.769 158.625 0.00339575 0.00361128 2026-02-22T22:37:37.533 INFO:tasks.radosbench.radosbench.0.trial023.stdout:Total time run: 30.0044 2026-02-22T22:37:37.533 INFO:tasks.radosbench.radosbench.0.trial023.stdout:Total writes made: 132867 2026-02-22T22:37:37.533 INFO:tasks.radosbench.radosbench.0.trial023.stdout:Write size: 65536 2026-02-22T22:37:37.533 INFO:tasks.radosbench.radosbench.0.trial023.stdout:Object size: 65536 2026-02-22T22:37:37.533 INFO:tasks.radosbench.radosbench.0.trial023.stdout:Bandwidth (MB/sec): 276.765 2026-02-22T22:37:37.533 INFO:tasks.radosbench.radosbench.0.trial023.stdout:Stddev Bandwidth: 111.606 2026-02-22T22:37:37.533 INFO:tasks.radosbench.radosbench.0.trial023.stdout:Max bandwidth (MB/sec): 550.812 2026-02-22T22:37:37.533 INFO:tasks.radosbench.radosbench.0.trial023.stdout:Min bandwidth (MB/sec): 124.938 2026-02-22T22:37:37.533 INFO:tasks.radosbench.radosbench.0.trial023.stdout:Average IOPS: 4428 2026-02-22T22:37:37.533 INFO:tasks.radosbench.radosbench.0.trial023.stdout:Stddev IOPS: 1785.7 2026-02-22T22:37:37.533 INFO:tasks.radosbench.radosbench.0.trial023.stdout:Max IOPS: 8813 2026-02-22T22:37:37.533 INFO:tasks.radosbench.radosbench.0.trial023.stdout:Min IOPS: 1999 2026-02-22T22:37:37.533 INFO:tasks.radosbench.radosbench.0.trial023.stdout:Average Latency(s): 0.00361139 2026-02-22T22:37:37.533 INFO:tasks.radosbench.radosbench.0.trial023.stdout:Stddev Latency(s): 0.00436895 2026-02-22T22:37:37.533 INFO:tasks.radosbench.radosbench.0.trial023.stdout:Max latency(s): 0.216672 2026-02-22T22:37:37.533 INFO:tasks.radosbench.radosbench.0.trial023.stdout:Min latency(s): 0.000682823 2026-02-22T22:37:37.537 INFO:tasks.radosbench.radosbench.0.trial023.stdout:Cleaning up (deleting benchmark objects) 2026-02-22T22:38:36.298 INFO:tasks.radosbench.radosbench.0.trial023.stdout:Removed 132867 objects 2026-02-22T22:38:36.298 INFO:tasks.radosbench.radosbench.0.trial023.stdout:Clean up completed and total clean up time :58.7596 2026-02-22T22:38:38.025 INFO:tasks.ceph.ceph_manager.ceph:removing pool_name unique_pool_1 2026-02-22T22:38:38.025 DEBUG:teuthology.orchestra.run.trial023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm unique_pool_1 unique_pool_1 --yes-i-really-really-mean-it 2026-02-22T22:38:38.575 INFO:teuthology.orchestra.run.trial023.stderr:pool 'unique_pool_1' removed 2026-02-22T22:38:38.591 DEBUG:teuthology.run_tasks:Unwinding manager ceph 2026-02-22T22:38:38.605 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2026-02-22T22:38:38.605 DEBUG:teuthology.orchestra.run.trial023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-22T22:38:38.753 INFO:teuthology.orchestra.run.trial023.stdout: 2026-02-22T22:38:38.753 INFO:teuthology.orchestra.run.trial023.stderr:dumped all 2026-02-22T22:38:38.764 INFO:teuthology.orchestra.run.trial023.stdout:{"pg_ready":true,"pg_map":{"version":73,"stamp":"2026-02-22T22:38:38.575285+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":81,"num_write_kb":1108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":466956,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":55,"ondisk_log_size":55,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":50,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":629096,"kb_used_data":6832,"kb_used_omap":13,"kb_used_meta":622194,"kb_avail":282486424,"statfs":{"total":289910292480,"available":289266098176,"internally_reserved":0,"allocated":6995968,"data_stored":96752518,"data_compressed":89802,"data_compressed_allocated":6025216,"data_compressed_original":96391168,"omap_allocated":13317,"internal_metadata":637127675},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,5,1,3,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-1857814552,"num_objects":-28349,"num_object_clones":0,"num_object_copies":-56698,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-28349,"num_whiteouts":0,"num_read":-1,"num_read_kb":-1,"num_write":-237387,"num_write_kb":-8503489,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"11.240245"},"pg_stats":[{"pgid":"1.0","version":"0'0","reported_seq":53,"reported_epoch":36,"state":"active+clean","last_fresh":"2026-02-22T22:37:06.533307+0000","last_change":"2026-02-22T22:36:52.342383+0000","last_active":"2026-02-22T22:37:06.533307+0000","last_peered":"2026-02-22T22:37:06.533307+0000","last_clean":"2026-02-22T22:37:06.533307+0000","last_became_active":"2026-02-22T22:36:52.342090+0000","last_became_peered":"2026-02-22T22:36:52.342090+0000","last_unstale":"2026-02-22T22:37:06.533307+0000","last_undegraded":"2026-02-22T22:37:06.533307+0000","last_fullsized":"2026-02-22T22:37:06.533307+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-22T22:34:05.232680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-22T22:34:05.232680+0000","last_clean_scrub_stamp":"2026-02-22T22:34:05.232680+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-24T03:43:43.610394+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00028225400000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":44,"reported_epoch":36,"state":"active+clean","last_fresh":"2026-02-22T22:37:06.489542+0000","last_change":"2026-02-22T22:37:01.433816+0000","last_active":"2026-02-22T22:37:06.489542+0000","last_peered":"2026-02-22T22:37:06.489542+0000","last_clean":"2026-02-22T22:37:06.489542+0000","last_became_active":"2026-02-22T22:37:01.433636+0000","last_became_peered":"2026-02-22T22:37:01.433636+0000","last_unstale":"2026-02-22T22:37:06.489542+0000","last_undegraded":"2026-02-22T22:37:06.489542+0000","last_fullsized":"2026-02-22T22:37:06.489542+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-22T22:34:05.232680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-22T22:34:05.232680+0000","last_clean_scrub_stamp":"2026-02-22T22:34:05.232680+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-24T02:57:31.972902+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"12'2","reported_seq":59,"reported_epoch":36,"state":"active+clean","last_fresh":"2026-02-22T22:37:06.490285+0000","last_change":"2026-02-22T22:36:52.343040+0000","last_active":"2026-02-22T22:37:06.490285+0000","last_peered":"2026-02-22T22:37:06.490285+0000","last_clean":"2026-02-22T22:37:06.490285+0000","last_became_active":"2026-02-22T22:36:52.342588+0000","last_became_peered":"2026-02-22T22:36:52.342588+0000","last_unstale":"2026-02-22T22:37:06.490285+0000","last_undegraded":"2026-02-22T22:37:06.490285+0000","last_fullsized":"2026-02-22T22:37:06.490285+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-22T22:34:05.232680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-22T22:34:05.232680+0000","last_clean_scrub_stamp":"2026-02-22T22:34:05.232680+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-24T03:37:21.715423+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00064906600000000001,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"30'52","reported_seq":100,"reported_epoch":36,"state":"active+clean","last_fresh":"2026-02-22T22:37:06.533102+0000","last_change":"2026-02-22T22:37:01.439717+0000","last_active":"2026-02-22T22:37:06.533102+0000","last_peered":"2026-02-22T22:37:06.533102+0000","last_clean":"2026-02-22T22:37:06.533102+0000","last_became_active":"2026-02-22T22:37:01.435620+0000","last_became_peered":"2026-02-22T22:37:01.435620+0000","last_unstale":"2026-02-22T22:37:06.533102+0000","last_undegraded":"2026-02-22T22:37:06.533102+0000","last_fullsized":"2026-02-22T22:37:06.533102+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-22T22:36:38.329086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-22T22:36:38.329086+0000","last_clean_scrub_stamp":"2026-02-22T22:36:38.329086+0000","objects_scrubbed":0,"log_size":52,"log_dups_size":0,"ondisk_log_size":52,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-24T10:03:30.349685+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":466956,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.3","version":"10'1","reported_seq":56,"reported_epoch":36,"state":"active+clean","last_fresh":"2026-02-22T22:37:06.533169+0000","last_change":"2026-02-22T22:37:01.435037+0000","last_active":"2026-02-22T22:37:06.533169+0000","last_peered":"2026-02-22T22:37:06.533169+0000","last_clean":"2026-02-22T22:37:06.533169+0000","last_became_active":"2026-02-22T22:37:01.434794+0000","last_became_peered":"2026-02-22T22:37:01.434794+0000","last_unstale":"2026-02-22T22:37:06.533169+0000","last_undegraded":"2026-02-22T22:37:06.533169+0000","last_fullsized":"2026-02-22T22:37:06.533169+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-22T22:34:05.232680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-22T22:34:05.232680+0000","last_clean_scrub_stamp":"2026-02-22T22:34:05.232680+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-23T22:56:31.019145+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00044206200000000001,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":53,"reported_epoch":36,"state":"active+clean","last_fresh":"2026-02-22T22:37:06.533286+0000","last_change":"2026-02-22T22:36:52.342697+0000","last_active":"2026-02-22T22:37:06.533286+0000","last_peered":"2026-02-22T22:37:06.533286+0000","last_clean":"2026-02-22T22:37:06.533286+0000","last_became_active":"2026-02-22T22:36:52.342463+0000","last_became_peered":"2026-02-22T22:36:52.342463+0000","last_unstale":"2026-02-22T22:37:06.533286+0000","last_undegraded":"2026-02-22T22:37:06.533286+0000","last_fullsized":"2026-02-22T22:37:06.533286+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-22T22:34:05.232680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-22T22:34:05.232680+0000","last_clean_scrub_stamp":"2026-02-22T22:34:05.232680+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-24T00:01:35.817682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000308413,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":44,"reported_epoch":36,"state":"active+clean","last_fresh":"2026-02-22T22:37:06.489870+0000","last_change":"2026-02-22T22:37:01.433582+0000","last_active":"2026-02-22T22:37:06.489870+0000","last_peered":"2026-02-22T22:37:06.489870+0000","last_clean":"2026-02-22T22:37:06.489870+0000","last_became_active":"2026-02-22T22:37:01.433380+0000","last_became_peered":"2026-02-22T22:37:01.433380+0000","last_unstale":"2026-02-22T22:37:06.489870+0000","last_undegraded":"2026-02-22T22:37:06.489870+0000","last_fullsized":"2026-02-22T22:37:06.489870+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-22T22:34:05.232680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-22T22:34:05.232680+0000","last_clean_scrub_stamp":"2026-02-22T22:34:05.232680+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-24T01:55:32.526508+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":53,"reported_epoch":36,"state":"active+clean","last_fresh":"2026-02-22T22:37:06.533514+0000","last_change":"2026-02-22T22:36:52.342309+0000","last_active":"2026-02-22T22:37:06.533514+0000","last_peered":"2026-02-22T22:37:06.533514+0000","last_clean":"2026-02-22T22:37:06.533514+0000","last_became_active":"2026-02-22T22:36:52.342082+0000","last_became_peered":"2026-02-22T22:36:52.342082+0000","last_unstale":"2026-02-22T22:37:06.533514+0000","last_undegraded":"2026-02-22T22:37:06.533514+0000","last_fullsized":"2026-02-22T22:37:06.533514+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-22T22:34:05.232680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-22T22:34:05.232680+0000","last_clean_scrub_stamp":"2026-02-22T22:34:05.232680+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-23T23:50:21.956500+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00060176900000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":55,"reported_epoch":36,"state":"active+clean","last_fresh":"2026-02-22T22:37:06.533637+0000","last_change":"2026-02-22T22:37:01.434974+0000","last_active":"2026-02-22T22:37:06.533637+0000","last_peered":"2026-02-22T22:37:06.533637+0000","last_clean":"2026-02-22T22:37:06.533637+0000","last_became_active":"2026-02-22T22:37:01.434745+0000","last_became_peered":"2026-02-22T22:37:01.434745+0000","last_unstale":"2026-02-22T22:37:06.533637+0000","last_undegraded":"2026-02-22T22:37:06.533637+0000","last_fullsized":"2026-02-22T22:37:06.533637+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-22T22:34:05.232680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-22T22:34:05.232680+0000","last_clean_scrub_stamp":"2026-02-22T22:34:05.232680+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-24T10:00:21.959586+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00045264099999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":466956,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":94208,"data_stored":933912,"data_compressed":6753,"data_compressed_allocated":57344,"data_compressed_original":905216,"omap_allocated":0,"internal_metadata":0},"log_size":52,"ondisk_log_size":52,"up":2,"acting":2,"num_store_stats":2},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":32,"seq":137438953492,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":173560,"kb_used_data":6176,"kb_used_omap":4,"kb_used_meta":167355,"kb_avail":94198280,"statfs":{"total":96636764160,"available":96459038720,"internally_reserved":0,"allocated":6324224,"data_stored":96063796,"data_compressed":86469,"data_compressed_allocated":5996544,"data_compressed_original":95940608,"omap_allocated":4438,"internal_metadata":171372202},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,4,1],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084311,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297200,"kb_used_data":352,"kb_used_omap":4,"kb_used_meta":296827,"kb_avail":94074640,"statfs":{"total":96636764160,"available":96332431360,"internally_reserved":0,"allocated":360448,"data_stored":577839,"data_compressed":3333,"data_compressed_allocated":28672,"data_compressed_original":450560,"omap_allocated":4439,"internal_metadata":303951529},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247832,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":158336,"kb_used_data":304,"kb_used_omap":4,"kb_used_meta":158011,"kb_avail":94213504,"statfs":{"total":96636764160,"available":96474628096,"internally_reserved":0,"allocated":311296,"data_stored":110883,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4440,"internal_metadata":161803944},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,1,0,3,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":466956,"data_compressed":3333,"data_compressed_allocated":28672,"data_compressed_original":450560,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":466956,"data_compressed":3420,"data_compressed_allocated":28672,"data_compressed_original":454656,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-22T22:38:38.766 DEBUG:teuthology.orchestra.run.trial023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-22T22:38:38.917 INFO:teuthology.orchestra.run.trial023.stdout: 2026-02-22T22:38:38.917 INFO:teuthology.orchestra.run.trial023.stderr:dumped all 2026-02-22T22:38:38.927 INFO:teuthology.orchestra.run.trial023.stdout:{"pg_ready":true,"pg_map":{"version":73,"stamp":"2026-02-22T22:38:38.575285+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":81,"num_write_kb":1108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":466956,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":55,"ondisk_log_size":55,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":50,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":629096,"kb_used_data":6832,"kb_used_omap":13,"kb_used_meta":622194,"kb_avail":282486424,"statfs":{"total":289910292480,"available":289266098176,"internally_reserved":0,"allocated":6995968,"data_stored":96752518,"data_compressed":89802,"data_compressed_allocated":6025216,"data_compressed_original":96391168,"omap_allocated":13317,"internal_metadata":637127675},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,5,1,3,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-1857814552,"num_objects":-28349,"num_object_clones":0,"num_object_copies":-56698,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-28349,"num_whiteouts":0,"num_read":-1,"num_read_kb":-1,"num_write":-237387,"num_write_kb":-8503489,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"11.240245"},"pg_stats":[{"pgid":"1.0","version":"0'0","reported_seq":53,"reported_epoch":36,"state":"active+clean","last_fresh":"2026-02-22T22:37:06.533307+0000","last_change":"2026-02-22T22:36:52.342383+0000","last_active":"2026-02-22T22:37:06.533307+0000","last_peered":"2026-02-22T22:37:06.533307+0000","last_clean":"2026-02-22T22:37:06.533307+0000","last_became_active":"2026-02-22T22:36:52.342090+0000","last_became_peered":"2026-02-22T22:36:52.342090+0000","last_unstale":"2026-02-22T22:37:06.533307+0000","last_undegraded":"2026-02-22T22:37:06.533307+0000","last_fullsized":"2026-02-22T22:37:06.533307+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-22T22:34:05.232680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-22T22:34:05.232680+0000","last_clean_scrub_stamp":"2026-02-22T22:34:05.232680+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-24T03:43:43.610394+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00028225400000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":44,"reported_epoch":36,"state":"active+clean","last_fresh":"2026-02-22T22:37:06.489542+0000","last_change":"2026-02-22T22:37:01.433816+0000","last_active":"2026-02-22T22:37:06.489542+0000","last_peered":"2026-02-22T22:37:06.489542+0000","last_clean":"2026-02-22T22:37:06.489542+0000","last_became_active":"2026-02-22T22:37:01.433636+0000","last_became_peered":"2026-02-22T22:37:01.433636+0000","last_unstale":"2026-02-22T22:37:06.489542+0000","last_undegraded":"2026-02-22T22:37:06.489542+0000","last_fullsized":"2026-02-22T22:37:06.489542+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-22T22:34:05.232680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-22T22:34:05.232680+0000","last_clean_scrub_stamp":"2026-02-22T22:34:05.232680+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-24T02:57:31.972902+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"12'2","reported_seq":59,"reported_epoch":36,"state":"active+clean","last_fresh":"2026-02-22T22:37:06.490285+0000","last_change":"2026-02-22T22:36:52.343040+0000","last_active":"2026-02-22T22:37:06.490285+0000","last_peered":"2026-02-22T22:37:06.490285+0000","last_clean":"2026-02-22T22:37:06.490285+0000","last_became_active":"2026-02-22T22:36:52.342588+0000","last_became_peered":"2026-02-22T22:36:52.342588+0000","last_unstale":"2026-02-22T22:37:06.490285+0000","last_undegraded":"2026-02-22T22:37:06.490285+0000","last_fullsized":"2026-02-22T22:37:06.490285+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-22T22:34:05.232680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-22T22:34:05.232680+0000","last_clean_scrub_stamp":"2026-02-22T22:34:05.232680+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-24T03:37:21.715423+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00064906600000000001,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"30'52","reported_seq":100,"reported_epoch":36,"state":"active+clean","last_fresh":"2026-02-22T22:37:06.533102+0000","last_change":"2026-02-22T22:37:01.439717+0000","last_active":"2026-02-22T22:37:06.533102+0000","last_peered":"2026-02-22T22:37:06.533102+0000","last_clean":"2026-02-22T22:37:06.533102+0000","last_became_active":"2026-02-22T22:37:01.435620+0000","last_became_peered":"2026-02-22T22:37:01.435620+0000","last_unstale":"2026-02-22T22:37:06.533102+0000","last_undegraded":"2026-02-22T22:37:06.533102+0000","last_fullsized":"2026-02-22T22:37:06.533102+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-22T22:36:38.329086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-22T22:36:38.329086+0000","last_clean_scrub_stamp":"2026-02-22T22:36:38.329086+0000","objects_scrubbed":0,"log_size":52,"log_dups_size":0,"ondisk_log_size":52,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-24T10:03:30.349685+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":466956,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.3","version":"10'1","reported_seq":56,"reported_epoch":36,"state":"active+clean","last_fresh":"2026-02-22T22:37:06.533169+0000","last_change":"2026-02-22T22:37:01.435037+0000","last_active":"2026-02-22T22:37:06.533169+0000","last_peered":"2026-02-22T22:37:06.533169+0000","last_clean":"2026-02-22T22:37:06.533169+0000","last_became_active":"2026-02-22T22:37:01.434794+0000","last_became_peered":"2026-02-22T22:37:01.434794+0000","last_unstale":"2026-02-22T22:37:06.533169+0000","last_undegraded":"2026-02-22T22:37:06.533169+0000","last_fullsized":"2026-02-22T22:37:06.533169+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-22T22:34:05.232680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-22T22:34:05.232680+0000","last_clean_scrub_stamp":"2026-02-22T22:34:05.232680+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-23T22:56:31.019145+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00044206200000000001,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":53,"reported_epoch":36,"state":"active+clean","last_fresh":"2026-02-22T22:37:06.533286+0000","last_change":"2026-02-22T22:36:52.342697+0000","last_active":"2026-02-22T22:37:06.533286+0000","last_peered":"2026-02-22T22:37:06.533286+0000","last_clean":"2026-02-22T22:37:06.533286+0000","last_became_active":"2026-02-22T22:36:52.342463+0000","last_became_peered":"2026-02-22T22:36:52.342463+0000","last_unstale":"2026-02-22T22:37:06.533286+0000","last_undegraded":"2026-02-22T22:37:06.533286+0000","last_fullsized":"2026-02-22T22:37:06.533286+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-22T22:34:05.232680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-22T22:34:05.232680+0000","last_clean_scrub_stamp":"2026-02-22T22:34:05.232680+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-24T00:01:35.817682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000308413,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":44,"reported_epoch":36,"state":"active+clean","last_fresh":"2026-02-22T22:37:06.489870+0000","last_change":"2026-02-22T22:37:01.433582+0000","last_active":"2026-02-22T22:37:06.489870+0000","last_peered":"2026-02-22T22:37:06.489870+0000","last_clean":"2026-02-22T22:37:06.489870+0000","last_became_active":"2026-02-22T22:37:01.433380+0000","last_became_peered":"2026-02-22T22:37:01.433380+0000","last_unstale":"2026-02-22T22:37:06.489870+0000","last_undegraded":"2026-02-22T22:37:06.489870+0000","last_fullsized":"2026-02-22T22:37:06.489870+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-22T22:34:05.232680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-22T22:34:05.232680+0000","last_clean_scrub_stamp":"2026-02-22T22:34:05.232680+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-24T01:55:32.526508+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":53,"reported_epoch":36,"state":"active+clean","last_fresh":"2026-02-22T22:37:06.533514+0000","last_change":"2026-02-22T22:36:52.342309+0000","last_active":"2026-02-22T22:37:06.533514+0000","last_peered":"2026-02-22T22:37:06.533514+0000","last_clean":"2026-02-22T22:37:06.533514+0000","last_became_active":"2026-02-22T22:36:52.342082+0000","last_became_peered":"2026-02-22T22:36:52.342082+0000","last_unstale":"2026-02-22T22:37:06.533514+0000","last_undegraded":"2026-02-22T22:37:06.533514+0000","last_fullsized":"2026-02-22T22:37:06.533514+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-22T22:34:05.232680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-22T22:34:05.232680+0000","last_clean_scrub_stamp":"2026-02-22T22:34:05.232680+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-23T23:50:21.956500+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00060176900000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":55,"reported_epoch":36,"state":"active+clean","last_fresh":"2026-02-22T22:37:06.533637+0000","last_change":"2026-02-22T22:37:01.434974+0000","last_active":"2026-02-22T22:37:06.533637+0000","last_peered":"2026-02-22T22:37:06.533637+0000","last_clean":"2026-02-22T22:37:06.533637+0000","last_became_active":"2026-02-22T22:37:01.434745+0000","last_became_peered":"2026-02-22T22:37:01.434745+0000","last_unstale":"2026-02-22T22:37:06.533637+0000","last_undegraded":"2026-02-22T22:37:06.533637+0000","last_fullsized":"2026-02-22T22:37:06.533637+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-22T22:34:05.232680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-22T22:34:05.232680+0000","last_clean_scrub_stamp":"2026-02-22T22:34:05.232680+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-24T10:00:21.959586+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00045264099999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":466956,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":94208,"data_stored":933912,"data_compressed":6753,"data_compressed_allocated":57344,"data_compressed_original":905216,"omap_allocated":0,"internal_metadata":0},"log_size":52,"ondisk_log_size":52,"up":2,"acting":2,"num_store_stats":2},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":32,"seq":137438953492,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":173560,"kb_used_data":6176,"kb_used_omap":4,"kb_used_meta":167355,"kb_avail":94198280,"statfs":{"total":96636764160,"available":96459038720,"internally_reserved":0,"allocated":6324224,"data_stored":96063796,"data_compressed":86469,"data_compressed_allocated":5996544,"data_compressed_original":95940608,"omap_allocated":4438,"internal_metadata":171372202},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,4,1],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084311,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297200,"kb_used_data":352,"kb_used_omap":4,"kb_used_meta":296827,"kb_avail":94074640,"statfs":{"total":96636764160,"available":96332431360,"internally_reserved":0,"allocated":360448,"data_stored":577839,"data_compressed":3333,"data_compressed_allocated":28672,"data_compressed_original":450560,"omap_allocated":4439,"internal_metadata":303951529},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247832,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":158336,"kb_used_data":304,"kb_used_omap":4,"kb_used_meta":158011,"kb_avail":94213504,"statfs":{"total":96636764160,"available":96474628096,"internally_reserved":0,"allocated":311296,"data_stored":110883,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4440,"internal_metadata":161803944},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,1,0,3,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":466956,"data_compressed":3333,"data_compressed_allocated":28672,"data_compressed_original":450560,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":466956,"data_compressed":3420,"data_compressed_allocated":28672,"data_compressed_original":454656,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-22T22:38:38.928 INFO:tasks.ceph.ceph_manager.ceph:clean! 2026-02-22T22:38:38.929 DEBUG:teuthology.orchestra.run.trial023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-22T22:38:39.082 INFO:teuthology.orchestra.run.trial023.stdout: 2026-02-22T22:38:39.082 INFO:teuthology.orchestra.run.trial023.stderr:dumped all 2026-02-22T22:38:39.093 INFO:teuthology.orchestra.run.trial023.stdout:{"pg_ready":true,"pg_map":{"version":73,"stamp":"2026-02-22T22:38:38.575285+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":81,"num_write_kb":1108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":466956,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":55,"ondisk_log_size":55,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":50,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":629096,"kb_used_data":6832,"kb_used_omap":13,"kb_used_meta":622194,"kb_avail":282486424,"statfs":{"total":289910292480,"available":289266098176,"internally_reserved":0,"allocated":6995968,"data_stored":96752518,"data_compressed":89802,"data_compressed_allocated":6025216,"data_compressed_original":96391168,"omap_allocated":13317,"internal_metadata":637127675},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,5,1,3,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-1857814552,"num_objects":-28349,"num_object_clones":0,"num_object_copies":-56698,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-28349,"num_whiteouts":0,"num_read":-1,"num_read_kb":-1,"num_write":-237387,"num_write_kb":-8503489,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"11.240245"},"pg_stats":[{"pgid":"1.0","version":"0'0","reported_seq":53,"reported_epoch":36,"state":"active+clean","last_fresh":"2026-02-22T22:37:06.533307+0000","last_change":"2026-02-22T22:36:52.342383+0000","last_active":"2026-02-22T22:37:06.533307+0000","last_peered":"2026-02-22T22:37:06.533307+0000","last_clean":"2026-02-22T22:37:06.533307+0000","last_became_active":"2026-02-22T22:36:52.342090+0000","last_became_peered":"2026-02-22T22:36:52.342090+0000","last_unstale":"2026-02-22T22:37:06.533307+0000","last_undegraded":"2026-02-22T22:37:06.533307+0000","last_fullsized":"2026-02-22T22:37:06.533307+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-22T22:34:05.232680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-22T22:34:05.232680+0000","last_clean_scrub_stamp":"2026-02-22T22:34:05.232680+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-24T03:43:43.610394+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00028225400000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":44,"reported_epoch":36,"state":"active+clean","last_fresh":"2026-02-22T22:37:06.489542+0000","last_change":"2026-02-22T22:37:01.433816+0000","last_active":"2026-02-22T22:37:06.489542+0000","last_peered":"2026-02-22T22:37:06.489542+0000","last_clean":"2026-02-22T22:37:06.489542+0000","last_became_active":"2026-02-22T22:37:01.433636+0000","last_became_peered":"2026-02-22T22:37:01.433636+0000","last_unstale":"2026-02-22T22:37:06.489542+0000","last_undegraded":"2026-02-22T22:37:06.489542+0000","last_fullsized":"2026-02-22T22:37:06.489542+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-22T22:34:05.232680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-22T22:34:05.232680+0000","last_clean_scrub_stamp":"2026-02-22T22:34:05.232680+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-24T02:57:31.972902+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"12'2","reported_seq":59,"reported_epoch":36,"state":"active+clean","last_fresh":"2026-02-22T22:37:06.490285+0000","last_change":"2026-02-22T22:36:52.343040+0000","last_active":"2026-02-22T22:37:06.490285+0000","last_peered":"2026-02-22T22:37:06.490285+0000","last_clean":"2026-02-22T22:37:06.490285+0000","last_became_active":"2026-02-22T22:36:52.342588+0000","last_became_peered":"2026-02-22T22:36:52.342588+0000","last_unstale":"2026-02-22T22:37:06.490285+0000","last_undegraded":"2026-02-22T22:37:06.490285+0000","last_fullsized":"2026-02-22T22:37:06.490285+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-22T22:34:05.232680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-22T22:34:05.232680+0000","last_clean_scrub_stamp":"2026-02-22T22:34:05.232680+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-24T03:37:21.715423+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00064906600000000001,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"30'52","reported_seq":100,"reported_epoch":36,"state":"active+clean","last_fresh":"2026-02-22T22:37:06.533102+0000","last_change":"2026-02-22T22:37:01.439717+0000","last_active":"2026-02-22T22:37:06.533102+0000","last_peered":"2026-02-22T22:37:06.533102+0000","last_clean":"2026-02-22T22:37:06.533102+0000","last_became_active":"2026-02-22T22:37:01.435620+0000","last_became_peered":"2026-02-22T22:37:01.435620+0000","last_unstale":"2026-02-22T22:37:06.533102+0000","last_undegraded":"2026-02-22T22:37:06.533102+0000","last_fullsized":"2026-02-22T22:37:06.533102+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-22T22:36:38.329086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-22T22:36:38.329086+0000","last_clean_scrub_stamp":"2026-02-22T22:36:38.329086+0000","objects_scrubbed":0,"log_size":52,"log_dups_size":0,"ondisk_log_size":52,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-24T10:03:30.349685+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":466956,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.3","version":"10'1","reported_seq":56,"reported_epoch":36,"state":"active+clean","last_fresh":"2026-02-22T22:37:06.533169+0000","last_change":"2026-02-22T22:37:01.435037+0000","last_active":"2026-02-22T22:37:06.533169+0000","last_peered":"2026-02-22T22:37:06.533169+0000","last_clean":"2026-02-22T22:37:06.533169+0000","last_became_active":"2026-02-22T22:37:01.434794+0000","last_became_peered":"2026-02-22T22:37:01.434794+0000","last_unstale":"2026-02-22T22:37:06.533169+0000","last_undegraded":"2026-02-22T22:37:06.533169+0000","last_fullsized":"2026-02-22T22:37:06.533169+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-22T22:34:05.232680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-22T22:34:05.232680+0000","last_clean_scrub_stamp":"2026-02-22T22:34:05.232680+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-23T22:56:31.019145+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00044206200000000001,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":53,"reported_epoch":36,"state":"active+clean","last_fresh":"2026-02-22T22:37:06.533286+0000","last_change":"2026-02-22T22:36:52.342697+0000","last_active":"2026-02-22T22:37:06.533286+0000","last_peered":"2026-02-22T22:37:06.533286+0000","last_clean":"2026-02-22T22:37:06.533286+0000","last_became_active":"2026-02-22T22:36:52.342463+0000","last_became_peered":"2026-02-22T22:36:52.342463+0000","last_unstale":"2026-02-22T22:37:06.533286+0000","last_undegraded":"2026-02-22T22:37:06.533286+0000","last_fullsized":"2026-02-22T22:37:06.533286+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-22T22:34:05.232680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-22T22:34:05.232680+0000","last_clean_scrub_stamp":"2026-02-22T22:34:05.232680+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-24T00:01:35.817682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000308413,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":44,"reported_epoch":36,"state":"active+clean","last_fresh":"2026-02-22T22:37:06.489870+0000","last_change":"2026-02-22T22:37:01.433582+0000","last_active":"2026-02-22T22:37:06.489870+0000","last_peered":"2026-02-22T22:37:06.489870+0000","last_clean":"2026-02-22T22:37:06.489870+0000","last_became_active":"2026-02-22T22:37:01.433380+0000","last_became_peered":"2026-02-22T22:37:01.433380+0000","last_unstale":"2026-02-22T22:37:06.489870+0000","last_undegraded":"2026-02-22T22:37:06.489870+0000","last_fullsized":"2026-02-22T22:37:06.489870+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-22T22:34:05.232680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-22T22:34:05.232680+0000","last_clean_scrub_stamp":"2026-02-22T22:34:05.232680+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-24T01:55:32.526508+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":53,"reported_epoch":36,"state":"active+clean","last_fresh":"2026-02-22T22:37:06.533514+0000","last_change":"2026-02-22T22:36:52.342309+0000","last_active":"2026-02-22T22:37:06.533514+0000","last_peered":"2026-02-22T22:37:06.533514+0000","last_clean":"2026-02-22T22:37:06.533514+0000","last_became_active":"2026-02-22T22:36:52.342082+0000","last_became_peered":"2026-02-22T22:36:52.342082+0000","last_unstale":"2026-02-22T22:37:06.533514+0000","last_undegraded":"2026-02-22T22:37:06.533514+0000","last_fullsized":"2026-02-22T22:37:06.533514+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-22T22:34:05.232680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-22T22:34:05.232680+0000","last_clean_scrub_stamp":"2026-02-22T22:34:05.232680+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-23T23:50:21.956500+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00060176900000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":55,"reported_epoch":36,"state":"active+clean","last_fresh":"2026-02-22T22:37:06.533637+0000","last_change":"2026-02-22T22:37:01.434974+0000","last_active":"2026-02-22T22:37:06.533637+0000","last_peered":"2026-02-22T22:37:06.533637+0000","last_clean":"2026-02-22T22:37:06.533637+0000","last_became_active":"2026-02-22T22:37:01.434745+0000","last_became_peered":"2026-02-22T22:37:01.434745+0000","last_unstale":"2026-02-22T22:37:06.533637+0000","last_undegraded":"2026-02-22T22:37:06.533637+0000","last_fullsized":"2026-02-22T22:37:06.533637+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-22T22:34:05.232680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-22T22:34:05.232680+0000","last_clean_scrub_stamp":"2026-02-22T22:34:05.232680+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-24T10:00:21.959586+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00045264099999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":466956,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":94208,"data_stored":933912,"data_compressed":6753,"data_compressed_allocated":57344,"data_compressed_original":905216,"omap_allocated":0,"internal_metadata":0},"log_size":52,"ondisk_log_size":52,"up":2,"acting":2,"num_store_stats":2},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":32,"seq":137438953492,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":173560,"kb_used_data":6176,"kb_used_omap":4,"kb_used_meta":167355,"kb_avail":94198280,"statfs":{"total":96636764160,"available":96459038720,"internally_reserved":0,"allocated":6324224,"data_stored":96063796,"data_compressed":86469,"data_compressed_allocated":5996544,"data_compressed_original":95940608,"omap_allocated":4438,"internal_metadata":171372202},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,4,1],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084311,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297200,"kb_used_data":352,"kb_used_omap":4,"kb_used_meta":296827,"kb_avail":94074640,"statfs":{"total":96636764160,"available":96332431360,"internally_reserved":0,"allocated":360448,"data_stored":577839,"data_compressed":3333,"data_compressed_allocated":28672,"data_compressed_original":450560,"omap_allocated":4439,"internal_metadata":303951529},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247832,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":158336,"kb_used_data":304,"kb_used_omap":4,"kb_used_meta":158011,"kb_avail":94213504,"statfs":{"total":96636764160,"available":96474628096,"internally_reserved":0,"allocated":311296,"data_stored":110883,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4440,"internal_metadata":161803944},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,1,0,3,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":466956,"data_compressed":3333,"data_compressed_allocated":28672,"data_compressed_original":450560,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":466956,"data_compressed":3420,"data_compressed_allocated":28672,"data_compressed_original":454656,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-22T22:38:39.095 DEBUG:teuthology.orchestra.run.trial023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2026-02-22T22:38:39.239 INFO:teuthology.orchestra.run.trial023.stdout: 2026-02-22T22:38:39.239 INFO:teuthology.orchestra.run.trial023.stdout:{"epoch":37,"fsid":"a296be36-532d-4148-b28e-0183b3968057","created":"2026-02-22T22:33:50.079561+0000","modified":"2026-02-22T22:38:38.573748+0000","last_up_change":"2026-02-22T22:37:00.427935+0000","last_in_change":"2026-02-22T22:33:50.733447+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":7,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":"rbd","create_time":"2026-02-22T22:34:04.294257+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"12","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":2,"snap_epoch":12,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":".mgr","create_time":"2026-02-22T22:36:37.356583+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"30","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.9900000095367432,"score_stable":2.9900000095367432,"optimal_score":0.67000001668930054,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"634275c9-6a87-4a66-8d94-04b38d5640bf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":8,"last_clean_end":19,"up_from":23,"up_thru":34,"down_at":22,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.23:6800","nonce":3271389585},{"type":"v1","addr":"10.20.193.23:6801","nonce":3271389585}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.23:6802","nonce":3271389585},{"type":"v1","addr":"10.20.193.23:6803","nonce":3271389585}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.23:6806","nonce":3271389585},{"type":"v1","addr":"10.20.193.23:6807","nonce":3271389585}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.23:6804","nonce":3271389585},{"type":"v1","addr":"10.20.193.23:6805","nonce":3271389585}]},"public_addr":"10.20.193.23:6801/3271389585","cluster_addr":"10.20.193.23:6803/3271389585","heartbeat_back_addr":"10.20.193.23:6807/3271389585","heartbeat_front_addr":"10.20.193.23:6805/3271389585","state":["exists","up"]},{"osd":1,"uuid":"f73635a0-7d9b-4967-9505-3f230f1a9463","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":8,"last_clean_end":19,"up_from":28,"up_thru":34,"down_at":27,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.23:6810","nonce":1240394092},{"type":"v1","addr":"10.20.193.23:6811","nonce":1240394092}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.23:6812","nonce":1240394092},{"type":"v1","addr":"10.20.193.23:6813","nonce":1240394092}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.23:6816","nonce":1240394092},{"type":"v1","addr":"10.20.193.23:6817","nonce":1240394092}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.23:6814","nonce":1240394092},{"type":"v1","addr":"10.20.193.23:6815","nonce":1240394092}]},"public_addr":"10.20.193.23:6811/1240394092","cluster_addr":"10.20.193.23:6813/1240394092","heartbeat_back_addr":"10.20.193.23:6817/1240394092","heartbeat_front_addr":"10.20.193.23:6815/1240394092","state":["exists","up"]},{"osd":2,"uuid":"23a9de65-37ff-46ca-83ab-c20c149a2276","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":8,"last_clean_end":19,"up_from":32,"up_thru":34,"down_at":24,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.23:6818","nonce":601947842},{"type":"v1","addr":"10.20.193.23:6819","nonce":601947842}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.23:6820","nonce":601947842},{"type":"v1","addr":"10.20.193.23:6821","nonce":601947842}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.23:6824","nonce":601947842},{"type":"v1","addr":"10.20.193.23:6825","nonce":601947842}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.23:6822","nonce":601947842},{"type":"v1","addr":"10.20.193.23:6823","nonce":601947842}]},"public_addr":"10.20.193.23:6819/601947842","cluster_addr":"10.20.193.23:6821/601947842","heartbeat_back_addr":"10.20.193.23:6825/601947842","heartbeat_front_addr":"10.20.193.23:6823/601947842","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"2026-02-22T22:36:40.338002+0000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2026-02-22T22:33:57.746206+0000","dead_epoch":0},{"osd":1,"down_stamp":"2026-02-22T22:36:50.332871+0000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2026-02-22T22:33:57.411823+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2026-02-22T22:33:57.826496+0000","dead_epoch":24}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-22T22:38:40.251 INFO:tasks.ceph:Scrubbing osd.0 2026-02-22T22:38:40.251 DEBUG:teuthology.orchestra.run.trial023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 config set osd_debug_deep_scrub_sleep 0 2026-02-22T22:38:40.336 INFO:teuthology.orchestra.run.trial023.stdout:{ 2026-02-22T22:38:40.336 INFO:teuthology.orchestra.run.trial023.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2026-02-22T22:38:40.336 INFO:teuthology.orchestra.run.trial023.stdout:} 2026-02-22T22:38:40.346 DEBUG:teuthology.orchestra.run.trial023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 0 2026-02-22T22:38:40.494 INFO:teuthology.orchestra.run.trial023.stderr:instructed osd(s) 0 to deep-scrub 2026-02-22T22:38:40.505 INFO:tasks.ceph:Scrubbing osd.1 2026-02-22T22:38:40.505 DEBUG:teuthology.orchestra.run.trial023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 config set osd_debug_deep_scrub_sleep 0 2026-02-22T22:38:40.580 INFO:teuthology.orchestra.run.trial023.stdout:{ 2026-02-22T22:38:40.580 INFO:teuthology.orchestra.run.trial023.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) osd_delete_sleep = '' osd_delete_sleep_hdd = '' osd_delete_sleep_hybrid = '' osd_delete_sleep_ssd = '' osd_max_backfills = '' osd_recovery_max_active = '' osd_recovery_max_active_hdd = '' osd_recovery_max_active_ssd = '' osd_recovery_sleep = '' osd_recovery_sleep_degraded = '' osd_recovery_sleep_degraded_hdd = '' osd_recovery_sleep_degraded_hybrid = '' osd_recovery_sleep_degraded_ssd = '' osd_recovery_sleep_hdd = '' osd_recovery_sleep_hybrid = '' osd_recovery_sleep_ssd = '' osd_scrub_sleep = '' osd_snap_trim_sleep = '' osd_snap_trim_sleep_hdd = '' osd_snap_trim_sleep_hybrid = '' osd_snap_trim_sleep_ssd = '' " 2026-02-22T22:38:40.581 INFO:teuthology.orchestra.run.trial023.stdout:} 2026-02-22T22:38:40.590 DEBUG:teuthology.orchestra.run.trial023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 1 2026-02-22T22:38:40.741 INFO:teuthology.orchestra.run.trial023.stderr:instructed osd(s) 1 to deep-scrub 2026-02-22T22:38:40.752 INFO:tasks.ceph:Scrubbing osd.2 2026-02-22T22:38:40.753 DEBUG:teuthology.orchestra.run.trial023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 config set osd_debug_deep_scrub_sleep 0 2026-02-22T22:38:40.827 INFO:teuthology.orchestra.run.trial023.stdout:{ 2026-02-22T22:38:40.828 INFO:teuthology.orchestra.run.trial023.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2026-02-22T22:38:40.828 INFO:teuthology.orchestra.run.trial023.stdout:} 2026-02-22T22:38:40.837 DEBUG:teuthology.orchestra.run.trial023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 2 2026-02-22T22:38:40.986 INFO:teuthology.orchestra.run.trial023.stderr:instructed osd(s) 2 to deep-scrub 2026-02-22T22:38:40.997 DEBUG:teuthology.orchestra.run.trial023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-22T22:38:41.147 INFO:teuthology.orchestra.run.trial023.stdout: 2026-02-22T22:38:41.147 INFO:teuthology.orchestra.run.trial023.stderr:dumped all 2026-02-22T22:38:41.157 INFO:teuthology.orchestra.run.trial023.stdout:{"pg_ready":true,"pg_map":{"version":74,"stamp":"2026-02-22T22:38:39.336653+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":81,"num_write_kb":1108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":466956,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":55,"ondisk_log_size":55,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":50,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":629096,"kb_used_data":6832,"kb_used_omap":13,"kb_used_meta":622194,"kb_avail":282486424,"statfs":{"total":289910292480,"available":289266098176,"internally_reserved":0,"allocated":6995968,"data_stored":96752518,"data_compressed":89802,"data_compressed_allocated":6025216,"data_compressed_original":96391168,"omap_allocated":13317,"internal_metadata":637127675},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,5,1,3,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-1270743064,"num_objects":-19391,"num_object_clones":0,"num_object_copies":-38782,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-19391,"num_whiteouts":0,"num_read":-1,"num_read_kb":-1,"num_write":-246345,"num_write_kb":-8503489,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.001357"},"pg_stats":[{"pgid":"1.0","version":"0'0","reported_seq":53,"reported_epoch":36,"state":"active+clean","last_fresh":"2026-02-22T22:37:06.533307+0000","last_change":"2026-02-22T22:36:52.342383+0000","last_active":"2026-02-22T22:37:06.533307+0000","last_peered":"2026-02-22T22:37:06.533307+0000","last_clean":"2026-02-22T22:37:06.533307+0000","last_became_active":"2026-02-22T22:36:52.342090+0000","last_became_peered":"2026-02-22T22:36:52.342090+0000","last_unstale":"2026-02-22T22:37:06.533307+0000","last_undegraded":"2026-02-22T22:37:06.533307+0000","last_fullsized":"2026-02-22T22:37:06.533307+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-22T22:34:05.232680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-22T22:34:05.232680+0000","last_clean_scrub_stamp":"2026-02-22T22:34:05.232680+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-24T03:43:43.610394+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00028225400000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":44,"reported_epoch":36,"state":"active+clean","last_fresh":"2026-02-22T22:37:06.489542+0000","last_change":"2026-02-22T22:37:01.433816+0000","last_active":"2026-02-22T22:37:06.489542+0000","last_peered":"2026-02-22T22:37:06.489542+0000","last_clean":"2026-02-22T22:37:06.489542+0000","last_became_active":"2026-02-22T22:37:01.433636+0000","last_became_peered":"2026-02-22T22:37:01.433636+0000","last_unstale":"2026-02-22T22:37:06.489542+0000","last_undegraded":"2026-02-22T22:37:06.489542+0000","last_fullsized":"2026-02-22T22:37:06.489542+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-22T22:34:05.232680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-22T22:34:05.232680+0000","last_clean_scrub_stamp":"2026-02-22T22:34:05.232680+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-24T02:57:31.972902+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"12'2","reported_seq":59,"reported_epoch":36,"state":"active+clean","last_fresh":"2026-02-22T22:37:06.490285+0000","last_change":"2026-02-22T22:36:52.343040+0000","last_active":"2026-02-22T22:37:06.490285+0000","last_peered":"2026-02-22T22:37:06.490285+0000","last_clean":"2026-02-22T22:37:06.490285+0000","last_became_active":"2026-02-22T22:36:52.342588+0000","last_became_peered":"2026-02-22T22:36:52.342588+0000","last_unstale":"2026-02-22T22:37:06.490285+0000","last_undegraded":"2026-02-22T22:37:06.490285+0000","last_fullsized":"2026-02-22T22:37:06.490285+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-22T22:34:05.232680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-22T22:34:05.232680+0000","last_clean_scrub_stamp":"2026-02-22T22:34:05.232680+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-24T03:37:21.715423+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00064906600000000001,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"30'52","reported_seq":100,"reported_epoch":36,"state":"active+clean","last_fresh":"2026-02-22T22:37:06.533102+0000","last_change":"2026-02-22T22:37:01.439717+0000","last_active":"2026-02-22T22:37:06.533102+0000","last_peered":"2026-02-22T22:37:06.533102+0000","last_clean":"2026-02-22T22:37:06.533102+0000","last_became_active":"2026-02-22T22:37:01.435620+0000","last_became_peered":"2026-02-22T22:37:01.435620+0000","last_unstale":"2026-02-22T22:37:06.533102+0000","last_undegraded":"2026-02-22T22:37:06.533102+0000","last_fullsized":"2026-02-22T22:37:06.533102+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-22T22:36:38.329086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-22T22:36:38.329086+0000","last_clean_scrub_stamp":"2026-02-22T22:36:38.329086+0000","objects_scrubbed":0,"log_size":52,"log_dups_size":0,"ondisk_log_size":52,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-24T10:03:30.349685+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":466956,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.3","version":"10'1","reported_seq":56,"reported_epoch":36,"state":"active+clean","last_fresh":"2026-02-22T22:37:06.533169+0000","last_change":"2026-02-22T22:37:01.435037+0000","last_active":"2026-02-22T22:37:06.533169+0000","last_peered":"2026-02-22T22:37:06.533169+0000","last_clean":"2026-02-22T22:37:06.533169+0000","last_became_active":"2026-02-22T22:37:01.434794+0000","last_became_peered":"2026-02-22T22:37:01.434794+0000","last_unstale":"2026-02-22T22:37:06.533169+0000","last_undegraded":"2026-02-22T22:37:06.533169+0000","last_fullsized":"2026-02-22T22:37:06.533169+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-22T22:34:05.232680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-22T22:34:05.232680+0000","last_clean_scrub_stamp":"2026-02-22T22:34:05.232680+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-23T22:56:31.019145+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00044206200000000001,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":53,"reported_epoch":36,"state":"active+clean","last_fresh":"2026-02-22T22:37:06.533286+0000","last_change":"2026-02-22T22:36:52.342697+0000","last_active":"2026-02-22T22:37:06.533286+0000","last_peered":"2026-02-22T22:37:06.533286+0000","last_clean":"2026-02-22T22:37:06.533286+0000","last_became_active":"2026-02-22T22:36:52.342463+0000","last_became_peered":"2026-02-22T22:36:52.342463+0000","last_unstale":"2026-02-22T22:37:06.533286+0000","last_undegraded":"2026-02-22T22:37:06.533286+0000","last_fullsized":"2026-02-22T22:37:06.533286+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-22T22:34:05.232680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-22T22:34:05.232680+0000","last_clean_scrub_stamp":"2026-02-22T22:34:05.232680+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-24T00:01:35.817682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000308413,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":44,"reported_epoch":36,"state":"active+clean","last_fresh":"2026-02-22T22:37:06.489870+0000","last_change":"2026-02-22T22:37:01.433582+0000","last_active":"2026-02-22T22:37:06.489870+0000","last_peered":"2026-02-22T22:37:06.489870+0000","last_clean":"2026-02-22T22:37:06.489870+0000","last_became_active":"2026-02-22T22:37:01.433380+0000","last_became_peered":"2026-02-22T22:37:01.433380+0000","last_unstale":"2026-02-22T22:37:06.489870+0000","last_undegraded":"2026-02-22T22:37:06.489870+0000","last_fullsized":"2026-02-22T22:37:06.489870+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-22T22:34:05.232680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-22T22:34:05.232680+0000","last_clean_scrub_stamp":"2026-02-22T22:34:05.232680+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-24T01:55:32.526508+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":53,"reported_epoch":36,"state":"active+clean","last_fresh":"2026-02-22T22:37:06.533514+0000","last_change":"2026-02-22T22:36:52.342309+0000","last_active":"2026-02-22T22:37:06.533514+0000","last_peered":"2026-02-22T22:37:06.533514+0000","last_clean":"2026-02-22T22:37:06.533514+0000","last_became_active":"2026-02-22T22:36:52.342082+0000","last_became_peered":"2026-02-22T22:36:52.342082+0000","last_unstale":"2026-02-22T22:37:06.533514+0000","last_undegraded":"2026-02-22T22:37:06.533514+0000","last_fullsized":"2026-02-22T22:37:06.533514+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-22T22:34:05.232680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-22T22:34:05.232680+0000","last_clean_scrub_stamp":"2026-02-22T22:34:05.232680+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-23T23:50:21.956500+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00060176900000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":55,"reported_epoch":36,"state":"active+clean","last_fresh":"2026-02-22T22:37:06.533637+0000","last_change":"2026-02-22T22:37:01.434974+0000","last_active":"2026-02-22T22:37:06.533637+0000","last_peered":"2026-02-22T22:37:06.533637+0000","last_clean":"2026-02-22T22:37:06.533637+0000","last_became_active":"2026-02-22T22:37:01.434745+0000","last_became_peered":"2026-02-22T22:37:01.434745+0000","last_unstale":"2026-02-22T22:37:06.533637+0000","last_undegraded":"2026-02-22T22:37:06.533637+0000","last_fullsized":"2026-02-22T22:37:06.533637+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-22T22:34:05.232680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-22T22:34:05.232680+0000","last_clean_scrub_stamp":"2026-02-22T22:34:05.232680+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-24T10:00:21.959586+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00045264099999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":466956,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":94208,"data_stored":933912,"data_compressed":6753,"data_compressed_allocated":57344,"data_compressed_original":905216,"omap_allocated":0,"internal_metadata":0},"log_size":52,"ondisk_log_size":52,"up":2,"acting":2,"num_store_stats":2},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":32,"seq":137438953492,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":173560,"kb_used_data":6176,"kb_used_omap":4,"kb_used_meta":167355,"kb_avail":94198280,"statfs":{"total":96636764160,"available":96459038720,"internally_reserved":0,"allocated":6324224,"data_stored":96063796,"data_compressed":86469,"data_compressed_allocated":5996544,"data_compressed_original":95940608,"omap_allocated":4438,"internal_metadata":171372202},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,4,1],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084311,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297200,"kb_used_data":352,"kb_used_omap":4,"kb_used_meta":296827,"kb_avail":94074640,"statfs":{"total":96636764160,"available":96332431360,"internally_reserved":0,"allocated":360448,"data_stored":577839,"data_compressed":3333,"data_compressed_allocated":28672,"data_compressed_original":450560,"omap_allocated":4439,"internal_metadata":303951529},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247832,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":158336,"kb_used_data":304,"kb_used_omap":4,"kb_used_meta":158011,"kb_avail":94213504,"statfs":{"total":96636764160,"available":96474628096,"internally_reserved":0,"allocated":311296,"data_stored":110883,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4440,"internal_metadata":161803944},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,1,0,3,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":466956,"data_compressed":3333,"data_compressed_allocated":28672,"data_compressed_original":450560,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":466956,"data_compressed":3420,"data_compressed_allocated":28672,"data_compressed_original":454656,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-22T22:38:41.162 INFO:tasks.ceph:pgid 1.0 last_scrub_stamp 2026-02-22T22:34:05.232680+0000 time.struct_time(tm_year=2026, tm_mon=2, tm_mday=22, tm_hour=22, tm_min=34, tm_sec=5, tm_wday=6, tm_yday=53, tm_isdst=-1) <= time.struct_time(tm_year=2026, tm_mon=2, tm_mday=22, tm_hour=22, tm_min=38, tm_sec=39, tm_wday=6, tm_yday=53, tm_isdst=0) 2026-02-22T22:38:41.163 INFO:tasks.ceph:pgid 1.1 last_scrub_stamp 2026-02-22T22:34:05.232680+0000 time.struct_time(tm_year=2026, tm_mon=2, tm_mday=22, tm_hour=22, tm_min=34, tm_sec=5, tm_wday=6, tm_yday=53, tm_isdst=-1) <= time.struct_time(tm_year=2026, tm_mon=2, tm_mday=22, tm_hour=22, tm_min=38, tm_sec=39, tm_wday=6, tm_yday=53, tm_isdst=0) 2026-02-22T22:38:41.163 INFO:tasks.ceph:pgid 1.2 last_scrub_stamp 2026-02-22T22:34:05.232680+0000 time.struct_time(tm_year=2026, tm_mon=2, tm_mday=22, tm_hour=22, tm_min=34, tm_sec=5, tm_wday=6, tm_yday=53, tm_isdst=-1) <= time.struct_time(tm_year=2026, tm_mon=2, tm_mday=22, tm_hour=22, tm_min=38, tm_sec=39, tm_wday=6, tm_yday=53, tm_isdst=0) 2026-02-22T22:38:41.163 INFO:tasks.ceph:pgid 3.0 last_scrub_stamp 2026-02-22T22:36:38.329086+0000 time.struct_time(tm_year=2026, tm_mon=2, tm_mday=22, tm_hour=22, tm_min=36, tm_sec=38, tm_wday=6, tm_yday=53, tm_isdst=-1) <= time.struct_time(tm_year=2026, tm_mon=2, tm_mday=22, tm_hour=22, tm_min=38, tm_sec=39, tm_wday=6, tm_yday=53, tm_isdst=0) 2026-02-22T22:38:41.163 INFO:tasks.ceph:pgid 1.3 last_scrub_stamp 2026-02-22T22:34:05.232680+0000 time.struct_time(tm_year=2026, tm_mon=2, tm_mday=22, tm_hour=22, tm_min=34, tm_sec=5, tm_wday=6, tm_yday=53, tm_isdst=-1) <= time.struct_time(tm_year=2026, tm_mon=2, tm_mday=22, tm_hour=22, tm_min=38, tm_sec=39, tm_wday=6, tm_yday=53, tm_isdst=0) 2026-02-22T22:38:41.163 INFO:tasks.ceph:pgid 1.4 last_scrub_stamp 2026-02-22T22:34:05.232680+0000 time.struct_time(tm_year=2026, tm_mon=2, tm_mday=22, tm_hour=22, tm_min=34, tm_sec=5, tm_wday=6, tm_yday=53, tm_isdst=-1) <= time.struct_time(tm_year=2026, tm_mon=2, tm_mday=22, tm_hour=22, tm_min=38, tm_sec=39, tm_wday=6, tm_yday=53, tm_isdst=0) 2026-02-22T22:38:41.164 INFO:tasks.ceph:pgid 1.5 last_scrub_stamp 2026-02-22T22:34:05.232680+0000 time.struct_time(tm_year=2026, tm_mon=2, tm_mday=22, tm_hour=22, tm_min=34, tm_sec=5, tm_wday=6, tm_yday=53, tm_isdst=-1) <= time.struct_time(tm_year=2026, tm_mon=2, tm_mday=22, tm_hour=22, tm_min=38, tm_sec=39, tm_wday=6, tm_yday=53, tm_isdst=0) 2026-02-22T22:38:41.164 INFO:tasks.ceph:pgid 1.6 last_scrub_stamp 2026-02-22T22:34:05.232680+0000 time.struct_time(tm_year=2026, tm_mon=2, tm_mday=22, tm_hour=22, tm_min=34, tm_sec=5, tm_wday=6, tm_yday=53, tm_isdst=-1) <= time.struct_time(tm_year=2026, tm_mon=2, tm_mday=22, tm_hour=22, tm_min=38, tm_sec=39, tm_wday=6, tm_yday=53, tm_isdst=0) 2026-02-22T22:38:41.164 INFO:tasks.ceph:pgid 1.7 last_scrub_stamp 2026-02-22T22:34:05.232680+0000 time.struct_time(tm_year=2026, tm_mon=2, tm_mday=22, tm_hour=22, tm_min=34, tm_sec=5, tm_wday=6, tm_yday=53, tm_isdst=-1) <= time.struct_time(tm_year=2026, tm_mon=2, tm_mday=22, tm_hour=22, tm_min=38, tm_sec=39, tm_wday=6, tm_yday=53, tm_isdst=0) 2026-02-22T22:38:41.164 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2026-02-22T22:39:01.165 DEBUG:teuthology.orchestra.run.trial023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-22T22:39:01.317 INFO:teuthology.orchestra.run.trial023.stdout: 2026-02-22T22:39:01.317 INFO:teuthology.orchestra.run.trial023.stderr:dumped all 2026-02-22T22:39:01.329 INFO:teuthology.orchestra.run.trial023.stdout:{"pg_ready":true,"pg_map":{"version":84,"stamp":"2026-02-22T22:38:59.338413+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":81,"num_write_kb":1108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":466956,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":669116,"kb_used_data":1036,"kb_used_omap":13,"kb_used_meta":668018,"kb_avail":282446404,"statfs":{"total":289910292480,"available":289225117696,"internally_reserved":0,"allocated":1060864,"data_stored":1277075,"data_compressed":6753,"data_compressed_allocated":57344,"data_compressed_original":905216,"omap_allocated":13317,"internal_metadata":684051451},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.000949"},"pg_stats":[{"pgid":"1.0","version":"0'0","reported_seq":65,"reported_epoch":37,"state":"active+clean","last_fresh":"2026-02-22T22:38:42.264772+0000","last_change":"2026-02-22T22:38:42.264772+0000","last_active":"2026-02-22T22:38:42.264772+0000","last_peered":"2026-02-22T22:38:42.264772+0000","last_clean":"2026-02-22T22:38:42.264772+0000","last_became_active":"2026-02-22T22:36:52.342090+0000","last_became_peered":"2026-02-22T22:36:52.342090+0000","last_unstale":"2026-02-22T22:38:42.264772+0000","last_undegraded":"2026-02-22T22:38:42.264772+0000","last_fullsized":"2026-02-22T22:38:42.264772+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-22T22:38:42.264669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-22T22:38:42.264669+0000","last_clean_scrub_stamp":"2026-02-22T22:38:42.264669+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-24T06:04:25.619182+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00028225400000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":55,"reported_epoch":37,"state":"active+clean","last_fresh":"2026-02-22T22:38:42.126502+0000","last_change":"2026-02-22T22:38:42.126528+0000","last_active":"2026-02-22T22:38:42.126502+0000","last_peered":"2026-02-22T22:38:42.126502+0000","last_clean":"2026-02-22T22:38:42.126502+0000","last_became_active":"2026-02-22T22:37:01.433636+0000","last_became_peered":"2026-02-22T22:37:01.433636+0000","last_unstale":"2026-02-22T22:38:42.126502+0000","last_undegraded":"2026-02-22T22:38:42.126502+0000","last_fullsized":"2026-02-22T22:38:42.126502+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-22T22:38:42.126390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-22T22:38:42.126390+0000","last_clean_scrub_stamp":"2026-02-22T22:38:42.126390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-24T00:18:46.320903+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"37'3","reported_seq":72,"reported_epoch":37,"state":"active+clean","last_fresh":"2026-02-22T22:38:40.637575+0000","last_change":"2026-02-22T22:38:40.637575+0000","last_active":"2026-02-22T22:38:40.637575+0000","last_peered":"2026-02-22T22:38:40.637575+0000","last_clean":"2026-02-22T22:38:40.637575+0000","last_became_active":"2026-02-22T22:36:52.342588+0000","last_became_peered":"2026-02-22T22:36:52.342588+0000","last_unstale":"2026-02-22T22:38:40.637575+0000","last_undegraded":"2026-02-22T22:38:40.637575+0000","last_fullsized":"2026-02-22T22:38:40.637575+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'3","last_scrub_stamp":"2026-02-22T22:38:40.637508+0000","last_deep_scrub":"37'3","last_deep_scrub_stamp":"2026-02-22T22:38:40.637508+0000","last_clean_scrub_stamp":"2026-02-22T22:38:40.637508+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2026-02-24T09:44:29.031366+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.00064906600000000001,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"37'54","reported_seq":114,"reported_epoch":37,"state":"active+clean","last_fresh":"2026-02-22T22:38:46.377268+0000","last_change":"2026-02-22T22:38:46.377268+0000","last_active":"2026-02-22T22:38:46.377268+0000","last_peered":"2026-02-22T22:38:46.377268+0000","last_clean":"2026-02-22T22:38:46.377268+0000","last_became_active":"2026-02-22T22:37:01.435620+0000","last_became_peered":"2026-02-22T22:37:01.435620+0000","last_unstale":"2026-02-22T22:38:46.377268+0000","last_undegraded":"2026-02-22T22:38:46.377268+0000","last_fullsized":"2026-02-22T22:38:46.377268+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'54","last_scrub_stamp":"2026-02-22T22:38:46.377179+0000","last_deep_scrub":"37'54","last_deep_scrub_stamp":"2026-02-22T22:38:46.377179+0000","last_clean_scrub_stamp":"2026-02-22T22:38:46.377179+0000","objects_scrubbed":2,"log_size":54,"log_dups_size":0,"ondisk_log_size":54,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2026-02-24T05:55:46.630364+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":466956,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.3","version":"37'2","reported_seq":69,"reported_epoch":37,"state":"active+clean","last_fresh":"2026-02-22T22:38:45.352154+0000","last_change":"2026-02-22T22:38:45.352154+0000","last_active":"2026-02-22T22:38:45.352154+0000","last_peered":"2026-02-22T22:38:45.352154+0000","last_clean":"2026-02-22T22:38:45.352154+0000","last_became_active":"2026-02-22T22:37:01.434794+0000","last_became_peered":"2026-02-22T22:37:01.434794+0000","last_unstale":"2026-02-22T22:38:45.352154+0000","last_undegraded":"2026-02-22T22:38:45.352154+0000","last_fullsized":"2026-02-22T22:38:45.352154+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'2","last_scrub_stamp":"2026-02-22T22:38:45.352061+0000","last_deep_scrub":"37'2","last_deep_scrub_stamp":"2026-02-22T22:38:45.352061+0000","last_clean_scrub_stamp":"2026-02-22T22:38:45.352061+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2026-02-24T00:29:03.149744+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.00044206200000000001,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":65,"reported_epoch":37,"state":"active+clean","last_fresh":"2026-02-22T22:38:43.308825+0000","last_change":"2026-02-22T22:38:43.308825+0000","last_active":"2026-02-22T22:38:43.308825+0000","last_peered":"2026-02-22T22:38:43.308825+0000","last_clean":"2026-02-22T22:38:43.308825+0000","last_became_active":"2026-02-22T22:36:52.342463+0000","last_became_peered":"2026-02-22T22:36:52.342463+0000","last_unstale":"2026-02-22T22:38:43.308825+0000","last_undegraded":"2026-02-22T22:38:43.308825+0000","last_fullsized":"2026-02-22T22:38:43.308825+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-22T22:38:43.308746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-22T22:38:43.308746+0000","last_clean_scrub_stamp":"2026-02-22T22:38:43.308746+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-24T08:59:22.906955+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000308413,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":55,"reported_epoch":37,"state":"active+clean","last_fresh":"2026-02-22T22:38:41.150799+0000","last_change":"2026-02-22T22:38:41.150820+0000","last_active":"2026-02-22T22:38:41.150799+0000","last_peered":"2026-02-22T22:38:41.150799+0000","last_clean":"2026-02-22T22:38:41.150799+0000","last_became_active":"2026-02-22T22:37:01.433380+0000","last_became_peered":"2026-02-22T22:37:01.433380+0000","last_unstale":"2026-02-22T22:38:41.150799+0000","last_undegraded":"2026-02-22T22:38:41.150799+0000","last_fullsized":"2026-02-22T22:38:41.150799+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-22T22:38:41.150714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-22T22:38:41.150714+0000","last_clean_scrub_stamp":"2026-02-22T22:38:41.150714+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-24T06:36:28.031507+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":65,"reported_epoch":37,"state":"active+clean","last_fresh":"2026-02-22T22:38:41.251983+0000","last_change":"2026-02-22T22:38:41.251983+0000","last_active":"2026-02-22T22:38:41.251983+0000","last_peered":"2026-02-22T22:38:41.251983+0000","last_clean":"2026-02-22T22:38:41.251983+0000","last_became_active":"2026-02-22T22:36:52.342082+0000","last_became_peered":"2026-02-22T22:36:52.342082+0000","last_unstale":"2026-02-22T22:38:41.251983+0000","last_undegraded":"2026-02-22T22:38:41.251983+0000","last_fullsized":"2026-02-22T22:38:41.251983+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-22T22:38:41.251895+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-22T22:38:41.251895+0000","last_clean_scrub_stamp":"2026-02-22T22:38:41.251895+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-24T07:10:23.782139+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00060176900000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":67,"reported_epoch":37,"state":"active+clean","last_fresh":"2026-02-22T22:38:44.308130+0000","last_change":"2026-02-22T22:38:44.308130+0000","last_active":"2026-02-22T22:38:44.308130+0000","last_peered":"2026-02-22T22:38:44.308130+0000","last_clean":"2026-02-22T22:38:44.308130+0000","last_became_active":"2026-02-22T22:37:01.434745+0000","last_became_peered":"2026-02-22T22:37:01.434745+0000","last_unstale":"2026-02-22T22:38:44.308130+0000","last_undegraded":"2026-02-22T22:38:44.308130+0000","last_fullsized":"2026-02-22T22:38:44.308130+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-22T22:38:44.308049+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-22T22:38:44.308049+0000","last_clean_scrub_stamp":"2026-02-22T22:38:44.308049+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2026-02-24T03:59:49.395013+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.00045264099999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":466956,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":94208,"data_stored":933912,"data_compressed":6753,"data_compressed_allocated":57344,"data_compressed_original":905216,"omap_allocated":0,"internal_metadata":0},"log_size":54,"ondisk_log_size":54,"up":2,"acting":2,"num_store_stats":2},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":5,"ondisk_log_size":5,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":32,"seq":137438953496,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":177460,"kb_used_data":356,"kb_used_omap":4,"kb_used_meta":177083,"kb_avail":94194380,"statfs":{"total":96636764160,"available":96455045120,"internally_reserved":0,"allocated":364544,"data_stored":581331,"data_compressed":3420,"data_compressed_allocated":28672,"data_compressed_original":454656,"omap_allocated":4438,"internal_metadata":181333674},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084315,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":315260,"kb_used_data":364,"kb_used_omap":4,"kb_used_meta":314875,"kb_avail":94056580,"statfs":{"total":96636764160,"available":96313937920,"internally_reserved":0,"allocated":372736,"data_stored":581350,"data_compressed":3333,"data_compressed_allocated":28672,"data_compressed_original":450560,"omap_allocated":4439,"internal_metadata":322432681},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247836,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":176396,"kb_used_data":316,"kb_used_omap":4,"kb_used_meta":176059,"kb_avail":94195444,"statfs":{"total":96636764160,"available":96456134656,"internally_reserved":0,"allocated":323584,"data_stored":114394,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4440,"internal_metadata":180285096},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":466956,"data_compressed":3333,"data_compressed_allocated":28672,"data_compressed_original":450560,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":466956,"data_compressed":3420,"data_compressed_allocated":28672,"data_compressed_original":454656,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-22T22:39:01.331 DEBUG:teuthology.orchestra.run.trial023:> sudo ceph --cluster ceph config set global mon_health_to_clog false 2026-02-22T22:39:01.491 INFO:teuthology.misc:Shutting down mds daemons... 2026-02-22T22:39:01.492 INFO:teuthology.misc:Shutting down osd daemons... 2026-02-22T22:39:01.492 DEBUG:tasks.ceph.osd.0:waiting for process to exit 2026-02-22T22:39:01.492 INFO:teuthology.orchestra.run:waiting for 300 2026-02-22T22:39:01.589 INFO:tasks.ceph.osd.0:Stopped 2026-02-22T22:39:01.590 DEBUG:tasks.ceph.osd.1:waiting for process to exit 2026-02-22T22:39:01.590 INFO:teuthology.orchestra.run:waiting for 300 2026-02-22T22:39:06.391 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~0s 2026-02-22T22:39:07.691 INFO:tasks.ceph.osd.1:Stopped 2026-02-22T22:39:07.692 DEBUG:tasks.ceph.osd.2:waiting for process to exit 2026-02-22T22:39:07.692 INFO:teuthology.orchestra.run:waiting for 300 2026-02-22T22:39:07.769 INFO:tasks.ceph.osd.2:Stopped 2026-02-22T22:39:07.770 INFO:teuthology.misc:Shutting down mgr daemons... 2026-02-22T22:39:07.770 DEBUG:tasks.ceph.mgr.x:waiting for process to exit 2026-02-22T22:39:07.770 INFO:teuthology.orchestra.run:waiting for 300 2026-02-22T22:39:07.806 INFO:tasks.ceph.mgr.x:Stopped 2026-02-22T22:39:07.806 INFO:teuthology.misc:Shutting down mon daemons... 2026-02-22T22:39:07.806 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2026-02-22T22:39:07.807 INFO:teuthology.orchestra.run:waiting for 300 2026-02-22T22:39:07.860 INFO:tasks.ceph.mon.a:Stopped 2026-02-22T22:39:07.860 DEBUG:tasks.ceph.mon.b:waiting for process to exit 2026-02-22T22:39:07.860 INFO:teuthology.orchestra.run:waiting for 300 2026-02-22T22:39:07.916 INFO:tasks.ceph.mon.b:Stopped 2026-02-22T22:39:07.916 DEBUG:tasks.ceph.mon.c:waiting for process to exit 2026-02-22T22:39:07.916 INFO:teuthology.orchestra.run:waiting for 300 2026-02-22T22:39:07.972 INFO:tasks.ceph.mon.c:Stopped 2026-02-22T22:39:07.972 INFO:tasks.ceph:Checking cluster log for badness... 2026-02-22T22:39:07.973 DEBUG:teuthology.orchestra.run.trial023:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/ceph.log | egrep -v 'no reply from' | egrep -v 'overall HEALTH_' | egrep -v '\(MON_DOWN\)' | egrep -v '\(MGR_DOWN\)' | egrep -v '\(OSDMAP_FLAGS\)' | egrep -v '\(OSD_' | egrep -v '\(PG_' | egrep -v '\(POOL_APP_NOT_ENABLED\)' | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(OSD_SLOW_PING_TIME' | egrep -v '\(MON_DOWN\)' | head -n 1 2026-02-22T22:39:08.025 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-0 on ubuntu@trial023.front.sepia.ceph.com 2026-02-22T22:39:08.025 DEBUG:teuthology.orchestra.run.trial023:> sync && sudo umount -f /var/lib/ceph/osd/ceph-0 2026-02-22T22:39:08.095 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-1 on ubuntu@trial023.front.sepia.ceph.com 2026-02-22T22:39:08.096 DEBUG:teuthology.orchestra.run.trial023:> sync && sudo umount -f /var/lib/ceph/osd/ceph-1 2026-02-22T22:39:08.165 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-2 on ubuntu@trial023.front.sepia.ceph.com 2026-02-22T22:39:08.165 DEBUG:teuthology.orchestra.run.trial023:> sync && sudo umount -f /var/lib/ceph/osd/ceph-2 2026-02-22T22:39:08.235 INFO:tasks.ceph:Archiving mon data... 2026-02-22T22:39:08.237 DEBUG:teuthology.misc:Transferring archived files from trial023:/var/lib/ceph/mon/ceph-a to /home/teuthworker/mnt/teuthology/yuriw-2026-02-22_19:35:40-rados-wip-yuri6-testing-2026-02-22-1629-squid-distro-default-trial/64252/data/mon.a.tgz 2026-02-22T22:39:08.237 DEBUG:teuthology.orchestra.run.trial023:> mktemp 2026-02-22T22:39:08.280 INFO:teuthology.orchestra.run.trial023.stdout:/tmp/tmp.dflwaWUkDT 2026-02-22T22:39:08.280 DEBUG:teuthology.orchestra.run.trial023:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-a -- . > /tmp/tmp.dflwaWUkDT 2026-02-22T22:39:08.361 DEBUG:teuthology.orchestra.run.trial023:> sudo chmod 0666 /tmp/tmp.dflwaWUkDT 2026-02-22T22:39:08.462 DEBUG:teuthology.orchestra.remote:trial023:/tmp/tmp.dflwaWUkDT is 195KB 2026-02-22T22:39:08.474 DEBUG:teuthology.orchestra.run.trial023:> rm -fr /tmp/tmp.dflwaWUkDT 2026-02-22T22:39:08.478 DEBUG:teuthology.misc:Transferring archived files from trial023:/var/lib/ceph/mon/ceph-b to /home/teuthworker/mnt/teuthology/yuriw-2026-02-22_19:35:40-rados-wip-yuri6-testing-2026-02-22-1629-squid-distro-default-trial/64252/data/mon.b.tgz 2026-02-22T22:39:08.478 DEBUG:teuthology.orchestra.run.trial023:> mktemp 2026-02-22T22:39:08.528 INFO:teuthology.orchestra.run.trial023.stdout:/tmp/tmp.DPTpQ8ikBP 2026-02-22T22:39:08.529 DEBUG:teuthology.orchestra.run.trial023:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-b -- . > /tmp/tmp.DPTpQ8ikBP 2026-02-22T22:39:08.620 DEBUG:teuthology.orchestra.run.trial023:> sudo chmod 0666 /tmp/tmp.DPTpQ8ikBP 2026-02-22T22:39:08.677 DEBUG:teuthology.orchestra.remote:trial023:/tmp/tmp.DPTpQ8ikBP is 259KB 2026-02-22T22:39:08.733 DEBUG:teuthology.orchestra.run.trial023:> rm -fr /tmp/tmp.DPTpQ8ikBP 2026-02-22T22:39:08.738 DEBUG:teuthology.misc:Transferring archived files from trial023:/var/lib/ceph/mon/ceph-c to /home/teuthworker/mnt/teuthology/yuriw-2026-02-22_19:35:40-rados-wip-yuri6-testing-2026-02-22-1629-squid-distro-default-trial/64252/data/mon.c.tgz 2026-02-22T22:39:08.738 DEBUG:teuthology.orchestra.run.trial023:> mktemp 2026-02-22T22:39:08.783 INFO:teuthology.orchestra.run.trial023.stdout:/tmp/tmp.aufTCvOJFt 2026-02-22T22:39:08.784 DEBUG:teuthology.orchestra.run.trial023:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-c -- . > /tmp/tmp.aufTCvOJFt 2026-02-22T22:39:08.872 DEBUG:teuthology.orchestra.run.trial023:> sudo chmod 0666 /tmp/tmp.aufTCvOJFt 2026-02-22T22:39:08.969 DEBUG:teuthology.orchestra.remote:trial023:/tmp/tmp.aufTCvOJFt is 259KB 2026-02-22T22:39:09.025 DEBUG:teuthology.orchestra.run.trial023:> rm -fr /tmp/tmp.aufTCvOJFt 2026-02-22T22:39:09.030 INFO:tasks.ceph:Cleaning ceph cluster... 2026-02-22T22:39:09.030 DEBUG:teuthology.orchestra.run.trial023:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2026-02-22T22:39:09.127 INFO:teuthology.util.scanner:summary_data or yaml_file is empty! 2026-02-22T22:39:09.128 INFO:tasks.ceph:Archiving crash dumps... 2026-02-22T22:39:09.130 DEBUG:teuthology.misc:Transferring archived files from trial023:/var/lib/ceph/crash to /home/teuthworker/mnt/teuthology/yuriw-2026-02-22_19:35:40-rados-wip-yuri6-testing-2026-02-22-1629-squid-distro-default-trial/64252/remote/trial023/crash 2026-02-22T22:39:09.131 DEBUG:teuthology.orchestra.run.trial023:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2026-02-22T22:39:09.175 INFO:tasks.ceph:Compressing logs... 2026-02-22T22:39:09.176 DEBUG:teuthology.orchestra.run.trial023:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2026-02-22T22:39:09.228 INFO:teuthology.orchestra.run.trial023.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.19726.log 2026-02-22T22:39:09.229 INFO:teuthology.orchestra.run.trial023.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.19363.log 2026-02-22T22:39:09.229 INFO:teuthology.orchestra.run.trial023.stderr:/var/log/ceph/ceph-client.admin.19726.log: gzip -5 --verbose -- /var/log/ceph/ceph-mon.b.log 2026-02-22T22:39:09.229 INFO:teuthology.orchestra.run.trial023.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.19726.log.gz 2026-02-22T22:39:09.229 INFO:teuthology.orchestra.run.trial023.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.25326.log 2026-02-22T22:39:09.230 INFO:teuthology.orchestra.run.trial023.stderr:/var/log/ceph/ceph-client.admin.19363.log: /var/log/ceph/ceph-mon.b.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.19363.log.gz 2026-02-22T22:39:09.230 INFO:teuthology.orchestra.run.trial023.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.25144.log 2026-02-22T22:39:09.235 INFO:teuthology.orchestra.run.trial023.stderr:/var/log/ceph/ceph-client.admin.25326.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.0.19642.log 2026-02-22T22:39:09.235 INFO:teuthology.orchestra.run.trial023.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.25326.log.gz 2026-02-22T22:39:09.236 INFO:teuthology.orchestra.run.trial023.stderr:/var/log/ceph/ceph-client.admin.25144.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.19076.log 2026-02-22T22:39:09.236 INFO:teuthology.orchestra.run.trial023.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.25144.log.gz 2026-02-22T22:39:09.236 INFO:teuthology.orchestra.run.trial023.stderr:/var/log/ceph/ceph-client.0.19642.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.19277.log 2026-02-22T22:39:09.236 INFO:teuthology.orchestra.run.trial023.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.19563.log 2026-02-22T22:39:09.236 INFO:teuthology.orchestra.run.trial023.stderr:/var/log/ceph/ceph-client.admin.19076.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.19076.log.gz 2026-02-22T22:39:09.236 INFO:teuthology.orchestra.run.trial023.stderr:/var/log/ceph/ceph-client.admin.19277.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.19277.log.gz 2026-02-22T22:39:09.236 INFO:teuthology.orchestra.run.trial023.stderr: 96.4% -- replaced with /var/log/ceph/ceph-client.0.19642.log.gz 2026-02-22T22:39:09.236 INFO:teuthology.orchestra.run.trial023.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.25222.log 2026-02-22T22:39:09.236 INFO:teuthology.orchestra.run.trial023.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.19433.log 2026-02-22T22:39:09.236 INFO:teuthology.orchestra.run.trial023.stderr:/var/log/ceph/ceph-client.admin.19563.log: /var/log/ceph/ceph-client.admin.25222.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.19563.log.gz 2026-02-22T22:39:09.236 INFO:teuthology.orchestra.run.trial023.stderr:gzip -5 0.0% --verbose -- -- replaced with /var/log/ceph/ceph-client.admin.25222.log.gz /var/log/ceph/ceph-client.admin.19752.log 2026-02-22T22:39:09.237 INFO:teuthology.orchestra.run.trial023.stderr: 2026-02-22T22:39:09.237 INFO:teuthology.orchestra.run.trial023.stderr:/var/log/ceph/ceph-client.admin.19433.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.25352.log 2026-02-22T22:39:09.237 INFO:teuthology.orchestra.run.trial023.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.19433.log.gz 2026-02-22T22:39:09.237 INFO:teuthology.orchestra.run.trial023.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.19537.log 2026-02-22T22:39:09.237 INFO:teuthology.orchestra.run.trial023.stderr:/var/log/ceph/ceph-client.admin.19752.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.19752.log.gz/var/log/ceph/ceph-client.admin.25352.log: 2026-02-22T22:39:09.237 INFO:teuthology.orchestra.run.trial023.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.16317.log 2026-02-22T22:39:09.237 INFO:teuthology.orchestra.run.trial023.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.25352.log.gz 2026-02-22T22:39:09.237 INFO:teuthology.orchestra.run.trial023.stderr:/var/log/ceph/ceph-client.admin.19537.log: gzip -5 --verbose -- /var/log/ceph/ceph-mgr.x.log 2026-02-22T22:39:09.237 INFO:teuthology.orchestra.run.trial023.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.19537.log.gz 2026-02-22T22:39:09.237 INFO:teuthology.orchestra.run.trial023.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.19511.log 2026-02-22T22:39:09.238 INFO:teuthology.orchestra.run.trial023.stderr:/var/log/ceph/ceph-client.admin.16317.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.16317.log.gz 2026-02-22T22:39:09.238 INFO:teuthology.orchestra.run.trial023.stderr:/var/log/ceph/ceph-mgr.x.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.19677.log 2026-02-22T22:39:09.238 INFO:teuthology.orchestra.run.trial023.stderr:/var/log/ceph/ceph-client.admin.19511.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.19205.log 2026-02-22T22:39:09.238 INFO:teuthology.orchestra.run.trial023.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.19511.log.gz 2026-02-22T22:39:09.238 INFO:teuthology.orchestra.run.trial023.stderr:/var/log/ceph/ceph-client.admin.19677.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.19677.log.gz 2026-02-22T22:39:09.238 INFO:teuthology.orchestra.run.trial023.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.19589.log 2026-02-22T22:39:09.238 INFO:teuthology.orchestra.run.trial023.stderr:/var/log/ceph/ceph-client.admin.19205.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.0.25083.log 2026-02-22T22:39:09.238 INFO:teuthology.orchestra.run.trial023.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.19205.log.gz 2026-02-22T22:39:09.238 INFO:teuthology.orchestra.run.trial023.stderr:/var/log/ceph/ceph-client.admin.19589.log: gzip -5 --verbose -- /var/log/ceph/ceph-mon.a.log 2026-02-22T22:39:09.238 INFO:teuthology.orchestra.run.trial023.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.19589.log.gz 2026-02-22T22:39:09.238 INFO:teuthology.orchestra.run.trial023.stderr:/var/log/ceph/ceph-client.0.25083.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.19207.log 2026-02-22T22:39:09.239 INFO:teuthology.orchestra.run.trial023.stderr:/var/log/ceph/ceph-mon.a.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.19051.log 2026-02-22T22:39:09.239 INFO:teuthology.orchestra.run.trial023.stderr:/var/log/ceph/ceph-client.admin.19207.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.19207.log.gz 2026-02-22T22:39:09.239 INFO:teuthology.orchestra.run.trial023.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.25196.log 2026-02-22T22:39:09.239 INFO:teuthology.orchestra.run.trial023.stderr: 96.5% -- replaced with /var/log/ceph/ceph-client.0.25083.log.gz 2026-02-22T22:39:09.239 INFO:teuthology.orchestra.run.trial023.stderr:/var/log/ceph/ceph-client.admin.19051.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.25300.log 2026-02-22T22:39:09.239 INFO:teuthology.orchestra.run.trial023.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.19051.log.gz 2026-02-22T22:39:09.250 INFO:teuthology.orchestra.run.trial023.stderr:gzip -5 --verbose --/var/log/ceph/ceph-client.admin.25196.log: /var/log/ceph/ceph-client.admin.19355.log 2026-02-22T22:39:09.250 INFO:teuthology.orchestra.run.trial023.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.25196.log.gz 2026-02-22T22:39:09.251 INFO:teuthology.orchestra.run.trial023.stderr:/var/log/ceph/ceph-client.admin.25300.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.16365.log 2026-02-22T22:39:09.251 INFO:teuthology.orchestra.run.trial023.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.25300.log.gz 2026-02-22T22:39:09.251 INFO:teuthology.orchestra.run.trial023.stderr:/var/log/ceph/ceph-client.admin.19355.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.25274.log 2026-02-22T22:39:09.251 INFO:teuthology.orchestra.run.trial023.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.19355.log.gz 2026-02-22T22:39:09.251 INFO:teuthology.orchestra.run.trial023.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.19284.log 2026-02-22T22:39:09.251 INFO:teuthology.orchestra.run.trial023.stderr:/var/log/ceph/ceph-client.admin.16365.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.16365.log.gz 2026-02-22T22:39:09.251 INFO:teuthology.orchestra.run.trial023.stderr:/var/log/ceph/ceph-client.admin.25274.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.25404.log 2026-02-22T22:39:09.251 INFO:teuthology.orchestra.run.trial023.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.25274.log.gz 2026-02-22T22:39:09.251 INFO:teuthology.orchestra.run.trial023.stderr:/var/log/ceph/ceph-client.admin.19284.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.19462.log 2026-02-22T22:39:09.251 INFO:teuthology.orchestra.run.trial023.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.19284.log.gz 2026-02-22T22:39:09.251 INFO:teuthology.orchestra.run.trial023.stderr:/var/log/ceph/ceph-client.admin.25404.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.19206.log 2026-02-22T22:39:09.251 INFO:teuthology.orchestra.run.trial023.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.25404.log.gz 2026-02-22T22:39:09.252 INFO:teuthology.orchestra.run.trial023.stderr:/var/log/ceph/ceph-client.admin.19462.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.19147.log 2026-02-22T22:39:09.252 INFO:teuthology.orchestra.run.trial023.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.19462.log.gz 2026-02-22T22:39:09.252 INFO:teuthology.orchestra.run.trial023.stderr:/var/log/ceph/ceph-client.admin.19206.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.19173.log 2026-02-22T22:39:09.252 INFO:teuthology.orchestra.run.trial023.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.19206.log.gz 2026-02-22T22:39:09.252 INFO:teuthology.orchestra.run.trial023.stderr:/var/log/ceph/ceph-client.admin.19147.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.16119.log 2026-02-22T22:39:09.252 INFO:teuthology.orchestra.run.trial023.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.19147.log.gz 2026-02-22T22:39:09.252 INFO:teuthology.orchestra.run.trial023.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.25454.log 2026-02-22T22:39:09.252 INFO:teuthology.orchestra.run.trial023.stderr:/var/log/ceph/ceph-client.admin.19173.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.19173.log.gz 2026-02-22T22:39:09.252 INFO:teuthology.orchestra.run.trial023.stderr:/var/log/ceph/ceph-client.admin.16119.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.16119.log.gz 2026-02-22T22:39:09.252 INFO:teuthology.orchestra.run.trial023.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.16341.log 2026-02-22T22:39:09.252 INFO:teuthology.orchestra.run.trial023.stderr:/var/log/ceph/ceph-client.admin.25454.log: gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2026-02-22T22:39:09.252 INFO:teuthology.orchestra.run.trial023.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.25454.log.gz 2026-02-22T22:39:09.252 INFO:teuthology.orchestra.run.trial023.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.2.log 2026-02-22T22:39:09.253 INFO:teuthology.orchestra.run.trial023.stderr:/var/log/ceph/ceph-client.admin.16341.log: 0.0%/var/log/ceph/ceph.audit.log: -- replaced with /var/log/ceph/ceph-client.admin.16341.log.gz 2026-02-22T22:39:09.253 INFO:teuthology.orchestra.run.trial023.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.25118.log 2026-02-22T22:39:09.253 INFO:teuthology.orchestra.run.trial023.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mon.c.log 2026-02-22T22:39:09.253 INFO:teuthology.orchestra.run.trial023.stderr:/var/log/ceph/ceph-osd.2.log: /var/log/ceph/ceph-client.admin.25118.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.25118.log.gz 2026-02-22T22:39:09.253 INFO:teuthology.orchestra.run.trial023.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.25170.log 2026-02-22T22:39:09.253 INFO:teuthology.orchestra.run.trial023.stderr:/var/log/ceph/ceph-mon.c.log: 95.2% -- replaced with /var/log/ceph/ceph.audit.log.gz 2026-02-22T22:39:09.253 INFO:teuthology.orchestra.run.trial023.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.16400.log 2026-02-22T22:39:09.253 INFO:teuthology.orchestra.run.trial023.stderr:/var/log/ceph/ceph-client.admin.25170.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.25170.log.gz 2026-02-22T22:39:09.253 INFO:teuthology.orchestra.run.trial023.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.log 2026-02-22T22:39:09.253 INFO:teuthology.orchestra.run.trial023.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.19121.log 2026-02-22T22:39:09.253 INFO:teuthology.orchestra.run.trial023.stderr:/var/log/ceph/ceph-client.admin.16400.log: /var/log/ceph/ceph.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.16400.log.gz 2026-02-22T22:39:09.253 INFO:teuthology.orchestra.run.trial023.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.19101.log 2026-02-22T22:39:09.254 INFO:teuthology.orchestra.run.trial023.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.19437.log 2026-02-22T22:39:09.254 INFO:teuthology.orchestra.run.trial023.stderr:/var/log/ceph/ceph-client.admin.19121.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.19121.log.gz 2026-02-22T22:39:09.254 INFO:teuthology.orchestra.run.trial023.stderr:/var/log/ceph/ceph-client.admin.19101.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.16263.log 2026-02-22T22:39:09.254 INFO:teuthology.orchestra.run.trial023.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.19101.log.gz 2026-02-22T22:39:09.254 INFO:teuthology.orchestra.run.trial023.stderr:/var/log/ceph/ceph-client.admin.19437.log: 93.5% -- replaced with /var/log/ceph/ceph.log.gz 2026-02-22T22:39:09.254 INFO:teuthology.orchestra.run.trial023.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.19437.log.gz 2026-02-22T22:39:09.254 INFO:teuthology.orchestra.run.trial023.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.25248.log 2026-02-22T22:39:09.254 INFO:teuthology.orchestra.run.trial023.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.25378.log 2026-02-22T22:39:09.254 INFO:teuthology.orchestra.run.trial023.stderr:/var/log/ceph/ceph-client.admin.16263.log: /var/log/ceph/ceph-client.admin.25248.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.16263.log.gz 2026-02-22T22:39:09.254 INFO:teuthology.orchestra.run.trial023.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.25248.log.gz 2026-02-22T22:39:09.254 INFO:teuthology.orchestra.run.trial023.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.19027.log 2026-02-22T22:39:09.254 INFO:teuthology.orchestra.run.trial023.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.0.log 2026-02-22T22:39:09.254 INFO:teuthology.orchestra.run.trial023.stderr:/var/log/ceph/ceph-client.admin.25378.log: /var/log/ceph/ceph-client.admin.19027.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.19027.log.gz 2026-02-22T22:39:09.255 INFO:teuthology.orchestra.run.trial023.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.19700.log 2026-02-22T22:39:09.255 INFO:teuthology.orchestra.run.trial023.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.25378.log.gz 2026-02-22T22:39:09.255 INFO:teuthology.orchestra.run.trial023.stderr:/var/log/ceph/ceph-osd.0.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.22278.log 2026-02-22T22:39:09.255 INFO:teuthology.orchestra.run.trial023.stderr:/var/log/ceph/ceph-client.admin.19700.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.19700.log.gz 2026-02-22T22:39:09.255 INFO:teuthology.orchestra.run.trial023.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.19285.log 2026-02-22T22:39:09.255 INFO:teuthology.orchestra.run.trial023.stderr:/var/log/ceph/ceph-client.admin.22278.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.25059.log 2026-02-22T22:39:09.255 INFO:teuthology.orchestra.run.trial023.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.22278.log.gz 2026-02-22T22:39:09.255 INFO:teuthology.orchestra.run.trial023.stderr:gzip/var/log/ceph/ceph-client.admin.19285.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.19362.log 2026-02-22T22:39:09.255 INFO:teuthology.orchestra.run.trial023.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.19285.log.gz 2026-02-22T22:39:09.255 INFO:teuthology.orchestra.run.trial023.stderr:/var/log/ceph/ceph-client.admin.25059.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.25059.log.gz 2026-02-22T22:39:09.255 INFO:teuthology.orchestra.run.trial023.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.1.log 2026-02-22T22:39:09.255 INFO:teuthology.orchestra.run.trial023.stderr:/var/log/ceph/ceph-client.admin.19362.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.19618.log 2026-02-22T22:39:09.255 INFO:teuthology.orchestra.run.trial023.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.19362.log.gz 2026-02-22T22:39:09.255 INFO:teuthology.orchestra.run.trial023.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.25030.log 2026-02-22T22:39:09.256 INFO:teuthology.orchestra.run.trial023.stderr:/var/log/ceph/ceph-osd.1.log: gzip -5 --verbose -- /var/log/ceph/ceph.tmp-client.admin.12481.log 2026-02-22T22:39:09.256 INFO:teuthology.orchestra.run.trial023.stderr:/var/log/ceph/ceph-client.admin.19618.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.19618.log.gz 2026-02-22T22:39:09.256 INFO:teuthology.orchestra.run.trial023.stderr:/var/log/ceph/ceph-client.admin.25030.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.25430.log 2026-02-22T22:39:09.256 INFO:teuthology.orchestra.run.trial023.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.25030.log.gz 2026-02-22T22:39:09.256 INFO:teuthology.orchestra.run.trial023.stderr:/var/log/ceph/ceph.tmp-client.admin.12481.log: 0.0% -- replaced with /var/log/ceph/ceph.tmp-client.admin.12481.log.gz 2026-02-22T22:39:09.256 INFO:teuthology.orchestra.run.trial023.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.16464.log 2026-02-22T22:39:09.256 INFO:teuthology.orchestra.run.trial023.stderr:/var/log/ceph/ceph-client.admin.25430.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.25430.log.gz 2026-02-22T22:39:09.256 INFO:teuthology.orchestra.run.trial023.stderr:/var/log/ceph/ceph-client.admin.16464.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.16464.log.gz 2026-02-22T22:39:09.282 INFO:teuthology.orchestra.run.trial023.stderr: 91.6% -- replaced with /var/log/ceph/ceph-mon.c.log.gz 2026-02-22T22:39:09.291 INFO:teuthology.orchestra.run.trial023.stderr: 91.3% -- replaced with /var/log/ceph/ceph-mon.b.log.gz 2026-02-22T22:39:09.327 INFO:teuthology.orchestra.run.trial023.stderr: 90.4% -- replaced with /var/log/ceph/ceph-mon.a.log.gz 2026-02-22T22:39:09.337 INFO:teuthology.orchestra.run.trial023.stderr: 96.5% -- replaced with /var/log/ceph/ceph-mgr.x.log.gz 2026-02-22T22:39:11.392 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 has been restored 2026-02-22T22:39:55.339 INFO:teuthology.orchestra.run.trial023.stderr: 91.2% -- replaced with /var/log/ceph/ceph-osd.2.log.gz 2026-02-22T22:40:05.341 INFO:teuthology.orchestra.run.trial023.stderr: 91.2% -- replaced with /var/log/ceph/ceph-osd.0.log.gz 2026-02-22T22:40:28.777 INFO:teuthology.orchestra.run.trial023.stderr: 91.2% -- replaced with /var/log/ceph/ceph-osd.1.log.gz 2026-02-22T22:40:28.778 INFO:teuthology.orchestra.run.trial023.stderr: 2026-02-22T22:40:28.778 INFO:teuthology.orchestra.run.trial023.stderr:real 1m19.556s 2026-02-22T22:40:28.778 INFO:teuthology.orchestra.run.trial023.stderr:user 2m55.253s 2026-02-22T22:40:28.779 INFO:teuthology.orchestra.run.trial023.stderr:sys 0m6.795s 2026-02-22T22:40:28.780 INFO:tasks.ceph:Archiving logs... 2026-02-22T22:40:28.780 DEBUG:teuthology.misc:Transferring archived files from trial023:/var/log/ceph to /home/teuthworker/mnt/teuthology/yuriw-2026-02-22_19:35:40-rados-wip-yuri6-testing-2026-02-22-1629-squid-distro-default-trial/64252/remote/trial023/log 2026-02-22T22:40:28.782 DEBUG:teuthology.orchestra.run.trial023:> sudo tar c -f - -C /var/log/ceph -- . 2026-02-22T22:40:44.159 DEBUG:teuthology.run_tasks:Unwinding manager install 2026-02-22T22:40:44.167 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2026-02-22T22:40:44.167 DEBUG:teuthology.orchestra.run.trial023:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2026-02-22T22:40:44.255 DEBUG:teuthology.run_tasks:Unwinding manager clock 2026-02-22T22:40:44.266 INFO:teuthology.task.clock:Checking final clock skew... 2026-02-22T22:40:44.266 DEBUG:teuthology.orchestra.run.trial023:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2026-02-22T22:40:44.299 INFO:teuthology.orchestra.run.trial023.stdout: remote refid st t when poll reach delay offset jitter 2026-02-22T22:40:44.299 INFO:teuthology.orchestra.run.trial023.stdout:============================================================================== 2026-02-22T22:40:44.299 INFO:teuthology.orchestra.run.trial023.stdout:+nyc3.us.ntp.li 17.253.2.35 2 u 56 64 177 3.245 -3.203 2.081 2026-02-22T22:40:44.299 INFO:teuthology.orchestra.run.trial023.stdout:-ntp.maxhost.io 132.163.97.4 2 u 57 64 127 69.743 -0.669 1.769 2026-02-22T22:40:44.299 INFO:teuthology.orchestra.run.trial023.stdout:*LAX.CALTICK.NET 17.253.83.251 2 u 59 64 177 63.023 +0.621 3.728 2026-02-22T22:40:44.299 INFO:teuthology.orchestra.run.trial023.stdout:+24-111-79-186-d 216.197.156.83 2 u 58 64 177 52.937 -0.957 1.184 2026-02-22T22:40:44.300 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2026-02-22T22:40:44.311 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2026-02-22T22:40:44.311 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2026-02-22T22:40:44.325 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2026-02-22T22:40:44.352 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2026-02-22T22:40:44.365 INFO:teuthology.task.internal:Duration was 601.839124 seconds 2026-02-22T22:40:44.365 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2026-02-22T22:40:44.379 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2026-02-22T22:40:44.379 DEBUG:teuthology.orchestra.run.trial023:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2026-02-22T22:40:44.403 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2026-02-22T22:40:44.403 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@trial023.front.sepia.ceph.com 2026-02-22T22:40:44.403 DEBUG:teuthology.orchestra.run.trial023:> grep -E --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | grep -E -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | grep -E -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2026-02-22T22:40:44.450 INFO:teuthology.task.internal.syslog:Gathering journactl... 2026-02-22T22:40:44.450 DEBUG:teuthology.orchestra.run.trial023:> sudo journalctl > /home/ubuntu/cephtest/archive/syslog/journalctl.log 2026-02-22T22:40:44.710 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2026-02-22T22:40:44.710 DEBUG:teuthology.orchestra.run.trial023:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --max-args=1 --max-procs=0 --verbose --no-run-if-empty -- gzip -5 --verbose -- 2026-02-22T22:40:44.719 INFO:teuthology.orchestra.run.trial023.stderr:gzip -5 --verbose -- /home/ubuntu/cephtest/archive/syslog/misc.log 2026-02-22T22:40:44.720 INFO:teuthology.orchestra.run.trial023.stderr:gzip -5 --verbose -- /home/ubuntu/cephtest/archive/syslog/journalctl.log 2026-02-22T22:40:44.720 INFO:teuthology.orchestra.run.trial023.stderr:/home/ubuntu/cephtest/archive/syslog/misc.log: 0.0% -- replaced with /home/ubuntu/cephtest/archive/syslog/misc.log.gz 2026-02-22T22:40:44.720 INFO:teuthology.orchestra.run.trial023.stderr:gzip -5 --verbose -- /home/ubuntu/cephtest/archive/syslog/kern.log 2026-02-22T22:40:44.721 INFO:teuthology.orchestra.run.trial023.stderr:/home/ubuntu/cephtest/archive/syslog/journalctl.log: /home/ubuntu/cephtest/archive/syslog/kern.log: 0.0% -- replaced with /home/ubuntu/cephtest/archive/syslog/kern.log.gz 2026-02-22T22:40:44.787 INFO:teuthology.orchestra.run.trial023.stderr: 81.5% -- replaced with /home/ubuntu/cephtest/archive/syslog/journalctl.log.gz 2026-02-22T22:40:44.789 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2026-02-22T22:40:44.805 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2026-02-22T22:40:44.806 DEBUG:teuthology.orchestra.run.trial023:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2026-02-22T22:40:44.840 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2026-02-22T22:40:44.856 DEBUG:teuthology.orchestra.run.trial023:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2026-02-22T22:40:44.892 INFO:teuthology.orchestra.run.trial023.stdout:kernel.core_pattern = core 2026-02-22T22:40:44.900 DEBUG:teuthology.orchestra.run.trial023:> test -e /home/ubuntu/cephtest/archive/coredump 2026-02-22T22:40:44.947 DEBUG:teuthology.orchestra.run:got remote process result: 1 2026-02-22T22:40:44.948 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2026-02-22T22:40:44.964 INFO:teuthology.task.internal:Transferring archived files... 2026-02-22T22:40:44.965 DEBUG:teuthology.misc:Transferring archived files from trial023:/home/ubuntu/cephtest/archive to /home/teuthworker/mnt/teuthology/yuriw-2026-02-22_19:35:40-rados-wip-yuri6-testing-2026-02-22-1629-squid-distro-default-trial/64252/remote/trial023 2026-02-22T22:40:44.965 DEBUG:teuthology.orchestra.run.trial023:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2026-02-22T22:40:45.030 INFO:teuthology.task.internal:Removing archive directory... 2026-02-22T22:40:45.031 DEBUG:teuthology.orchestra.run.trial023:> rm -rf -- /home/ubuntu/cephtest/archive 2026-02-22T22:40:45.056 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2026-02-22T22:40:45.073 INFO:teuthology.task.internal:Not uploading archives. 2026-02-22T22:40:45.073 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2026-02-22T22:40:45.090 INFO:teuthology.task.internal:Tidying up after the test... 2026-02-22T22:40:45.090 DEBUG:teuthology.orchestra.run.trial023:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2026-02-22T22:40:45.100 INFO:teuthology.orchestra.run.trial023.stdout: 1179687 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Feb 22 22:40 /home/ubuntu/cephtest 2026-02-22T22:40:45.101 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2026-02-22T22:40:45.118 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2026-02-22T22:40:45.155 INFO:teuthology.run:Summary data: description: rados/singleton/{all/rebuild-mondb mon_election/connectivity msgr-failures/few msgr/async objectstore/bluestore-comp-zstd rados supported-random-distro$/{ubuntu_latest}} duration: 601.8391242027283 flavor: default owner: scheduled_yuriw@soko04.front.sepia.ceph.com success: true 2026-02-22T22:40:45.155 DEBUG:teuthology.report:Pushing job info to https://paddles-paddles.apps.pok.os.sepia.ceph.com 2026-02-22T22:40:45.283 INFO:teuthology.run:pass