2026-02-26T03:15:12.598 INFO:root:teuthology version: 1.2.4.dev11+g6e3d0419c 2026-02-26T03:15:12.616 DEBUG:teuthology.report:Pushing job info to https://paddles-paddles.apps.pok.os.sepia.ceph.com 2026-02-26T03:15:12.696 INFO:teuthology.run:Config: archive_path: /home/teuthworker/mnt/teuthology/shraddhaag-2026-02-26_02:22:56-crimson-rados-main-distro-debug-trial/71215 branch: main description: crimson-rados/rbd/{0-crimson_install clusters/crimson-fixed-1 crimson-supported-all-distro/centos_latest crimson_qa_overrides deploy/ceph objectstore/seastore/segmented$/{crimson_seastore_segmented_2q} tasks/rbd_lock_and_fence} email: null first_in_suite: false flavor: debug job_id: '71215' kernel: branch: distro kdb: true sha1: distro last_in_suite: false machine_type: trial name: shraddhaag-2026-02-26_02:22:56-crimson-rados-main-distro-debug-trial no_nested_subset: false os_type: centos os_version: 9.stream overrides: admin_socket: branch: main ceph: conf: global: enable experimental unrecoverable data corrupting features: crimson ms client mode: crc ms cluster mode: crc ms mon client mode: crc ms mon cluster mode: crc ms mon service mode: crc ms service mode: crc mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 osd pool default crimson: true osd: crimson cpu num: 4 crimson osd obc lru size: 10 debug ms: 20 debug osd: 20 debug seastore: 20 debug seastore backref: 20 debug seastore cache: 20 debug seastore cleaner: 20 debug seastore device: 20 debug seastore epm: 20 debug seastore fixedkv tree: 20 debug seastore journal: 20 debug seastore lba: 20 debug seastore odata: 20 debug seastore omap: 20 debug seastore onode: 20 debug seastore t: 20 debug seastore tm: 20 osd objectstore: seastore osd shutdown pgref assert: true seastore cachepin size pershard: 64M seastore cachepin type: 2Q seastore max concurrent transactions: 8 crimson_compat: true flavor: debug log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - \(POOL_APP_NOT_ENABLED\) sha1: abffb617683951511fccf3c3cab6e219f2ec83a8 ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log global: osd crush chooseleaf type: 0 osd pool default pg num: 128 osd pool default pgp num: 128 osd pool default size: 2 mon: {} install: ceph: flavor: debug sha1: abffb617683951511fccf3c3cab6e219f2ec83a8 workunit: branch: main env: CRIMSON_COMPAT: '1' sha1: 1f87b69b1a8041112e26ae0ea3c0a2a0fd14d996 owner: scheduled_shraddhaag@soko04.front.sepia.ceph.com priority: 101 repo: https://git.ceph.com/ceph-ci.git roles: - - mon.a - mgr.x - osd.0 - osd.1 - osd.2 - client.0 seed: 5117 sha1: abffb617683951511fccf3c3cab6e219f2ec83a8 sleep_before_teardown: 0 suite: crimson-rados suite_branch: main suite_path: /home/teuthworker/src/github.com_ceph_ceph_1f87b69b1a8041112e26ae0ea3c0a2a0fd14d996/qa suite_relpath: qa suite_repo: https://github.com/ceph/ceph.git suite_sha1: 1f87b69b1a8041112e26ae0ea3c0a2a0fd14d996 targets: trial132.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBJIsuoPOk/pW9C2qwYD17/s6eAz2ubvSBrM1JvHl7ScRFJU0FMVfQGaVW5S283uWQnOTSubNmVvvqJmKYuwXDGg= tasks: - install: extra_packages: deb: - ceph-osd-crimson - ceph-osd-crimson-dbg rpm: - ceph-osd-crimson - ceph-osd-crimson-debuginfo - pexec: all: - sudo update-alternatives --set ceph-osd /usr/bin/ceph-osd-crimson - install: null - ceph: conf: mon: mon min osdmap epochs: 50 mon osdmap full prune interval: 2 mon osdmap full prune min: 15 mon osdmap full prune txsize: 2 paxos service trim min: 10 osd: debug monc: 20 - workunit: clients: client.0: - rbd/test_lock_fence.sh teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: 6e3d0419ccca0e75a03e08feb708946e198236f4 timestamp: 2026-02-26_02:22:56 tube: trial user: shraddhaag verbose: false worker_log: /home/teuthworker/mnt/teuthology/worker_logs/dispatcher.trial.904699 2026-02-26T03:15:12.696 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/github.com_ceph_ceph_1f87b69b1a8041112e26ae0ea3c0a2a0fd14d996/qa; will attempt to use it 2026-02-26T03:15:12.696 INFO:teuthology.run:Found tasks at /home/teuthworker/src/github.com_ceph_ceph_1f87b69b1a8041112e26ae0ea3c0a2a0fd14d996/qa/tasks 2026-02-26T03:15:12.697 INFO:teuthology.run_tasks:Running task internal.check_packages... 2026-02-26T03:15:12.698 INFO:teuthology.task.internal:Checking packages... 2026-02-26T03:15:12.698 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'debug' and ceph hash 'abffb617683951511fccf3c3cab6e219f2ec83a8' 2026-02-26T03:15:12.698 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2026-02-26T03:15:12.698 INFO:teuthology.packaging:ref: None 2026-02-26T03:15:12.698 INFO:teuthology.packaging:tag: None 2026-02-26T03:15:12.698 INFO:teuthology.packaging:branch: main 2026-02-26T03:15:12.698 INFO:teuthology.packaging:sha1: abffb617683951511fccf3c3cab6e219f2ec83a8 2026-02-26T03:15:12.698 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=debug&distros=centos%2F9%2Fx86_64&ref=main 2026-02-26T03:15:12.834 INFO:teuthology.task.internal:Found packages for ceph version 20.3.0-5593.g63656871 2026-02-26T03:15:12.835 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2026-02-26T03:15:12.847 INFO:teuthology.task.internal:no buildpackages task found 2026-02-26T03:15:12.847 INFO:teuthology.run_tasks:Running task internal.save_config... 2026-02-26T03:15:12.860 INFO:teuthology.task.internal:Saving configuration 2026-02-26T03:15:12.872 INFO:teuthology.run_tasks:Running task internal.check_lock... 2026-02-26T03:15:12.885 INFO:teuthology.task.internal.check_lock:Checking locks... 2026-02-26T03:15:12.969 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'trial132.front.sepia.ceph.com', 'description': '/home/teuthworker/mnt/teuthology/shraddhaag-2026-02-26_02:22:56-crimson-rados-main-distro-debug-trial/71215', 'up': True, 'machine_type': 'trial', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9.stream', 'arch': 'x86_64', 'locked': True, 'locked_since': '2026-02-26 03:11:43.667815', 'locked_by': 'scheduled_shraddhaag@soko04.front.sepia.ceph.com', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBJIsuoPOk/pW9C2qwYD17/s6eAz2ubvSBrM1JvHl7ScRFJU0FMVfQGaVW5S283uWQnOTSubNmVvvqJmKYuwXDGg='} 2026-02-26T03:15:12.969 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2026-02-26T03:15:12.980 INFO:teuthology.task.internal:roles: ubuntu@trial132.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'osd.0', 'osd.1', 'osd.2', 'client.0'] 2026-02-26T03:15:12.980 INFO:teuthology.run_tasks:Running task console_log... 2026-02-26T03:15:13.073 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7e4d1a65e660>, signals=[15]) 2026-02-26T03:15:13.073 INFO:teuthology.run_tasks:Running task internal.connect... 2026-02-26T03:15:13.084 INFO:teuthology.task.internal:Opening connections... 2026-02-26T03:15:13.084 DEBUG:teuthology.task.internal:connecting to ubuntu@trial132.front.sepia.ceph.com 2026-02-26T03:15:13.085 DEBUG:teuthology.orchestra.connection:{'hostname': 'trial132.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2026-02-26T03:15:13.146 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2026-02-26T03:15:13.161 DEBUG:teuthology.orchestra.run.trial132:> uname -m 2026-02-26T03:15:13.220 INFO:teuthology.orchestra.run.trial132.stdout:x86_64 2026-02-26T03:15:13.220 DEBUG:teuthology.orchestra.run.trial132:> cat /etc/os-release 2026-02-26T03:15:13.275 INFO:teuthology.orchestra.run.trial132.stdout:NAME="CentOS Stream" 2026-02-26T03:15:13.275 INFO:teuthology.orchestra.run.trial132.stdout:VERSION="9" 2026-02-26T03:15:13.275 INFO:teuthology.orchestra.run.trial132.stdout:ID="centos" 2026-02-26T03:15:13.275 INFO:teuthology.orchestra.run.trial132.stdout:ID_LIKE="rhel fedora" 2026-02-26T03:15:13.275 INFO:teuthology.orchestra.run.trial132.stdout:VERSION_ID="9" 2026-02-26T03:15:13.275 INFO:teuthology.orchestra.run.trial132.stdout:PLATFORM_ID="platform:el9" 2026-02-26T03:15:13.275 INFO:teuthology.orchestra.run.trial132.stdout:PRETTY_NAME="CentOS Stream 9" 2026-02-26T03:15:13.275 INFO:teuthology.orchestra.run.trial132.stdout:ANSI_COLOR="0;31" 2026-02-26T03:15:13.275 INFO:teuthology.orchestra.run.trial132.stdout:LOGO="fedora-logo-icon" 2026-02-26T03:15:13.275 INFO:teuthology.orchestra.run.trial132.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2026-02-26T03:15:13.275 INFO:teuthology.orchestra.run.trial132.stdout:HOME_URL="https://centos.org/" 2026-02-26T03:15:13.275 INFO:teuthology.orchestra.run.trial132.stdout:BUG_REPORT_URL="https://issues.redhat.com/" 2026-02-26T03:15:13.275 INFO:teuthology.orchestra.run.trial132.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2026-02-26T03:15:13.275 INFO:teuthology.orchestra.run.trial132.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2026-02-26T03:15:13.275 INFO:teuthology.lock.ops:Updating trial132.front.sepia.ceph.com on lock server 2026-02-26T03:15:13.353 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2026-02-26T03:15:13.365 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2026-02-26T03:15:13.378 INFO:teuthology.task.internal:Checking for old test directory... 2026-02-26T03:15:13.378 DEBUG:teuthology.orchestra.run.trial132:> test '!' -e /home/ubuntu/cephtest 2026-02-26T03:15:13.393 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2026-02-26T03:15:13.403 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2026-02-26T03:15:13.403 DEBUG:teuthology.orchestra.run.trial132:> test -z $(ls -A /var/lib/ceph) 2026-02-26T03:15:13.448 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2026-02-26T03:15:13.553 INFO:teuthology.run_tasks:Running task kernel... 2026-02-26T03:15:13.574 INFO:teuthology.task.kernel:normalize config orig: {'branch': 'distro', 'kdb': True, 'sha1': 'distro'} 2026-02-26T03:15:13.574 INFO:teuthology.task.kernel:config {'mon.a': {'branch': 'distro', 'kdb': True, 'sha1': 'distro'}}, timeout 300 2026-02-26T03:15:13.574 DEBUG:teuthology.orchestra.run.trial132:> test -f /run/.containerenv -o -f /.dockerenv 2026-02-26T03:15:13.588 DEBUG:teuthology.orchestra.run:got remote process result: 1 2026-02-26T03:15:13.589 DEBUG:teuthology.orchestra.run.trial132:> uname -r 2026-02-26T03:15:13.645 INFO:teuthology.orchestra.run.trial132.stdout:5.14.0-681.el9.x86_64 2026-02-26T03:15:13.645 INFO:teuthology.task.kernel:Running kernel on trial132: 5.14.0-681.el9.x86_64 2026-02-26T03:15:13.645 DEBUG:teuthology.orchestra.run.trial132:> sudo yum install -y kernel 2026-02-26T03:15:14.782 INFO:teuthology.orchestra.run.trial132.stdout:CentOS Stream 9 - BaseOS 12 MB/s | 8.9 MB 00:00 2026-02-26T03:15:18.073 INFO:teuthology.orchestra.run.trial132.stdout:CentOS Stream 9 - AppStream 9.8 MB/s | 27 MB 00:02 2026-02-26T03:15:21.415 INFO:teuthology.orchestra.run.trial132.stdout:CentOS Stream 9 - CRB 8.2 MB/s | 8.0 MB 00:00 2026-02-26T03:15:22.609 INFO:teuthology.orchestra.run.trial132.stdout:CentOS Stream 9 - Extras packages 44 kB/s | 20 kB 00:00 2026-02-26T03:15:23.837 INFO:teuthology.orchestra.run.trial132.stdout:Extra Packages for Enterprise Linux 17 MB/s | 20 MB 00:01 2026-02-26T03:15:27.049 INFO:teuthology.orchestra.run.trial132.stdout:lab-extras 1.5 MB/s | 50 kB 00:00 2026-02-26T03:15:28.078 INFO:teuthology.orchestra.run.trial132.stdout:Package kernel-5.14.0-645.el9.x86_64 is already installed. 2026-02-26T03:15:28.078 INFO:teuthology.orchestra.run.trial132.stdout:Package kernel-5.14.0-681.el9.x86_64 is already installed. 2026-02-26T03:15:28.098 INFO:teuthology.orchestra.run.trial132.stdout:Dependencies resolved. 2026-02-26T03:15:28.100 INFO:teuthology.orchestra.run.trial132.stdout:================================================================================ 2026-02-26T03:15:28.100 INFO:teuthology.orchestra.run.trial132.stdout: Package Architecture Version Repository Size 2026-02-26T03:15:28.100 INFO:teuthology.orchestra.run.trial132.stdout:================================================================================ 2026-02-26T03:15:28.100 INFO:teuthology.orchestra.run.trial132.stdout:Installing: 2026-02-26T03:15:28.100 INFO:teuthology.orchestra.run.trial132.stdout: kernel x86_64 5.14.0-686.el9 baseos 901 k 2026-02-26T03:15:28.100 INFO:teuthology.orchestra.run.trial132.stdout:Installing dependencies: 2026-02-26T03:15:28.100 INFO:teuthology.orchestra.run.trial132.stdout: kernel-core x86_64 5.14.0-686.el9 baseos 17 M 2026-02-26T03:15:28.100 INFO:teuthology.orchestra.run.trial132.stdout: kernel-modules x86_64 5.14.0-686.el9 baseos 40 M 2026-02-26T03:15:28.100 INFO:teuthology.orchestra.run.trial132.stdout: kernel-modules-core x86_64 5.14.0-686.el9 baseos 31 M 2026-02-26T03:15:28.100 INFO:teuthology.orchestra.run.trial132.stdout: 2026-02-26T03:15:28.101 INFO:teuthology.orchestra.run.trial132.stdout:Transaction Summary 2026-02-26T03:15:28.101 INFO:teuthology.orchestra.run.trial132.stdout:================================================================================ 2026-02-26T03:15:28.101 INFO:teuthology.orchestra.run.trial132.stdout:Install 4 Packages 2026-02-26T03:15:28.101 INFO:teuthology.orchestra.run.trial132.stdout: 2026-02-26T03:15:28.101 INFO:teuthology.orchestra.run.trial132.stdout:Total download size: 89 M 2026-02-26T03:15:28.101 INFO:teuthology.orchestra.run.trial132.stdout:Installed size: 135 M 2026-02-26T03:15:28.101 INFO:teuthology.orchestra.run.trial132.stdout:Downloading Packages: 2026-02-26T03:15:28.378 INFO:teuthology.orchestra.run.trial132.stdout:(1/4): kernel-5.14.0-686.el9.x86_64.rpm 30 MB/s | 901 kB 00:00 2026-02-26T03:15:28.449 INFO:teuthology.orchestra.run.trial132.stdout:(2/4): kernel-core-5.14.0-686.el9.x86_64.rpm 170 MB/s | 17 MB 00:00 2026-02-26T03:15:28.550 INFO:teuthology.orchestra.run.trial132.stdout:(3/4): kernel-modules-5.14.0-686.el9.x86_64.rpm 198 MB/s | 40 MB 00:00 2026-02-26T03:15:28.592 INFO:teuthology.orchestra.run.trial132.stdout:(4/4): kernel-modules-core-5.14.0-686.el9.x86_6 145 MB/s | 31 MB 00:00 2026-02-26T03:15:28.592 INFO:teuthology.orchestra.run.trial132.stdout:-------------------------------------------------------------------------------- 2026-02-26T03:15:28.593 INFO:teuthology.orchestra.run.trial132.stdout:Total 180 MB/s | 89 MB 00:00 2026-02-26T03:15:28.784 INFO:teuthology.orchestra.run.trial132.stdout:Running transaction check 2026-02-26T03:15:28.984 INFO:teuthology.orchestra.run.trial132.stdout:Transaction check succeeded. 2026-02-26T03:15:28.984 INFO:teuthology.orchestra.run.trial132.stdout:Running transaction test 2026-02-26T03:15:29.366 INFO:teuthology.orchestra.run.trial132.stdout:Transaction test succeeded. 2026-02-26T03:15:29.366 INFO:teuthology.orchestra.run.trial132.stdout:Running transaction 2026-02-26T03:15:30.469 INFO:teuthology.orchestra.run.trial132.stdout: Preparing : 1/1 2026-02-26T03:15:30.756 INFO:teuthology.orchestra.run.trial132.stdout: Installing : kernel-modules-core-5.14.0-686.el9.x86_64 1/4 2026-02-26T03:15:30.778 INFO:teuthology.orchestra.run.trial132.stdout: Installing : kernel-core-5.14.0-686.el9.x86_64 2/4 2026-02-26T03:15:33.352 INFO:teuthology.orchestra.run.trial132.stdout: Running scriptlet: kernel-core-5.14.0-686.el9.x86_64 2/4 2026-02-26T03:15:33.385 INFO:teuthology.orchestra.run.trial132.stdout: Installing : kernel-modules-5.14.0-686.el9.x86_64 3/4 2026-02-26T03:15:36.424 INFO:teuthology.orchestra.run.trial132.stdout: Running scriptlet: kernel-modules-5.14.0-686.el9.x86_64 3/4 2026-02-26T03:15:36.432 INFO:teuthology.orchestra.run.trial132.stdout: Installing : kernel-5.14.0-686.el9.x86_64 4/4 2026-02-26T03:15:39.477 INFO:teuthology.orchestra.run.trial132.stdout: Running scriptlet: kernel-modules-core-5.14.0-686.el9.x86_64 4/4 2026-02-26T03:15:57.026 INFO:teuthology.orchestra.run.trial132.stdout: Running scriptlet: kernel-core-5.14.0-686.el9.x86_64 4/4 2026-02-26T03:15:57.047 INFO:teuthology.orchestra.run.trial132.stdout: Running scriptlet: kernel-modules-5.14.0-686.el9.x86_64 4/4 2026-02-26T03:15:57.234 INFO:teuthology.orchestra.run.trial132.stdout: Running scriptlet: kernel-5.14.0-686.el9.x86_64 4/4 2026-02-26T03:15:57.235 INFO:teuthology.orchestra.run.trial132.stdout: Verifying : kernel-5.14.0-686.el9.x86_64 1/4 2026-02-26T03:15:57.235 INFO:teuthology.orchestra.run.trial132.stdout: Verifying : kernel-core-5.14.0-686.el9.x86_64 2/4 2026-02-26T03:15:57.235 INFO:teuthology.orchestra.run.trial132.stdout: Verifying : kernel-modules-5.14.0-686.el9.x86_64 3/4 2026-02-26T03:15:57.292 INFO:teuthology.orchestra.run.trial132.stdout: Verifying : kernel-modules-core-5.14.0-686.el9.x86_64 4/4 2026-02-26T03:15:57.292 INFO:teuthology.orchestra.run.trial132.stdout: 2026-02-26T03:15:57.292 INFO:teuthology.orchestra.run.trial132.stdout:Installed: 2026-02-26T03:15:57.292 INFO:teuthology.orchestra.run.trial132.stdout: kernel-5.14.0-686.el9.x86_64 2026-02-26T03:15:57.293 INFO:teuthology.orchestra.run.trial132.stdout: kernel-core-5.14.0-686.el9.x86_64 2026-02-26T03:15:57.293 INFO:teuthology.orchestra.run.trial132.stdout: kernel-modules-5.14.0-686.el9.x86_64 2026-02-26T03:15:57.293 INFO:teuthology.orchestra.run.trial132.stdout: kernel-modules-core-5.14.0-686.el9.x86_64 2026-02-26T03:15:57.293 INFO:teuthology.orchestra.run.trial132.stdout: 2026-02-26T03:15:57.293 INFO:teuthology.orchestra.run.trial132.stdout:Complete! 2026-02-26T03:15:57.354 DEBUG:teuthology.orchestra.run.trial132:> rpm -q kernel | sort -rV | head -n 1 2026-02-26T03:15:57.384 INFO:teuthology.orchestra.run.trial132.stdout:kernel-5.14.0-686.el9.x86_64 2026-02-26T03:15:57.385 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-686.el9.x86_64 2026-02-26T03:15:57.385 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-681.el9.x86_64 Expected: 5.14.0-686.el9.x86_64 2026-02-26T03:15:57.385 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2026-02-26T03:15:57.385 DEBUG:teuthology.task.kernel:src is distro, skipping download 2026-02-26T03:15:57.385 INFO:teuthology.task.kernel:Installing distro kernel on mon.a... 2026-02-26T03:15:57.385 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@trial132.front.sepia.ceph.com, path=None, version=distro) 2026-02-26T03:15:57.386 DEBUG:teuthology.orchestra.run.trial132:> rpm -q kernel | sort -rV | head -n 1 2026-02-26T03:15:57.449 INFO:teuthology.orchestra.run.trial132.stdout:kernel-5.14.0-686.el9.x86_64 2026-02-26T03:15:57.450 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-686.el9.x86_64 2026-02-26T03:15:57.451 DEBUG:teuthology.orchestra.run.trial132:> sudo rpm -qi grub2-tools 2026-02-26T03:15:57.519 INFO:teuthology.orchestra.run.trial132.stdout:Name : grub2-tools 2026-02-26T03:15:57.519 INFO:teuthology.orchestra.run.trial132.stdout:Epoch : 1 2026-02-26T03:15:57.519 INFO:teuthology.orchestra.run.trial132.stdout:Version : 2.06 2026-02-26T03:15:57.519 INFO:teuthology.orchestra.run.trial132.stdout:Release : 120.el9 2026-02-26T03:15:57.519 INFO:teuthology.orchestra.run.trial132.stdout:Architecture: x86_64 2026-02-26T03:15:57.519 INFO:teuthology.orchestra.run.trial132.stdout:Install Date: Wed 18 Feb 2026 05:18:42 PM UTC 2026-02-26T03:15:57.519 INFO:teuthology.orchestra.run.trial132.stdout:Group : Unspecified 2026-02-26T03:15:57.519 INFO:teuthology.orchestra.run.trial132.stdout:Size : 9108611 2026-02-26T03:15:57.520 INFO:teuthology.orchestra.run.trial132.stdout:License : GPLv3+ 2026-02-26T03:15:57.520 INFO:teuthology.orchestra.run.trial132.stdout:Signature : RSA/SHA256, Thu 04 Dec 2025 09:42:59 AM UTC, Key ID 05b555b38483c65d 2026-02-26T03:15:57.520 INFO:teuthology.orchestra.run.trial132.stdout:Source RPM : grub2-2.06-120.el9.src.rpm 2026-02-26T03:15:57.520 INFO:teuthology.orchestra.run.trial132.stdout:Build Date : Wed 03 Dec 2025 12:10:38 PM UTC 2026-02-26T03:15:57.520 INFO:teuthology.orchestra.run.trial132.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2026-02-26T03:15:57.520 INFO:teuthology.orchestra.run.trial132.stdout:Packager : builder@centos.org 2026-02-26T03:15:57.520 INFO:teuthology.orchestra.run.trial132.stdout:Vendor : CentOS 2026-02-26T03:15:57.520 INFO:teuthology.orchestra.run.trial132.stdout:URL : http://www.gnu.org/software/grub/ 2026-02-26T03:15:57.520 INFO:teuthology.orchestra.run.trial132.stdout:Summary : Support tools for GRUB. 2026-02-26T03:15:57.520 INFO:teuthology.orchestra.run.trial132.stdout:Description : 2026-02-26T03:15:57.520 INFO:teuthology.orchestra.run.trial132.stdout: 2026-02-26T03:15:57.521 INFO:teuthology.orchestra.run.trial132.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2026-02-26T03:15:57.521 INFO:teuthology.orchestra.run.trial132.stdout:customizable bootloader with modular architecture. It supports a rich 2026-02-26T03:15:57.521 INFO:teuthology.orchestra.run.trial132.stdout:variety of kernel formats, file systems, computer architectures and 2026-02-26T03:15:57.521 INFO:teuthology.orchestra.run.trial132.stdout:hardware devices. 2026-02-26T03:15:57.521 INFO:teuthology.orchestra.run.trial132.stdout: 2026-02-26T03:15:57.521 INFO:teuthology.orchestra.run.trial132.stdout:This subpackage provides tools for support of all platforms. 2026-02-26T03:15:57.522 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2026-02-26T03:15:57.522 INFO:teuthology.task.kernel:Updating grub on trial132 to boot 5.14.0-686.el9.x86_64 2026-02-26T03:15:57.523 DEBUG:teuthology.orchestra.run.trial132:> sudo find /boot/loader/entries -maxdepth 1 -name "*.conf" -type f -print -quit 2026-02-26T03:15:57.587 INFO:teuthology.orchestra.run.trial132.stdout:/boot/loader/entries/b0373be789df49cdafedf423703d2566-5.14.0-681.el9.x86_64.conf 2026-02-26T03:15:57.588 DEBUG:teuthology.orchestra.run.trial132:> sudo command -v grubby && echo yes || echo no 2026-02-26T03:15:57.652 INFO:teuthology.orchestra.run.trial132.stdout:/sbin/grubby 2026-02-26T03:15:57.654 INFO:teuthology.orchestra.run.trial132.stdout:yes 2026-02-26T03:15:57.654 DEBUG:teuthology.orchestra.run.trial132:> sudo find /boot/loader/entries -maxdepth 1 -name "*.conf" -type f | grep -F "5.14.0-686.el9.x86_64" | head -n 1 2026-02-26T03:15:57.718 INFO:teuthology.orchestra.run.trial132.stdout:/boot/loader/entries/b0373be789df49cdafedf423703d2566-5.14.0-686.el9.x86_64.conf 2026-02-26T03:15:57.719 DEBUG:teuthology.orchestra.run.trial132:> sudo grub2-set-default b0373be789df49cdafedf423703d2566-5.14.0-686.el9.x86_64 2026-02-26T03:15:57.793 DEBUG:teuthology.orchestra.run.trial132:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2026-02-26T03:15:57.995 INFO:teuthology.orchestra.run.trial132.stderr:Generating grub configuration file ... 2026-02-26T03:15:58.061 INFO:teuthology.orchestra.run.trial132.stderr:Found linux image: /boot/vmlinuz-5.14.0-686.el9.x86_64 2026-02-26T03:15:58.068 INFO:teuthology.orchestra.run.trial132.stderr:Found initrd image: /boot/initramfs-5.14.0-686.el9.x86_64.img 2026-02-26T03:15:58.321 INFO:teuthology.orchestra.run.trial132.stderr:Found linux image: /boot/vmlinuz-5.14.0-681.el9.x86_64 2026-02-26T03:15:58.328 INFO:teuthology.orchestra.run.trial132.stderr:Found initrd image: /boot/initramfs-5.14.0-681.el9.x86_64.img 2026-02-26T03:15:58.361 INFO:teuthology.orchestra.run.trial132.stderr:Found linux image: /boot/vmlinuz-5.14.0-645.el9.x86_64 2026-02-26T03:15:58.368 INFO:teuthology.orchestra.run.trial132.stderr:Found initrd image: /boot/initramfs-5.14.0-645.el9.x86_64.img 2026-02-26T03:15:58.399 INFO:teuthology.orchestra.run.trial132.stderr:Found linux image: /boot/vmlinuz-0-rescue-b0373be789df49cdafedf423703d2566 2026-02-26T03:15:58.406 INFO:teuthology.orchestra.run.trial132.stderr:Found initrd image: /boot/initramfs-0-rescue-b0373be789df49cdafedf423703d2566.img 2026-02-26T03:15:58.429 INFO:teuthology.orchestra.run.trial132.stderr:Found linux image: /boot/vmlinuz-0-rescue-2734456005d743dca7f3dc77c35a2d0d 2026-02-26T03:15:58.436 INFO:teuthology.orchestra.run.trial132.stderr:Found initrd image: /boot/initramfs-0-rescue-2734456005d743dca7f3dc77c35a2d0d.img 2026-02-26T03:15:58.522 INFO:teuthology.orchestra.run.trial132.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2026-02-26T03:15:58.532 INFO:teuthology.orchestra.run.trial132.stderr:done 2026-02-26T03:15:58.534 DEBUG:teuthology.orchestra.run.trial132:> test -d /sys/firmware/efi 2026-02-26T03:15:58.551 DEBUG:teuthology.orchestra.run.trial132:> sudo find /boot/efi/EFI -mindepth 1 -maxdepth 1 -type d || true 2026-02-26T03:15:58.618 INFO:teuthology.orchestra.run.trial132.stdout:/boot/efi/EFI/BOOT 2026-02-26T03:15:58.618 INFO:teuthology.orchestra.run.trial132.stdout:/boot/efi/EFI/centos 2026-02-26T03:15:58.620 DEBUG:teuthology.orchestra.run.trial132:> sudo test -f /boot/efi/EFI/BOOT/grub.cfg && echo yes || echo no 2026-02-26T03:15:58.687 INFO:teuthology.orchestra.run.trial132.stdout:no 2026-02-26T03:15:58.688 DEBUG:teuthology.orchestra.run.trial132:> sudo test -f /boot/efi/EFI/centos/grub.cfg && echo yes || echo no 2026-02-26T03:15:58.752 INFO:teuthology.orchestra.run.trial132.stdout:yes 2026-02-26T03:15:58.752 DEBUG:teuthology.orchestra.run.trial132:> sudo cp -f /boot/grub2/grub.cfg /boot/efi/EFI/centos/grub.cfg 2026-02-26T03:15:58.815 INFO:teuthology.orchestra.remote:Triggering safe hard reboot via SysRq on trial132 2026-02-26T03:15:58.815 DEBUG:teuthology.orchestra.run.trial132:> sudo bash -c ' 2026-02-26T03:15:58.815 DEBUG:teuthology.orchestra.run.trial132:> echo pci > /sys/kernel/reboot/type && echo 1 > /sys/kernel/reboot/force && sync && echo s > /proc/sysrq-trigger && echo u > /proc/sysrq-trigger && sleep 2 && echo b > /proc/sysrq-trigger 2026-02-26T03:15:58.815 DEBUG:teuthology.orchestra.run.trial132:> ' 2026-02-26T03:16:28.858 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version... 2026-02-26T03:16:28.859 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@trial132.front.sepia.ceph.com' 2026-02-26T03:16:28.859 DEBUG:teuthology.orchestra.connection:{'hostname': 'trial132.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2026-02-26T03:17:07.585 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 10.20.193.132 2026-02-26T03:17:16.593 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@trial132.front.sepia.ceph.com' 2026-02-26T03:17:16.594 DEBUG:teuthology.orchestra.connection:{'hostname': 'trial132.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2026-02-26T03:17:52.759 DEBUG:teuthology.orchestra.run.trial132:> true 2026-02-26T03:17:52.931 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@trial132.front.sepia.ceph.com' 2026-02-26T03:17:52.931 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "5.14.0-686.el9.x86_64"... 2026-02-26T03:17:52.931 DEBUG:teuthology.orchestra.run.trial132:> uname -r 2026-02-26T03:17:52.989 INFO:teuthology.orchestra.run.trial132.stdout:5.14.0-686.el9.x86_64 2026-02-26T03:17:52.989 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-686.el9.x86_64 vs 5.14.0-686.el9.x86_64 2026-02-26T03:17:52.989 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2026-02-26T03:17:52.990 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2026-02-26T03:17:53.990 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2026-02-26T03:17:53.990 DEBUG:teuthology.orchestra.run.trial132:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2026-02-26T03:17:54.020 INFO:teuthology.orchestra.run.trial132.stdout:ttyS1 2026-02-26T03:17:54.054 DEBUG:teuthology.parallel:result is None 2026-02-26T03:17:54.055 INFO:teuthology.run_tasks:Running task internal.base... 2026-02-26T03:17:54.068 INFO:teuthology.task.internal:Creating test directory... 2026-02-26T03:17:54.069 DEBUG:teuthology.orchestra.run.trial132:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2026-02-26T03:17:54.111 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2026-02-26T03:17:54.121 INFO:teuthology.run_tasks:Running task internal.archive... 2026-02-26T03:17:54.135 INFO:teuthology.task.internal:Creating archive directory... 2026-02-26T03:17:54.135 DEBUG:teuthology.orchestra.run.trial132:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2026-02-26T03:17:54.174 INFO:teuthology.run_tasks:Running task internal.coredump... 2026-02-26T03:17:54.187 INFO:teuthology.task.internal:Enabling coredump saving... 2026-02-26T03:17:54.187 DEBUG:teuthology.orchestra.run.trial132:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2026-02-26T03:17:54.230 INFO:teuthology.orchestra.run.trial132.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-26T03:17:54.239 INFO:teuthology.orchestra.run.trial132.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-26T03:17:54.241 INFO:teuthology.run_tasks:Running task internal.sudo... 2026-02-26T03:17:54.253 INFO:teuthology.task.internal:Configuring sudo... 2026-02-26T03:17:54.254 DEBUG:teuthology.orchestra.run.trial132:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2026-02-26T03:17:54.322 INFO:teuthology.run_tasks:Running task internal.syslog... 2026-02-26T03:17:54.339 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2026-02-26T03:17:54.339 DEBUG:teuthology.orchestra.run.trial132:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2026-02-26T03:17:54.426 DEBUG:teuthology.orchestra.run.trial132:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2026-02-26T03:17:54.550 DEBUG:teuthology.orchestra.run.trial132:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2026-02-26T03:17:54.575 DEBUG:teuthology.orchestra.run.trial132:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2026-02-26T03:17:54.705 DEBUG:teuthology.orchestra.run.trial132:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2026-02-26T03:17:54.729 DEBUG:teuthology.orchestra.run.trial132:> set -ex 2026-02-26T03:17:54.729 DEBUG:teuthology.orchestra.run.trial132:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2026-02-26T03:17:54.793 DEBUG:teuthology.orchestra.run.trial132:> sudo service rsyslog restart 2026-02-26T03:17:54.863 INFO:teuthology.orchestra.run.trial132.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2026-02-26T03:17:55.260 INFO:teuthology.run_tasks:Running task internal.timer... 2026-02-26T03:17:55.274 INFO:teuthology.task.internal:Starting timer... 2026-02-26T03:17:55.274 INFO:teuthology.run_tasks:Running task pcp... 2026-02-26T03:17:55.295 INFO:teuthology.run_tasks:Running task selinux... 2026-02-26T03:17:55.313 DEBUG:teuthology.orchestra.run.trial132:> sudo service auditd rotate 2026-02-26T03:17:55.343 INFO:teuthology.orchestra.run.trial132.stdout:Rotating logs: 2026-02-26T03:17:55.345 DEBUG:teuthology.task.selinux:Getting current SELinux state 2026-02-26T03:17:55.346 DEBUG:teuthology.orchestra.run.trial132:> /usr/sbin/getenforce 2026-02-26T03:17:55.402 INFO:teuthology.orchestra.run.trial132.stdout:Permissive 2026-02-26T03:17:55.402 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@trial132.front.sepia.ceph.com': 'permissive'} 2026-02-26T03:17:55.402 DEBUG:teuthology.orchestra.run.trial132:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 2026-02-26T03:17:55.466 DEBUG:teuthology.orchestra.run:got remote process result: 1 2026-02-26T03:17:55.467 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2026-02-26T03:17:55.467 DEBUG:teuthology.orchestra.run.trial132:> sudo /usr/sbin/setenforce permissive 2026-02-26T03:17:55.530 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2026-02-26T03:17:55.544 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2026-02-26T03:17:55.555 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2026-02-26T03:17:55.556 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit trial132.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2026-02-26T03:20:51.908 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@trial132.front.sepia.ceph.com')] 2026-02-26T03:20:51.916 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@trial132.front.sepia.ceph.com' 2026-02-26T03:20:51.918 DEBUG:teuthology.orchestra.connection:{'hostname': 'trial132.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2026-02-26T03:20:51.986 DEBUG:teuthology.orchestra.run.trial132:> true 2026-02-26T03:20:52.064 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@trial132.front.sepia.ceph.com' 2026-02-26T03:20:52.064 INFO:teuthology.run_tasks:Running task clock... 2026-02-26T03:20:52.073 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2026-02-26T03:20:52.073 INFO:teuthology.orchestra.run:Running command with timeout 360 2026-02-26T03:20:52.074 DEBUG:teuthology.orchestra.run.trial132:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2026-02-26T03:20:52.131 INFO:teuthology.orchestra.run.trial132.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2026-02-26T03:20:52.144 INFO:teuthology.orchestra.run.trial132.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2026-02-26T03:20:52.165 INFO:teuthology.orchestra.run.trial132.stderr:sudo: ntpd: command not found 2026-02-26T03:20:52.175 INFO:teuthology.orchestra.run.trial132.stdout:506 Cannot talk to daemon 2026-02-26T03:20:52.187 INFO:teuthology.orchestra.run.trial132.stderr:Failed to start ntp.service: Unit ntp.service not found. 2026-02-26T03:20:52.200 INFO:teuthology.orchestra.run.trial132.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2026-02-26T03:20:52.248 INFO:teuthology.orchestra.run.trial132.stderr:bash: line 1: ntpq: command not found 2026-02-26T03:20:52.250 INFO:teuthology.orchestra.run.trial132.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2026-02-26T03:20:52.250 INFO:teuthology.orchestra.run.trial132.stdout:=============================================================================== 2026-02-26T03:20:52.251 INFO:teuthology.run_tasks:Running task install... 2026-02-26T03:20:52.265 DEBUG:teuthology.task.install:project ceph 2026-02-26T03:20:52.266 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'debug', 'sha1': 'abffb617683951511fccf3c3cab6e219f2ec83a8'}} 2026-02-26T03:20:52.266 DEBUG:teuthology.task.install:config {'extra_packages': {'deb': ['ceph-osd-crimson', 'ceph-osd-crimson-dbg'], 'rpm': ['ceph-osd-crimson', 'ceph-osd-crimson-debuginfo']}, 'flavor': 'debug', 'sha1': 'abffb617683951511fccf3c3cab6e219f2ec83a8'} 2026-02-26T03:20:52.266 INFO:teuthology.task.install:Using flavor: debug 2026-02-26T03:20:52.276 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-osd', 'ceph-osd-classic', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-osd', 'ceph-osd-classic', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2026-02-26T03:20:52.277 INFO:teuthology.task.install:extra packages: {'deb': ['ceph-osd-crimson', 'ceph-osd-crimson-dbg'], 'rpm': ['ceph-osd-crimson', 'ceph-osd-crimson-debuginfo']} 2026-02-26T03:20:52.277 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['ceph-osd-crimson', 'ceph-osd-crimson-dbg'], 'rpm': ['ceph-osd-crimson', 'ceph-osd-crimson-debuginfo']}, 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'debug', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': 'abffb617683951511fccf3c3cab6e219f2ec83a8', 'tag': None, 'wait_for_package': False} 2026-02-26T03:20:52.278 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=debug&distros=centos%2F9%2Fx86_64&sha1=abffb617683951511fccf3c3cab6e219f2ec83a8 2026-02-26T03:20:52.442 INFO:teuthology.task.install.rpm:Pulling from https://1.chacra.ceph.com/r/ceph/wip-shraddhaag-peering-bug/abffb617683951511fccf3c3cab6e219f2ec83a8/centos/9/flavors/debug/ 2026-02-26T03:20:52.442 INFO:teuthology.task.install.rpm:Package version is 20.3.0-5312.gabffb617 2026-02-26T03:20:52.823 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://1.chacra.ceph.com/r/ceph/wip-shraddhaag-peering-bug/abffb617683951511fccf3c3cab6e219f2ec83a8/centos/9/flavors/debug/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://1.chacra.ceph.com/r/ceph/wip-shraddhaag-peering-bug/abffb617683951511fccf3c3cab6e219f2ec83a8/centos/9/flavors/debug/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://1.chacra.ceph.com/r/ceph/wip-shraddhaag-peering-bug/abffb617683951511fccf3c3cab6e219f2ec83a8/centos/9/flavors/debug/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2026-02-26T03:20:52.824 DEBUG:teuthology.orchestra.run.trial132:> set -ex 2026-02-26T03:20:52.824 DEBUG:teuthology.orchestra.run.trial132:> sudo dd of=/etc/yum.repos.d/ceph.repo 2026-02-26T03:20:52.850 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-osd, ceph-osd-classic, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, ceph-osd-crimson, ceph-osd-crimson-debuginfo on remote rpm x86_64 2026-02-26T03:20:52.850 DEBUG:teuthology.orchestra.run.trial132:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' /etc/yum.repos.d/ceph.repo ; fi 2026-02-26T03:20:52.915 DEBUG:teuthology.orchestra.run.trial132:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2026-02-26T03:20:52.996 DEBUG:teuthology.orchestra.run.trial132:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2026-02-26T03:20:53.060 INFO:teuthology.orchestra.run.trial132.stdout:check_obsoletes = 1 2026-02-26T03:20:53.061 DEBUG:teuthology.orchestra.run.trial132:> sudo yum clean all 2026-02-26T03:20:53.211 INFO:teuthology.orchestra.run.trial132.stdout:41 files removed 2026-02-26T03:20:53.225 DEBUG:teuthology.orchestra.run.trial132:> sudo yum -y install ceph-radosgw 2026-02-26T03:20:54.085 INFO:teuthology.orchestra.run.trial132.stdout:ceph packages for x86_64 131 kB/s | 94 kB 00:00 2026-02-26T03:20:54.754 INFO:teuthology.orchestra.run.trial132.stdout:ceph noarch packages 38 kB/s | 24 kB 00:00 2026-02-26T03:20:55.350 INFO:teuthology.orchestra.run.trial132.stdout:ceph source packages 3.6 kB/s | 2.1 kB 00:00 2026-02-26T03:20:56.169 INFO:teuthology.orchestra.run.trial132.stdout:CentOS Stream 9 - BaseOS 11 MB/s | 8.9 MB 00:00 2026-02-26T03:21:07.842 INFO:teuthology.orchestra.run.trial132.stdout:CentOS Stream 9 - AppStream 2.4 MB/s | 27 MB 00:11 2026-02-26T03:21:16.520 INFO:teuthology.orchestra.run.trial132.stdout:CentOS Stream 9 - CRB 1.3 MB/s | 8.0 MB 00:06 2026-02-26T03:21:17.488 INFO:teuthology.orchestra.run.trial132.stdout:CentOS Stream 9 - Extras packages 90 kB/s | 20 kB 00:00 2026-02-26T03:21:17.816 INFO:teuthology.orchestra.run.trial132.stdout:Extra Packages for Enterprise Linux 77 MB/s | 20 MB 00:00 2026-02-26T03:21:20.984 INFO:teuthology.orchestra.run.trial132.stdout:lab-extras 1.6 MB/s | 50 kB 00:00 2026-02-26T03:21:22.216 INFO:teuthology.orchestra.run.trial132.stdout:Dependencies resolved. 2026-02-26T03:21:22.218 INFO:teuthology.orchestra.run.trial132.stdout:================================================================================ 2026-02-26T03:21:22.218 INFO:teuthology.orchestra.run.trial132.stdout: Package Arch Version Repository Size 2026-02-26T03:21:22.218 INFO:teuthology.orchestra.run.trial132.stdout:================================================================================ 2026-02-26T03:21:22.218 INFO:teuthology.orchestra.run.trial132.stdout:Installing: 2026-02-26T03:21:22.218 INFO:teuthology.orchestra.run.trial132.stdout: ceph-radosgw x86_64 2:20.3.0-5312.gabffb617.el9 ceph 11 M 2026-02-26T03:21:22.218 INFO:teuthology.orchestra.run.trial132.stdout:Upgrading: 2026-02-26T03:21:22.218 INFO:teuthology.orchestra.run.trial132.stdout: librados2 x86_64 2:20.3.0-5312.gabffb617.el9 ceph 3.3 M 2026-02-26T03:21:22.218 INFO:teuthology.orchestra.run.trial132.stdout: librbd1 x86_64 2:20.3.0-5312.gabffb617.el9 ceph 2.7 M 2026-02-26T03:21:22.219 INFO:teuthology.orchestra.run.trial132.stdout:Installing dependencies: 2026-02-26T03:21:22.219 INFO:teuthology.orchestra.run.trial132.stdout: boost-program-options x86_64 1.75.0-13.el9 appstream 104 k 2026-02-26T03:21:22.219 INFO:teuthology.orchestra.run.trial132.stdout: ceph-base x86_64 2:20.3.0-5312.gabffb617.el9 ceph 5.1 M 2026-02-26T03:21:22.219 INFO:teuthology.orchestra.run.trial132.stdout: ceph-common x86_64 2:20.3.0-5312.gabffb617.el9 ceph 21 M 2026-02-26T03:21:22.219 INFO:teuthology.orchestra.run.trial132.stdout: ceph-selinux x86_64 2:20.3.0-5312.gabffb617.el9 ceph 25 k 2026-02-26T03:21:22.219 INFO:teuthology.orchestra.run.trial132.stdout: gperftools-libs x86_64 2.9.1-3.el9 epel 308 k 2026-02-26T03:21:22.219 INFO:teuthology.orchestra.run.trial132.stdout: libarrow x86_64 9.0.0-15.el9 epel 4.4 M 2026-02-26T03:21:22.219 INFO:teuthology.orchestra.run.trial132.stdout: libarrow-doc noarch 9.0.0-15.el9 epel 25 k 2026-02-26T03:21:22.219 INFO:teuthology.orchestra.run.trial132.stdout: libcephfs2 x86_64 2:20.3.0-5312.gabffb617.el9 ceph 859 k 2026-02-26T03:21:22.219 INFO:teuthology.orchestra.run.trial132.stdout: libnbd x86_64 1.20.3-4.el9 appstream 164 k 2026-02-26T03:21:22.219 INFO:teuthology.orchestra.run.trial132.stdout: liboath x86_64 2.6.12-1.el9 epel 49 k 2026-02-26T03:21:22.220 INFO:teuthology.orchestra.run.trial132.stdout: libpmemobj x86_64 1.12.1-1.el9 appstream 160 k 2026-02-26T03:21:22.220 INFO:teuthology.orchestra.run.trial132.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2026-02-26T03:21:22.220 INFO:teuthology.orchestra.run.trial132.stdout: libradosstriper1 x86_64 2:20.3.0-5312.gabffb617.el9 ceph 263 k 2026-02-26T03:21:22.220 INFO:teuthology.orchestra.run.trial132.stdout: librdkafka x86_64 1.6.1-102.el9 appstream 662 k 2026-02-26T03:21:22.220 INFO:teuthology.orchestra.run.trial132.stdout: librgw2 x86_64 2:20.3.0-5312.gabffb617.el9 ceph 5.6 M 2026-02-26T03:21:22.220 INFO:teuthology.orchestra.run.trial132.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2026-02-26T03:21:22.220 INFO:teuthology.orchestra.run.trial132.stdout: lttng-ust x86_64 2.12.0-6.el9 appstream 292 k 2026-02-26T03:21:22.220 INFO:teuthology.orchestra.run.trial132.stdout: mailcap noarch 2.1.49-5.el9 baseos 33 k 2026-02-26T03:21:22.220 INFO:teuthology.orchestra.run.trial132.stdout: parquet-libs x86_64 9.0.0-15.el9 epel 838 k 2026-02-26T03:21:22.220 INFO:teuthology.orchestra.run.trial132.stdout: pciutils x86_64 3.7.0-7.el9 baseos 93 k 2026-02-26T03:21:22.220 INFO:teuthology.orchestra.run.trial132.stdout: python3-ceph-argparse x86_64 2:20.3.0-5312.gabffb617.el9 ceph 45 k 2026-02-26T03:21:22.220 INFO:teuthology.orchestra.run.trial132.stdout: python3-ceph-common x86_64 2:20.3.0-5312.gabffb617.el9 ceph 185 k 2026-02-26T03:21:22.220 INFO:teuthology.orchestra.run.trial132.stdout: python3-cephfs x86_64 2:20.3.0-5312.gabffb617.el9 ceph 203 k 2026-02-26T03:21:22.221 INFO:teuthology.orchestra.run.trial132.stdout: python3-rados x86_64 2:20.3.0-5312.gabffb617.el9 ceph 324 k 2026-02-26T03:21:22.221 INFO:teuthology.orchestra.run.trial132.stdout: python3-rbd x86_64 2:20.3.0-5312.gabffb617.el9 ceph 305 k 2026-02-26T03:21:22.221 INFO:teuthology.orchestra.run.trial132.stdout: python3-rgw x86_64 2:20.3.0-5312.gabffb617.el9 ceph 100 k 2026-02-26T03:21:22.221 INFO:teuthology.orchestra.run.trial132.stdout: qatlib x86_64 25.08.0-2.el9 appstream 240 k 2026-02-26T03:21:22.221 INFO:teuthology.orchestra.run.trial132.stdout: qatzip-libs x86_64 1.3.1-1.el9 appstream 66 k 2026-02-26T03:21:22.221 INFO:teuthology.orchestra.run.trial132.stdout: re2 x86_64 1:20211101-20.el9 epel 191 k 2026-02-26T03:21:22.221 INFO:teuthology.orchestra.run.trial132.stdout: thrift x86_64 0.15.0-4.el9 epel 1.6 M 2026-02-26T03:21:22.221 INFO:teuthology.orchestra.run.trial132.stdout:Installing weak dependencies: 2026-02-26T03:21:22.221 INFO:teuthology.orchestra.run.trial132.stdout: libcephfs-daemon x86_64 2:20.3.0-5312.gabffb617.el9 ceph 37 k 2026-02-26T03:21:22.221 INFO:teuthology.orchestra.run.trial132.stdout: libcephfs-proxy2 x86_64 2:20.3.0-5312.gabffb617.el9 ceph 26 k 2026-02-26T03:21:22.221 INFO:teuthology.orchestra.run.trial132.stdout: nvme-cli x86_64 2.16-1.el9 baseos 1.2 M 2026-02-26T03:21:22.221 INFO:teuthology.orchestra.run.trial132.stdout: qatlib-service x86_64 25.08.0-2.el9 appstream 37 k 2026-02-26T03:21:22.222 INFO:teuthology.orchestra.run.trial132.stdout: 2026-02-26T03:21:22.222 INFO:teuthology.orchestra.run.trial132.stdout:Transaction Summary 2026-02-26T03:21:22.222 INFO:teuthology.orchestra.run.trial132.stdout:================================================================================ 2026-02-26T03:21:22.222 INFO:teuthology.orchestra.run.trial132.stdout:Install 35 Packages 2026-02-26T03:21:22.222 INFO:teuthology.orchestra.run.trial132.stdout:Upgrade 2 Packages 2026-02-26T03:21:22.222 INFO:teuthology.orchestra.run.trial132.stdout: 2026-02-26T03:21:22.223 INFO:teuthology.orchestra.run.trial132.stdout:Total download size: 61 M 2026-02-26T03:21:22.223 INFO:teuthology.orchestra.run.trial132.stdout:Downloading Packages: 2026-02-26T03:21:35.616 INFO:teuthology.orchestra.run.trial132.stdout:(1/37): ceph-base-20.3.0-5312.gabffb617.el9.x86 405 kB/s | 5.1 MB 00:12 2026-02-26T03:21:35.681 INFO:teuthology.orchestra.run.trial132.stdout:(2/37): ceph-selinux-20.3.0-5312.gabffb617.el9. 384 kB/s | 25 kB 00:00 2026-02-26T03:21:35.815 INFO:teuthology.orchestra.run.trial132.stdout:(3/37): libcephfs-daemon-20.3.0-5312.gabffb617. 276 kB/s | 37 kB 00:00 2026-02-26T03:21:35.885 INFO:teuthology.orchestra.run.trial132.stdout:(4/37): libcephfs-proxy2-20.3.0-5312.gabffb617. 367 kB/s | 26 kB 00:00 2026-02-26T03:21:38.455 INFO:teuthology.orchestra.run.trial132.stdout:(5/37): libcephfs2-20.3.0-5312.gabffb617.el9.x8 334 kB/s | 859 kB 00:02 2026-02-26T03:21:39.087 INFO:teuthology.orchestra.run.trial132.stdout:(6/37): libradosstriper1-20.3.0-5312.gabffb617. 416 kB/s | 263 kB 00:00 2026-02-26T03:21:41.206 INFO:teuthology.orchestra.run.trial132.stdout:(7/37): ceph-radosgw-20.3.0-5312.gabffb617.el9. 618 kB/s | 11 MB 00:18 2026-02-26T03:21:41.408 INFO:teuthology.orchestra.run.trial132.stdout:(8/37): python3-ceph-argparse-20.3.0-5312.gabff 223 kB/s | 45 kB 00:00 2026-02-26T03:21:41.969 INFO:teuthology.orchestra.run.trial132.stdout:(9/37): python3-ceph-common-20.3.0-5312.gabffb6 330 kB/s | 185 kB 00:00 2026-02-26T03:21:42.525 INFO:teuthology.orchestra.run.trial132.stdout:(10/37): python3-cephfs-20.3.0-5312.gabffb617.e 366 kB/s | 203 kB 00:00 2026-02-26T03:21:43.689 INFO:teuthology.orchestra.run.trial132.stdout:(11/37): python3-rados-20.3.0-5312.gabffb617.el 278 kB/s | 324 kB 00:01 2026-02-26T03:21:44.472 INFO:teuthology.orchestra.run.trial132.stdout:(12/37): python3-rbd-20.3.0-5312.gabffb617.el9. 390 kB/s | 305 kB 00:00 2026-02-26T03:21:44.948 INFO:teuthology.orchestra.run.trial132.stdout:(13/37): python3-rgw-20.3.0-5312.gabffb617.el9. 211 kB/s | 100 kB 00:00 2026-02-26T03:21:45.385 INFO:teuthology.orchestra.run.trial132.stdout:(14/37): mailcap-2.1.49-5.el9.noarch.rpm 76 kB/s | 33 kB 00:00 2026-02-26T03:21:46.805 INFO:teuthology.orchestra.run.trial132.stdout:(15/37): nvme-cli-2.16-1.el9.x86_64.rpm 830 kB/s | 1.2 MB 00:01 2026-02-26T03:21:46.980 INFO:teuthology.orchestra.run.trial132.stdout:(16/37): pciutils-3.7.0-7.el9.x86_64.rpm 530 kB/s | 93 kB 00:00 2026-02-26T03:21:47.158 INFO:teuthology.orchestra.run.trial132.stdout:(17/37): boost-program-options-1.75.0-13.el9.x8 586 kB/s | 104 kB 00:00 2026-02-26T03:21:47.202 INFO:teuthology.orchestra.run.trial132.stdout:(18/37): libnbd-1.20.3-4.el9.x86_64.rpm 3.6 MB/s | 164 kB 00:00 2026-02-26T03:21:47.229 INFO:teuthology.orchestra.run.trial132.stdout:(19/37): libpmemobj-1.12.1-1.el9.x86_64.rpm 5.8 MB/s | 160 kB 00:00 2026-02-26T03:21:47.250 INFO:teuthology.orchestra.run.trial132.stdout:(20/37): librabbitmq-0.11.0-7.el9.x86_64.rpm 2.1 MB/s | 45 kB 00:00 2026-02-26T03:21:47.303 INFO:teuthology.orchestra.run.trial132.stdout:(21/37): librdkafka-1.6.1-102.el9.x86_64.rpm 12 MB/s | 662 kB 00:00 2026-02-26T03:21:47.328 INFO:teuthology.orchestra.run.trial132.stdout:(22/37): lttng-ust-2.12.0-6.el9.x86_64.rpm 12 MB/s | 292 kB 00:00 2026-02-26T03:21:47.350 INFO:teuthology.orchestra.run.trial132.stdout:(23/37): qatlib-25.08.0-2.el9.x86_64.rpm 10 MB/s | 240 kB 00:00 2026-02-26T03:21:47.370 INFO:teuthology.orchestra.run.trial132.stdout:(24/37): qatlib-service-25.08.0-2.el9.x86_64.rp 1.8 MB/s | 37 kB 00:00 2026-02-26T03:21:47.390 INFO:teuthology.orchestra.run.trial132.stdout:(25/37): qatzip-libs-1.3.1-1.el9.x86_64.rpm 3.2 MB/s | 66 kB 00:00 2026-02-26T03:21:47.423 INFO:teuthology.orchestra.run.trial132.stdout:(26/37): gperftools-libs-2.9.1-3.el9.x86_64.rpm 9.2 MB/s | 308 kB 00:00 2026-02-26T03:21:47.452 INFO:teuthology.orchestra.run.trial132.stdout:(27/37): libarrow-9.0.0-15.el9.x86_64.rpm 151 MB/s | 4.4 MB 00:00 2026-02-26T03:21:47.457 INFO:teuthology.orchestra.run.trial132.stdout:(28/37): libarrow-doc-9.0.0-15.el9.noarch.rpm 5.1 MB/s | 25 kB 00:00 2026-02-26T03:21:47.462 INFO:teuthology.orchestra.run.trial132.stdout:(29/37): liboath-2.6.12-1.el9.x86_64.rpm 9.7 MB/s | 49 kB 00:00 2026-02-26T03:21:47.467 INFO:teuthology.orchestra.run.trial132.stdout:(30/37): libunwind-1.6.2-1.el9.x86_64.rpm 13 MB/s | 67 kB 00:00 2026-02-26T03:21:47.476 INFO:teuthology.orchestra.run.trial132.stdout:(31/37): parquet-libs-9.0.0-15.el9.x86_64.rpm 97 MB/s | 838 kB 00:00 2026-02-26T03:21:47.483 INFO:teuthology.orchestra.run.trial132.stdout:(32/37): re2-20211101-20.el9.x86_64.rpm 29 MB/s | 191 kB 00:00 2026-02-26T03:21:47.493 INFO:teuthology.orchestra.run.trial132.stdout:(33/37): thrift-0.15.0-4.el9.x86_64.rpm 151 MB/s | 1.6 MB 00:00 2026-02-26T03:21:50.094 INFO:teuthology.orchestra.run.trial132.stdout:(34/37): librgw2-20.3.0-5312.gabffb617.el9.x86_ 523 kB/s | 5.6 MB 00:11 2026-02-26T03:21:56.599 INFO:teuthology.orchestra.run.trial132.stdout:(35/37): librbd1-20.3.0-5312.gabffb617.el9.x86_ 423 kB/s | 2.7 MB 00:06 2026-02-26T03:21:56.671 INFO:teuthology.orchestra.run.trial132.stdout:(36/37): librados2-20.3.0-5312.gabffb617.el9.x8 366 kB/s | 3.3 MB 00:09 2026-02-26T03:22:02.134 INFO:teuthology.orchestra.run.trial132.stdout:(37/37): ceph-common-20.3.0-5312.gabffb617.el9. 540 kB/s | 21 MB 00:39 2026-02-26T03:22:02.136 INFO:teuthology.orchestra.run.trial132.stdout:-------------------------------------------------------------------------------- 2026-02-26T03:22:02.136 INFO:teuthology.orchestra.run.trial132.stdout:Total 1.5 MB/s | 61 MB 00:39 2026-02-26T03:22:02.227 INFO:teuthology.orchestra.run.trial132.stdout:Running transaction check 2026-02-26T03:22:02.246 INFO:teuthology.orchestra.run.trial132.stdout:Transaction check succeeded. 2026-02-26T03:22:02.246 INFO:teuthology.orchestra.run.trial132.stdout:Running transaction test 2026-02-26T03:22:02.493 INFO:teuthology.orchestra.run.trial132.stdout:Transaction test succeeded. 2026-02-26T03:22:02.493 INFO:teuthology.orchestra.run.trial132.stdout:Running transaction 2026-02-26T03:22:02.783 INFO:teuthology.orchestra.run.trial132.stdout: Preparing : 1/1 2026-02-26T03:22:02.923 INFO:teuthology.orchestra.run.trial132.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/39 2026-02-26T03:22:02.928 INFO:teuthology.orchestra.run.trial132.stdout: Upgrading : librados2-2:20.3.0-5312.gabffb617.el9.x86_64 2/39 2026-02-26T03:22:02.978 INFO:teuthology.orchestra.run.trial132.stdout: Running scriptlet: librados2-2:20.3.0-5312.gabffb617.el9.x86_64 2/39 2026-02-26T03:22:02.979 INFO:teuthology.orchestra.run.trial132.stdout: Installing : libcephfs2-2:20.3.0-5312.gabffb617.el9.x86_64 3/39 2026-02-26T03:22:02.991 INFO:teuthology.orchestra.run.trial132.stdout: Running scriptlet: libcephfs2-2:20.3.0-5312.gabffb617.el9.x86_64 3/39 2026-02-26T03:22:02.993 INFO:teuthology.orchestra.run.trial132.stdout: Installing : libcephfs-daemon-2:20.3.0-5312.gabffb617.el9.x86_6 4/39 2026-02-26T03:22:02.993 INFO:teuthology.orchestra.run.trial132.stdout: Installing : libcephfs-proxy2-2:20.3.0-5312.gabffb617.el9.x86_6 5/39 2026-02-26T03:22:03.021 INFO:teuthology.orchestra.run.trial132.stdout: Running scriptlet: libcephfs-proxy2-2:20.3.0-5312.gabffb617.el9.x86_6 5/39 2026-02-26T03:22:03.023 INFO:teuthology.orchestra.run.trial132.stdout: Installing : python3-rados-2:20.3.0-5312.gabffb617.el9.x86_64 6/39 2026-02-26T03:22:03.031 INFO:teuthology.orchestra.run.trial132.stdout: Installing : liboath-2.6.12-1.el9.x86_64 7/39 2026-02-26T03:22:03.033 INFO:teuthology.orchestra.run.trial132.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 8/39 2026-02-26T03:22:03.036 INFO:teuthology.orchestra.run.trial132.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 9/39 2026-02-26T03:22:03.040 INFO:teuthology.orchestra.run.trial132.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 10/39 2026-02-26T03:22:03.051 INFO:teuthology.orchestra.run.trial132.stdout: Installing : python3-ceph-argparse-2:20.3.0-5312.gabffb617.el9. 11/39 2026-02-26T03:22:03.064 INFO:teuthology.orchestra.run.trial132.stdout: Installing : python3-cephfs-2:20.3.0-5312.gabffb617.el9.x86_64 12/39 2026-02-26T03:22:03.065 INFO:teuthology.orchestra.run.trial132.stdout: Installing : libradosstriper1-2:20.3.0-5312.gabffb617.el9.x86_6 13/39 2026-02-26T03:22:03.086 INFO:teuthology.orchestra.run.trial132.stdout: Running scriptlet: libradosstriper1-2:20.3.0-5312.gabffb617.el9.x86_6 13/39 2026-02-26T03:22:03.089 INFO:teuthology.orchestra.run.trial132.stdout: Installing : thrift-0.15.0-4.el9.x86_64 14/39 2026-02-26T03:22:03.120 INFO:teuthology.orchestra.run.trial132.stdout: Installing : re2-1:20211101-20.el9.x86_64 15/39 2026-02-26T03:22:03.123 INFO:teuthology.orchestra.run.trial132.stdout: Installing : libarrow-9.0.0-15.el9.x86_64 16/39 2026-02-26T03:22:03.130 INFO:teuthology.orchestra.run.trial132.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 17/39 2026-02-26T03:22:03.132 INFO:teuthology.orchestra.run.trial132.stdout: Installing : gperftools-libs-2.9.1-3.el9.x86_64 18/39 2026-02-26T03:22:03.136 INFO:teuthology.orchestra.run.trial132.stdout: Installing : libarrow-doc-9.0.0-15.el9.noarch 19/39 2026-02-26T03:22:03.252 INFO:teuthology.orchestra.run.trial132.stdout: Installing : libnbd-1.20.3-4.el9.x86_64 20/39 2026-02-26T03:22:03.253 INFO:teuthology.orchestra.run.trial132.stdout: Upgrading : librbd1-2:20.3.0-5312.gabffb617.el9.x86_64 21/39 2026-02-26T03:22:03.280 INFO:teuthology.orchestra.run.trial132.stdout: Running scriptlet: librbd1-2:20.3.0-5312.gabffb617.el9.x86_64 21/39 2026-02-26T03:22:03.282 INFO:teuthology.orchestra.run.trial132.stdout: Installing : python3-rbd-2:20.3.0-5312.gabffb617.el9.x86_64 22/39 2026-02-26T03:22:03.292 INFO:teuthology.orchestra.run.trial132.stdout: Installing : boost-program-options-1.75.0-13.el9.x86_64 23/39 2026-02-26T03:22:03.540 INFO:teuthology.orchestra.run.trial132.stdout: Installing : parquet-libs-9.0.0-15.el9.x86_64 24/39 2026-02-26T03:22:03.541 INFO:teuthology.orchestra.run.trial132.stdout: Installing : librgw2-2:20.3.0-5312.gabffb617.el9.x86_64 25/39 2026-02-26T03:22:03.558 INFO:teuthology.orchestra.run.trial132.stdout: Running scriptlet: librgw2-2:20.3.0-5312.gabffb617.el9.x86_64 25/39 2026-02-26T03:22:03.562 INFO:teuthology.orchestra.run.trial132.stdout: Installing : python3-rgw-2:20.3.0-5312.gabffb617.el9.x86_64 26/39 2026-02-26T03:22:03.569 INFO:teuthology.orchestra.run.trial132.stdout: Installing : pciutils-3.7.0-7.el9.x86_64 27/39 2026-02-26T03:22:03.571 INFO:teuthology.orchestra.run.trial132.stdout: Installing : qatlib-25.08.0-2.el9.x86_64 28/39 2026-02-26T03:22:03.572 INFO:teuthology.orchestra.run.trial132.stdout: Installing : qatlib-service-25.08.0-2.el9.x86_64 29/39 2026-02-26T03:22:03.589 INFO:teuthology.orchestra.run.trial132.stdout: Running scriptlet: qatlib-service-25.08.0-2.el9.x86_64 29/39 2026-02-26T03:22:03.703 INFO:teuthology.orchestra.run.trial132.stdout: Installing : qatzip-libs-1.3.1-1.el9.x86_64 30/39 2026-02-26T03:22:03.705 INFO:teuthology.orchestra.run.trial132.stdout: Installing : nvme-cli-2.16-1.el9.x86_64 31/39 2026-02-26T03:22:03.991 INFO:teuthology.orchestra.run.trial132.stdout: Running scriptlet: nvme-cli-2.16-1.el9.x86_64 31/39 2026-02-26T03:22:03.991 INFO:teuthology.orchestra.run.trial132.stdout:Created symlink /etc/systemd/system/default.target.wants/nvmefc-boot-connections.service → /usr/lib/systemd/system/nvmefc-boot-connections.service. 2026-02-26T03:22:03.991 INFO:teuthology.orchestra.run.trial132.stdout: 2026-02-26T03:22:04.035 INFO:teuthology.orchestra.run.trial132.stdout: Installing : mailcap-2.1.49-5.el9.noarch 32/39 2026-02-26T03:22:04.037 INFO:teuthology.orchestra.run.trial132.stdout: Installing : python3-ceph-common-2:20.3.0-5312.gabffb617.el9.x8 33/39 2026-02-26T03:22:04.980 INFO:teuthology.orchestra.run.trial132.stdout: Running scriptlet: ceph-common-2:20.3.0-5312.gabffb617.el9.x86_64 34/39 2026-02-26T03:22:04.983 INFO:teuthology.orchestra.run.trial132.stdout: Installing : ceph-common-2:20.3.0-5312.gabffb617.el9.x86_64 34/39 2026-02-26T03:22:05.247 INFO:teuthology.orchestra.run.trial132.stdout: Running scriptlet: ceph-common-2:20.3.0-5312.gabffb617.el9.x86_64 34/39 2026-02-26T03:22:05.251 INFO:teuthology.orchestra.run.trial132.stdout: Installing : ceph-base-2:20.3.0-5312.gabffb617.el9.x86_64 35/39 2026-02-26T03:22:05.290 INFO:teuthology.orchestra.run.trial132.stdout: Running scriptlet: ceph-base-2:20.3.0-5312.gabffb617.el9.x86_64 35/39 2026-02-26T03:22:05.290 INFO:teuthology.orchestra.run.trial132.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2026-02-26T03:22:05.290 INFO:teuthology.orchestra.run.trial132.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2026-02-26T03:22:05.291 INFO:teuthology.orchestra.run.trial132.stdout: 2026-02-26T03:22:05.293 INFO:teuthology.orchestra.run.trial132.stdout: Installing : ceph-selinux-2:20.3.0-5312.gabffb617.el9.x86_64 36/39 2026-02-26T03:22:09.753 INFO:teuthology.orchestra.run.trial132.stdout: Running scriptlet: ceph-selinux-2:20.3.0-5312.gabffb617.el9.x86_64 36/39 2026-02-26T03:22:09.753 INFO:teuthology.orchestra.run.trial132.stdout:skipping the directory /sys 2026-02-26T03:22:09.753 INFO:teuthology.orchestra.run.trial132.stdout:skipping the directory /proc 2026-02-26T03:22:09.753 INFO:teuthology.orchestra.run.trial132.stdout:skipping the directory /mnt 2026-02-26T03:22:09.754 INFO:teuthology.orchestra.run.trial132.stdout:skipping the directory /var/tmp 2026-02-26T03:22:09.754 INFO:teuthology.orchestra.run.trial132.stdout:skipping the directory /home 2026-02-26T03:22:09.754 INFO:teuthology.orchestra.run.trial132.stdout:skipping the directory /root 2026-02-26T03:22:09.754 INFO:teuthology.orchestra.run.trial132.stdout:skipping the directory /tmp 2026-02-26T03:22:09.754 INFO:teuthology.orchestra.run.trial132.stdout: 2026-02-26T03:22:10.249 INFO:teuthology.orchestra.run.trial132.stdout: Installing : ceph-radosgw-2:20.3.0-5312.gabffb617.el9.x86_64 37/39 2026-02-26T03:22:10.273 INFO:teuthology.orchestra.run.trial132.stdout: Running scriptlet: ceph-radosgw-2:20.3.0-5312.gabffb617.el9.x86_64 37/39 2026-02-26T03:22:10.273 INFO:teuthology.orchestra.run.trial132.stdout:Glob pattern passed to enable, but globs are not supported for this. 2026-02-26T03:22:10.273 INFO:teuthology.orchestra.run.trial132.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2026-02-26T03:22:10.273 INFO:teuthology.orchestra.run.trial132.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2026-02-26T03:22:10.273 INFO:teuthology.orchestra.run.trial132.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2026-02-26T03:22:10.273 INFO:teuthology.orchestra.run.trial132.stdout: 2026-02-26T03:22:10.275 INFO:teuthology.orchestra.run.trial132.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 38/39 2026-02-26T03:22:10.286 INFO:teuthology.orchestra.run.trial132.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 38/39 2026-02-26T03:22:10.286 INFO:teuthology.orchestra.run.trial132.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 39/39 2026-02-26T03:22:10.340 INFO:teuthology.orchestra.run.trial132.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 39/39 2026-02-26T03:22:10.340 INFO:teuthology.orchestra.run.trial132.stdout:Creating group 'qat' with GID 992. 2026-02-26T03:22:10.340 INFO:teuthology.orchestra.run.trial132.stdout: 2026-02-26T03:22:10.630 INFO:teuthology.orchestra.run.trial132.stdout: Verifying : ceph-base-2:20.3.0-5312.gabffb617.el9.x86_64 1/39 2026-02-26T03:22:10.630 INFO:teuthology.orchestra.run.trial132.stdout: Verifying : ceph-common-2:20.3.0-5312.gabffb617.el9.x86_64 2/39 2026-02-26T03:22:10.630 INFO:teuthology.orchestra.run.trial132.stdout: Verifying : ceph-radosgw-2:20.3.0-5312.gabffb617.el9.x86_64 3/39 2026-02-26T03:22:10.631 INFO:teuthology.orchestra.run.trial132.stdout: Verifying : ceph-selinux-2:20.3.0-5312.gabffb617.el9.x86_64 4/39 2026-02-26T03:22:10.631 INFO:teuthology.orchestra.run.trial132.stdout: Verifying : libcephfs-daemon-2:20.3.0-5312.gabffb617.el9.x86_6 5/39 2026-02-26T03:22:10.631 INFO:teuthology.orchestra.run.trial132.stdout: Verifying : libcephfs-proxy2-2:20.3.0-5312.gabffb617.el9.x86_6 6/39 2026-02-26T03:22:10.631 INFO:teuthology.orchestra.run.trial132.stdout: Verifying : libcephfs2-2:20.3.0-5312.gabffb617.el9.x86_64 7/39 2026-02-26T03:22:10.631 INFO:teuthology.orchestra.run.trial132.stdout: Verifying : libradosstriper1-2:20.3.0-5312.gabffb617.el9.x86_6 8/39 2026-02-26T03:22:10.631 INFO:teuthology.orchestra.run.trial132.stdout: Verifying : librgw2-2:20.3.0-5312.gabffb617.el9.x86_64 9/39 2026-02-26T03:22:10.631 INFO:teuthology.orchestra.run.trial132.stdout: Verifying : python3-ceph-argparse-2:20.3.0-5312.gabffb617.el9. 10/39 2026-02-26T03:22:10.631 INFO:teuthology.orchestra.run.trial132.stdout: Verifying : python3-ceph-common-2:20.3.0-5312.gabffb617.el9.x8 11/39 2026-02-26T03:22:10.631 INFO:teuthology.orchestra.run.trial132.stdout: Verifying : python3-cephfs-2:20.3.0-5312.gabffb617.el9.x86_64 12/39 2026-02-26T03:22:10.631 INFO:teuthology.orchestra.run.trial132.stdout: Verifying : python3-rados-2:20.3.0-5312.gabffb617.el9.x86_64 13/39 2026-02-26T03:22:10.631 INFO:teuthology.orchestra.run.trial132.stdout: Verifying : python3-rbd-2:20.3.0-5312.gabffb617.el9.x86_64 14/39 2026-02-26T03:22:10.631 INFO:teuthology.orchestra.run.trial132.stdout: Verifying : python3-rgw-2:20.3.0-5312.gabffb617.el9.x86_64 15/39 2026-02-26T03:22:10.631 INFO:teuthology.orchestra.run.trial132.stdout: Verifying : mailcap-2.1.49-5.el9.noarch 16/39 2026-02-26T03:22:10.631 INFO:teuthology.orchestra.run.trial132.stdout: Verifying : nvme-cli-2.16-1.el9.x86_64 17/39 2026-02-26T03:22:10.631 INFO:teuthology.orchestra.run.trial132.stdout: Verifying : pciutils-3.7.0-7.el9.x86_64 18/39 2026-02-26T03:22:10.631 INFO:teuthology.orchestra.run.trial132.stdout: Verifying : boost-program-options-1.75.0-13.el9.x86_64 19/39 2026-02-26T03:22:10.631 INFO:teuthology.orchestra.run.trial132.stdout: Verifying : libnbd-1.20.3-4.el9.x86_64 20/39 2026-02-26T03:22:10.631 INFO:teuthology.orchestra.run.trial132.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 21/39 2026-02-26T03:22:10.631 INFO:teuthology.orchestra.run.trial132.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 22/39 2026-02-26T03:22:10.631 INFO:teuthology.orchestra.run.trial132.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 23/39 2026-02-26T03:22:10.631 INFO:teuthology.orchestra.run.trial132.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 24/39 2026-02-26T03:22:10.631 INFO:teuthology.orchestra.run.trial132.stdout: Verifying : qatlib-25.08.0-2.el9.x86_64 25/39 2026-02-26T03:22:10.631 INFO:teuthology.orchestra.run.trial132.stdout: Verifying : qatlib-service-25.08.0-2.el9.x86_64 26/39 2026-02-26T03:22:10.631 INFO:teuthology.orchestra.run.trial132.stdout: Verifying : qatzip-libs-1.3.1-1.el9.x86_64 27/39 2026-02-26T03:22:10.631 INFO:teuthology.orchestra.run.trial132.stdout: Verifying : gperftools-libs-2.9.1-3.el9.x86_64 28/39 2026-02-26T03:22:10.631 INFO:teuthology.orchestra.run.trial132.stdout: Verifying : libarrow-9.0.0-15.el9.x86_64 29/39 2026-02-26T03:22:10.631 INFO:teuthology.orchestra.run.trial132.stdout: Verifying : libarrow-doc-9.0.0-15.el9.noarch 30/39 2026-02-26T03:22:10.631 INFO:teuthology.orchestra.run.trial132.stdout: Verifying : liboath-2.6.12-1.el9.x86_64 31/39 2026-02-26T03:22:10.631 INFO:teuthology.orchestra.run.trial132.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 32/39 2026-02-26T03:22:10.631 INFO:teuthology.orchestra.run.trial132.stdout: Verifying : parquet-libs-9.0.0-15.el9.x86_64 33/39 2026-02-26T03:22:10.631 INFO:teuthology.orchestra.run.trial132.stdout: Verifying : re2-1:20211101-20.el9.x86_64 34/39 2026-02-26T03:22:10.631 INFO:teuthology.orchestra.run.trial132.stdout: Verifying : thrift-0.15.0-4.el9.x86_64 35/39 2026-02-26T03:22:10.631 INFO:teuthology.orchestra.run.trial132.stdout: Verifying : librados2-2:20.3.0-5312.gabffb617.el9.x86_64 36/39 2026-02-26T03:22:10.631 INFO:teuthology.orchestra.run.trial132.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 37/39 2026-02-26T03:22:10.631 INFO:teuthology.orchestra.run.trial132.stdout: Verifying : librbd1-2:20.3.0-5312.gabffb617.el9.x86_64 38/39 2026-02-26T03:22:10.700 INFO:teuthology.orchestra.run.trial132.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 39/39 2026-02-26T03:22:10.700 INFO:teuthology.orchestra.run.trial132.stdout: 2026-02-26T03:22:10.700 INFO:teuthology.orchestra.run.trial132.stdout:Upgraded: 2026-02-26T03:22:10.700 INFO:teuthology.orchestra.run.trial132.stdout: librados2-2:20.3.0-5312.gabffb617.el9.x86_64 2026-02-26T03:22:10.700 INFO:teuthology.orchestra.run.trial132.stdout: librbd1-2:20.3.0-5312.gabffb617.el9.x86_64 2026-02-26T03:22:10.700 INFO:teuthology.orchestra.run.trial132.stdout:Installed: 2026-02-26T03:22:10.700 INFO:teuthology.orchestra.run.trial132.stdout: boost-program-options-1.75.0-13.el9.x86_64 2026-02-26T03:22:10.700 INFO:teuthology.orchestra.run.trial132.stdout: ceph-base-2:20.3.0-5312.gabffb617.el9.x86_64 2026-02-26T03:22:10.701 INFO:teuthology.orchestra.run.trial132.stdout: ceph-common-2:20.3.0-5312.gabffb617.el9.x86_64 2026-02-26T03:22:10.701 INFO:teuthology.orchestra.run.trial132.stdout: ceph-radosgw-2:20.3.0-5312.gabffb617.el9.x86_64 2026-02-26T03:22:10.701 INFO:teuthology.orchestra.run.trial132.stdout: ceph-selinux-2:20.3.0-5312.gabffb617.el9.x86_64 2026-02-26T03:22:10.701 INFO:teuthology.orchestra.run.trial132.stdout: gperftools-libs-2.9.1-3.el9.x86_64 2026-02-26T03:22:10.701 INFO:teuthology.orchestra.run.trial132.stdout: libarrow-9.0.0-15.el9.x86_64 2026-02-26T03:22:10.701 INFO:teuthology.orchestra.run.trial132.stdout: libarrow-doc-9.0.0-15.el9.noarch 2026-02-26T03:22:10.701 INFO:teuthology.orchestra.run.trial132.stdout: libcephfs-daemon-2:20.3.0-5312.gabffb617.el9.x86_64 2026-02-26T03:22:10.701 INFO:teuthology.orchestra.run.trial132.stdout: libcephfs-proxy2-2:20.3.0-5312.gabffb617.el9.x86_64 2026-02-26T03:22:10.701 INFO:teuthology.orchestra.run.trial132.stdout: libcephfs2-2:20.3.0-5312.gabffb617.el9.x86_64 2026-02-26T03:22:10.701 INFO:teuthology.orchestra.run.trial132.stdout: libnbd-1.20.3-4.el9.x86_64 2026-02-26T03:22:10.701 INFO:teuthology.orchestra.run.trial132.stdout: liboath-2.6.12-1.el9.x86_64 2026-02-26T03:22:10.701 INFO:teuthology.orchestra.run.trial132.stdout: libpmemobj-1.12.1-1.el9.x86_64 2026-02-26T03:22:10.702 INFO:teuthology.orchestra.run.trial132.stdout: librabbitmq-0.11.0-7.el9.x86_64 2026-02-26T03:22:10.702 INFO:teuthology.orchestra.run.trial132.stdout: libradosstriper1-2:20.3.0-5312.gabffb617.el9.x86_64 2026-02-26T03:22:10.702 INFO:teuthology.orchestra.run.trial132.stdout: librdkafka-1.6.1-102.el9.x86_64 2026-02-26T03:22:10.702 INFO:teuthology.orchestra.run.trial132.stdout: librgw2-2:20.3.0-5312.gabffb617.el9.x86_64 2026-02-26T03:22:10.702 INFO:teuthology.orchestra.run.trial132.stdout: libunwind-1.6.2-1.el9.x86_64 2026-02-26T03:22:10.702 INFO:teuthology.orchestra.run.trial132.stdout: lttng-ust-2.12.0-6.el9.x86_64 2026-02-26T03:22:10.702 INFO:teuthology.orchestra.run.trial132.stdout: mailcap-2.1.49-5.el9.noarch 2026-02-26T03:22:10.702 INFO:teuthology.orchestra.run.trial132.stdout: nvme-cli-2.16-1.el9.x86_64 2026-02-26T03:22:10.702 INFO:teuthology.orchestra.run.trial132.stdout: parquet-libs-9.0.0-15.el9.x86_64 2026-02-26T03:22:10.702 INFO:teuthology.orchestra.run.trial132.stdout: pciutils-3.7.0-7.el9.x86_64 2026-02-26T03:22:10.702 INFO:teuthology.orchestra.run.trial132.stdout: python3-ceph-argparse-2:20.3.0-5312.gabffb617.el9.x86_64 2026-02-26T03:22:10.702 INFO:teuthology.orchestra.run.trial132.stdout: python3-ceph-common-2:20.3.0-5312.gabffb617.el9.x86_64 2026-02-26T03:22:10.702 INFO:teuthology.orchestra.run.trial132.stdout: python3-cephfs-2:20.3.0-5312.gabffb617.el9.x86_64 2026-02-26T03:22:10.703 INFO:teuthology.orchestra.run.trial132.stdout: python3-rados-2:20.3.0-5312.gabffb617.el9.x86_64 2026-02-26T03:22:10.703 INFO:teuthology.orchestra.run.trial132.stdout: python3-rbd-2:20.3.0-5312.gabffb617.el9.x86_64 2026-02-26T03:22:10.703 INFO:teuthology.orchestra.run.trial132.stdout: python3-rgw-2:20.3.0-5312.gabffb617.el9.x86_64 2026-02-26T03:22:10.703 INFO:teuthology.orchestra.run.trial132.stdout: qatlib-25.08.0-2.el9.x86_64 2026-02-26T03:22:10.703 INFO:teuthology.orchestra.run.trial132.stdout: qatlib-service-25.08.0-2.el9.x86_64 2026-02-26T03:22:10.703 INFO:teuthology.orchestra.run.trial132.stdout: qatzip-libs-1.3.1-1.el9.x86_64 2026-02-26T03:22:10.703 INFO:teuthology.orchestra.run.trial132.stdout: re2-1:20211101-20.el9.x86_64 2026-02-26T03:22:10.703 INFO:teuthology.orchestra.run.trial132.stdout: thrift-0.15.0-4.el9.x86_64 2026-02-26T03:22:10.703 INFO:teuthology.orchestra.run.trial132.stdout: 2026-02-26T03:22:10.703 INFO:teuthology.orchestra.run.trial132.stdout:Complete! 2026-02-26T03:22:10.740 DEBUG:teuthology.orchestra.run.trial132:> sudo yum -y install ceph-test 2026-02-26T03:22:11.058 INFO:teuthology.orchestra.run.trial132.stdout:Last metadata expiration check: 0:00:51 ago on Thu 26 Feb 2026 03:21:20 AM UTC. 2026-02-26T03:22:11.142 INFO:teuthology.orchestra.run.trial132.stdout:Dependencies resolved. 2026-02-26T03:22:11.143 INFO:teuthology.orchestra.run.trial132.stdout:================================================================================ 2026-02-26T03:22:11.144 INFO:teuthology.orchestra.run.trial132.stdout: Package Arch Version Repository Size 2026-02-26T03:22:11.144 INFO:teuthology.orchestra.run.trial132.stdout:================================================================================ 2026-02-26T03:22:11.144 INFO:teuthology.orchestra.run.trial132.stdout:Installing: 2026-02-26T03:22:11.144 INFO:teuthology.orchestra.run.trial132.stdout: ceph-test x86_64 2:20.3.0-5312.gabffb617.el9 ceph 126 M 2026-02-26T03:22:11.144 INFO:teuthology.orchestra.run.trial132.stdout:Installing dependencies: 2026-02-26T03:22:11.144 INFO:teuthology.orchestra.run.trial132.stdout: hwloc-libs x86_64 2.4.1-6.el9 baseos 2.1 M 2026-02-26T03:22:11.144 INFO:teuthology.orchestra.run.trial132.stdout: libasan8 x86_64 15.2.1-7.el9 appstream 525 k 2026-02-26T03:22:11.144 INFO:teuthology.orchestra.run.trial132.stdout: libatomic x86_64 11.5.0-14.el9 baseos 23 k 2026-02-26T03:22:11.144 INFO:teuthology.orchestra.run.trial132.stdout: libcephsqlite x86_64 2:20.3.0-5312.gabffb617.el9 ceph 153 k 2026-02-26T03:22:11.144 INFO:teuthology.orchestra.run.trial132.stdout: libubsan x86_64 11.5.0-14.el9 appstream 176 k 2026-02-26T03:22:11.144 INFO:teuthology.orchestra.run.trial132.stdout: libxslt x86_64 1.1.34-12.el9 appstream 233 k 2026-02-26T03:22:11.145 INFO:teuthology.orchestra.run.trial132.stdout: lksctp-tools x86_64 1.0.19-2.el9 baseos 94 k 2026-02-26T03:22:11.145 INFO:teuthology.orchestra.run.trial132.stdout: protobuf x86_64 3.14.0-17.el9 appstream 1.0 M 2026-02-26T03:22:11.145 INFO:teuthology.orchestra.run.trial132.stdout: socat x86_64 1.7.4.1-8.el9 appstream 303 k 2026-02-26T03:22:11.145 INFO:teuthology.orchestra.run.trial132.stdout: xmlstarlet x86_64 1.6.1-20.el9 appstream 64 k 2026-02-26T03:22:11.145 INFO:teuthology.orchestra.run.trial132.stdout: yaml-cpp x86_64 0.6.3-5.el9 epel 122 k 2026-02-26T03:22:11.145 INFO:teuthology.orchestra.run.trial132.stdout: 2026-02-26T03:22:11.145 INFO:teuthology.orchestra.run.trial132.stdout:Transaction Summary 2026-02-26T03:22:11.145 INFO:teuthology.orchestra.run.trial132.stdout:================================================================================ 2026-02-26T03:22:11.145 INFO:teuthology.orchestra.run.trial132.stdout:Install 12 Packages 2026-02-26T03:22:11.145 INFO:teuthology.orchestra.run.trial132.stdout: 2026-02-26T03:22:11.145 INFO:teuthology.orchestra.run.trial132.stdout:Total download size: 131 M 2026-02-26T03:22:11.146 INFO:teuthology.orchestra.run.trial132.stdout:Installed size: 976 M 2026-02-26T03:22:11.146 INFO:teuthology.orchestra.run.trial132.stdout:Downloading Packages: 2026-02-26T03:22:12.801 INFO:teuthology.orchestra.run.trial132.stdout:(1/12): hwloc-libs-2.4.1-6.el9.x86_64.rpm 14 MB/s | 2.1 MB 00:00 2026-02-26T03:22:12.823 INFO:teuthology.orchestra.run.trial132.stdout:(2/12): libatomic-11.5.0-14.el9.x86_64.rpm 1.0 MB/s | 23 kB 00:00 2026-02-26T03:22:12.846 INFO:teuthology.orchestra.run.trial132.stdout:(3/12): lksctp-tools-1.0.19-2.el9.x86_64.rpm 4.1 MB/s | 94 kB 00:00 2026-02-26T03:22:12.917 INFO:teuthology.orchestra.run.trial132.stdout:(4/12): libasan8-15.2.1-7.el9.x86_64.rpm 7.1 MB/s | 525 kB 00:00 2026-02-26T03:22:12.942 INFO:teuthology.orchestra.run.trial132.stdout:(5/12): libubsan-11.5.0-14.el9.x86_64.rpm 7.0 MB/s | 176 kB 00:00 2026-02-26T03:22:12.964 INFO:teuthology.orchestra.run.trial132.stdout:(6/12): libxslt-1.1.34-12.el9.x86_64.rpm 10 MB/s | 233 kB 00:00 2026-02-26T03:22:12.988 INFO:teuthology.orchestra.run.trial132.stdout:(7/12): protobuf-3.14.0-17.el9.x86_64.rpm 42 MB/s | 1.0 MB 00:00 2026-02-26T03:22:13.011 INFO:teuthology.orchestra.run.trial132.stdout:(8/12): socat-1.7.4.1-8.el9.x86_64.rpm 13 MB/s | 303 kB 00:00 2026-02-26T03:22:13.033 INFO:teuthology.orchestra.run.trial132.stdout:(9/12): xmlstarlet-1.6.1-20.el9.x86_64.rpm 2.8 MB/s | 64 kB 00:00 2026-02-26T03:22:13.236 INFO:teuthology.orchestra.run.trial132.stdout:(10/12): libcephsqlite-20.3.0-5312.gabffb617.el 262 kB/s | 153 kB 00:00 2026-02-26T03:22:13.450 INFO:teuthology.orchestra.run.trial132.stdout:(11/12): yaml-cpp-0.6.3-5.el9.x86_64.rpm 292 kB/s | 122 kB 00:00 2026-02-26T03:26:29.219 INFO:teuthology.orchestra.run.trial132.stdout:(12/12): ceph-test-20.3.0-5312.gabffb617.el9.x8 503 kB/s | 126 MB 04:16 2026-02-26T03:26:29.221 INFO:teuthology.orchestra.run.trial132.stdout:-------------------------------------------------------------------------------- 2026-02-26T03:26:29.221 INFO:teuthology.orchestra.run.trial132.stdout:Total 519 kB/s | 131 MB 04:18 2026-02-26T03:26:29.289 INFO:teuthology.orchestra.run.trial132.stdout:Running transaction check 2026-02-26T03:26:29.302 INFO:teuthology.orchestra.run.trial132.stdout:Transaction check succeeded. 2026-02-26T03:26:29.302 INFO:teuthology.orchestra.run.trial132.stdout:Running transaction test 2026-02-26T03:26:29.729 INFO:teuthology.orchestra.run.trial132.stdout:Transaction test succeeded. 2026-02-26T03:26:29.729 INFO:teuthology.orchestra.run.trial132.stdout:Running transaction 2026-02-26T03:26:30.183 INFO:teuthology.orchestra.run.trial132.stdout: Preparing : 1/1 2026-02-26T03:26:30.192 INFO:teuthology.orchestra.run.trial132.stdout: Installing : yaml-cpp-0.6.3-5.el9.x86_64 1/12 2026-02-26T03:26:30.201 INFO:teuthology.orchestra.run.trial132.stdout: Installing : socat-1.7.4.1-8.el9.x86_64 2/12 2026-02-26T03:26:30.206 INFO:teuthology.orchestra.run.trial132.stdout: Installing : protobuf-3.14.0-17.el9.x86_64 3/12 2026-02-26T03:26:30.210 INFO:teuthology.orchestra.run.trial132.stdout: Installing : libxslt-1.1.34-12.el9.x86_64 4/12 2026-02-26T03:26:30.212 INFO:teuthology.orchestra.run.trial132.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 5/12 2026-02-26T03:26:30.216 INFO:teuthology.orchestra.run.trial132.stdout: Installing : libubsan-11.5.0-14.el9.x86_64 6/12 2026-02-26T03:26:30.217 INFO:teuthology.orchestra.run.trial132.stdout: Installing : libasan8-15.2.1-7.el9.x86_64 7/12 2026-02-26T03:26:30.232 INFO:teuthology.orchestra.run.trial132.stdout: Running scriptlet: libasan8-15.2.1-7.el9.x86_64 7/12 2026-02-26T03:26:30.236 INFO:teuthology.orchestra.run.trial132.stdout: Installing : lksctp-tools-1.0.19-2.el9.x86_64 8/12 2026-02-26T03:26:30.247 INFO:teuthology.orchestra.run.trial132.stdout: Installing : libatomic-11.5.0-14.el9.x86_64 9/12 2026-02-26T03:26:30.255 INFO:teuthology.orchestra.run.trial132.stdout: Installing : hwloc-libs-2.4.1-6.el9.x86_64 10/12 2026-02-26T03:26:30.256 INFO:teuthology.orchestra.run.trial132.stdout: Installing : libcephsqlite-2:20.3.0-5312.gabffb617.el9.x86_64 11/12 2026-02-26T03:26:37.324 INFO:teuthology.orchestra.run.trial132.stdout: Running scriptlet: libcephsqlite-2:20.3.0-5312.gabffb617.el9.x86_64 11/12 2026-02-26T03:26:37.342 INFO:teuthology.orchestra.run.trial132.stdout: Installing : ceph-test-2:20.3.0-5312.gabffb617.el9.x86_64 12/12 2026-02-26T03:26:38.085 INFO:teuthology.orchestra.run.trial132.stdout: Running scriptlet: ceph-test-2:20.3.0-5312.gabffb617.el9.x86_64 12/12 2026-02-26T03:26:38.086 INFO:teuthology.orchestra.run.trial132.stdout: Verifying : ceph-test-2:20.3.0-5312.gabffb617.el9.x86_64 1/12 2026-02-26T03:26:38.086 INFO:teuthology.orchestra.run.trial132.stdout: Verifying : libcephsqlite-2:20.3.0-5312.gabffb617.el9.x86_64 2/12 2026-02-26T03:26:38.086 INFO:teuthology.orchestra.run.trial132.stdout: Verifying : hwloc-libs-2.4.1-6.el9.x86_64 3/12 2026-02-26T03:26:38.086 INFO:teuthology.orchestra.run.trial132.stdout: Verifying : libatomic-11.5.0-14.el9.x86_64 4/12 2026-02-26T03:26:38.086 INFO:teuthology.orchestra.run.trial132.stdout: Verifying : lksctp-tools-1.0.19-2.el9.x86_64 5/12 2026-02-26T03:26:38.086 INFO:teuthology.orchestra.run.trial132.stdout: Verifying : libasan8-15.2.1-7.el9.x86_64 6/12 2026-02-26T03:26:38.086 INFO:teuthology.orchestra.run.trial132.stdout: Verifying : libubsan-11.5.0-14.el9.x86_64 7/12 2026-02-26T03:26:38.086 INFO:teuthology.orchestra.run.trial132.stdout: Verifying : libxslt-1.1.34-12.el9.x86_64 8/12 2026-02-26T03:26:38.086 INFO:teuthology.orchestra.run.trial132.stdout: Verifying : protobuf-3.14.0-17.el9.x86_64 9/12 2026-02-26T03:26:38.086 INFO:teuthology.orchestra.run.trial132.stdout: Verifying : socat-1.7.4.1-8.el9.x86_64 10/12 2026-02-26T03:26:38.086 INFO:teuthology.orchestra.run.trial132.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 11/12 2026-02-26T03:26:38.146 INFO:teuthology.orchestra.run.trial132.stdout: Verifying : yaml-cpp-0.6.3-5.el9.x86_64 12/12 2026-02-26T03:26:38.146 INFO:teuthology.orchestra.run.trial132.stdout: 2026-02-26T03:26:38.146 INFO:teuthology.orchestra.run.trial132.stdout:Installed: 2026-02-26T03:26:38.146 INFO:teuthology.orchestra.run.trial132.stdout: ceph-test-2:20.3.0-5312.gabffb617.el9.x86_64 2026-02-26T03:26:38.147 INFO:teuthology.orchestra.run.trial132.stdout: hwloc-libs-2.4.1-6.el9.x86_64 2026-02-26T03:26:38.147 INFO:teuthology.orchestra.run.trial132.stdout: libasan8-15.2.1-7.el9.x86_64 2026-02-26T03:26:38.147 INFO:teuthology.orchestra.run.trial132.stdout: libatomic-11.5.0-14.el9.x86_64 2026-02-26T03:26:38.147 INFO:teuthology.orchestra.run.trial132.stdout: libcephsqlite-2:20.3.0-5312.gabffb617.el9.x86_64 2026-02-26T03:26:38.147 INFO:teuthology.orchestra.run.trial132.stdout: libubsan-11.5.0-14.el9.x86_64 2026-02-26T03:26:38.147 INFO:teuthology.orchestra.run.trial132.stdout: libxslt-1.1.34-12.el9.x86_64 2026-02-26T03:26:38.147 INFO:teuthology.orchestra.run.trial132.stdout: lksctp-tools-1.0.19-2.el9.x86_64 2026-02-26T03:26:38.147 INFO:teuthology.orchestra.run.trial132.stdout: protobuf-3.14.0-17.el9.x86_64 2026-02-26T03:26:38.147 INFO:teuthology.orchestra.run.trial132.stdout: socat-1.7.4.1-8.el9.x86_64 2026-02-26T03:26:38.147 INFO:teuthology.orchestra.run.trial132.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2026-02-26T03:26:38.147 INFO:teuthology.orchestra.run.trial132.stdout: yaml-cpp-0.6.3-5.el9.x86_64 2026-02-26T03:26:38.147 INFO:teuthology.orchestra.run.trial132.stdout: 2026-02-26T03:26:38.147 INFO:teuthology.orchestra.run.trial132.stdout:Complete! 2026-02-26T03:26:38.188 DEBUG:teuthology.orchestra.run.trial132:> sudo yum -y install ceph 2026-02-26T03:26:38.503 INFO:teuthology.orchestra.run.trial132.stdout:Last metadata expiration check: 0:05:18 ago on Thu 26 Feb 2026 03:21:20 AM UTC. 2026-02-26T03:26:38.591 INFO:teuthology.orchestra.run.trial132.stdout:Dependencies resolved. 2026-02-26T03:26:38.594 INFO:teuthology.orchestra.run.trial132.stdout:======================================================================================= 2026-02-26T03:26:38.594 INFO:teuthology.orchestra.run.trial132.stdout: Package Arch Version Repository Size 2026-02-26T03:26:38.594 INFO:teuthology.orchestra.run.trial132.stdout:======================================================================================= 2026-02-26T03:26:38.594 INFO:teuthology.orchestra.run.trial132.stdout:Installing: 2026-02-26T03:26:38.595 INFO:teuthology.orchestra.run.trial132.stdout: ceph x86_64 2:20.3.0-5312.gabffb617.el9 ceph 6.5 k 2026-02-26T03:26:38.595 INFO:teuthology.orchestra.run.trial132.stdout:Installing dependencies: 2026-02-26T03:26:38.595 INFO:teuthology.orchestra.run.trial132.stdout: abseil-cpp x86_64 20211102.0-4.el9 epel 551 k 2026-02-26T03:26:38.595 INFO:teuthology.orchestra.run.trial132.stdout: ceph-grafana-dashboards noarch 2:20.3.0-5312.gabffb617.el9 ceph-noarch 57 k 2026-02-26T03:26:38.595 INFO:teuthology.orchestra.run.trial132.stdout: ceph-mds x86_64 2:20.3.0-5312.gabffb617.el9 ceph 2.0 M 2026-02-26T03:26:38.595 INFO:teuthology.orchestra.run.trial132.stdout: ceph-mgr x86_64 2:20.3.0-5312.gabffb617.el9 ceph 937 k 2026-02-26T03:26:38.595 INFO:teuthology.orchestra.run.trial132.stdout: ceph-mgr-modules-core noarch 2:20.3.0-5312.gabffb617.el9 ceph-noarch 297 k 2026-02-26T03:26:38.595 INFO:teuthology.orchestra.run.trial132.stdout: ceph-mon x86_64 2:20.3.0-5312.gabffb617.el9 ceph 4.9 M 2026-02-26T03:26:38.595 INFO:teuthology.orchestra.run.trial132.stdout: ceph-osd x86_64 2:20.3.0-5312.gabffb617.el9 ceph 163 k 2026-02-26T03:26:38.595 INFO:teuthology.orchestra.run.trial132.stdout: ceph-osd-classic x86_64 2:20.3.0-5312.gabffb617.el9 ceph 15 M 2026-02-26T03:26:38.595 INFO:teuthology.orchestra.run.trial132.stdout: ceph-osd-crimson x86_64 2:20.3.0-5312.gabffb617.el9 ceph 87 M 2026-02-26T03:26:38.595 INFO:teuthology.orchestra.run.trial132.stdout: ceph-prometheus-alerts noarch 2:20.3.0-5312.gabffb617.el9 ceph-noarch 17 k 2026-02-26T03:26:38.595 INFO:teuthology.orchestra.run.trial132.stdout: cephadm noarch 2:20.3.0-5312.gabffb617.el9 ceph-noarch 1.0 M 2026-02-26T03:26:38.596 INFO:teuthology.orchestra.run.trial132.stdout: cryptsetup x86_64 2.8.1-3.el9 baseos 351 k 2026-02-26T03:26:38.596 INFO:teuthology.orchestra.run.trial132.stdout: flexiblas x86_64 3.0.4-9.el9 appstream 30 k 2026-02-26T03:26:38.596 INFO:teuthology.orchestra.run.trial132.stdout: flexiblas-netlib x86_64 3.0.4-9.el9 appstream 3.0 M 2026-02-26T03:26:38.596 INFO:teuthology.orchestra.run.trial132.stdout: flexiblas-openblas-openmp x86_64 3.0.4-9.el9 appstream 15 k 2026-02-26T03:26:38.596 INFO:teuthology.orchestra.run.trial132.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2026-02-26T03:26:38.596 INFO:teuthology.orchestra.run.trial132.stdout: ledmon-libs x86_64 1.1.0-3.el9 baseos 40 k 2026-02-26T03:26:38.596 INFO:teuthology.orchestra.run.trial132.stdout: libconfig x86_64 1.7.2-9.el9 baseos 72 k 2026-02-26T03:26:38.596 INFO:teuthology.orchestra.run.trial132.stdout: libgfortran x86_64 11.5.0-14.el9 baseos 794 k 2026-02-26T03:26:38.596 INFO:teuthology.orchestra.run.trial132.stdout: libquadmath x86_64 11.5.0-14.el9 baseos 184 k 2026-02-26T03:26:38.596 INFO:teuthology.orchestra.run.trial132.stdout: libstoragemgmt x86_64 1.10.1-1.el9 appstream 246 k 2026-02-26T03:26:38.596 INFO:teuthology.orchestra.run.trial132.stdout: lua x86_64 5.4.4-4.el9 appstream 188 k 2026-02-26T03:26:38.596 INFO:teuthology.orchestra.run.trial132.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2026-02-26T03:26:38.596 INFO:teuthology.orchestra.run.trial132.stdout: luarocks noarch 3.9.2-5.el9 epel 151 k 2026-02-26T03:26:38.597 INFO:teuthology.orchestra.run.trial132.stdout: openblas x86_64 0.3.29-1.el9 appstream 42 k 2026-02-26T03:26:38.597 INFO:teuthology.orchestra.run.trial132.stdout: openblas-openmp x86_64 0.3.29-1.el9 appstream 5.3 M 2026-02-26T03:26:38.597 INFO:teuthology.orchestra.run.trial132.stdout: protobuf-compiler x86_64 3.14.0-17.el9 crb 862 k 2026-02-26T03:26:38.597 INFO:teuthology.orchestra.run.trial132.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2026-02-26T03:26:38.597 INFO:teuthology.orchestra.run.trial132.stdout: python3-autocommand noarch 2.2.2-8.el9 epel 29 k 2026-02-26T03:26:38.597 INFO:teuthology.orchestra.run.trial132.stdout: python3-babel noarch 2.9.1-2.el9 appstream 6.0 M 2026-02-26T03:26:38.597 INFO:teuthology.orchestra.run.trial132.stdout: python3-backports-tarfile noarch 1.2.0-1.el9 epel 60 k 2026-02-26T03:26:38.597 INFO:teuthology.orchestra.run.trial132.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2026-02-26T03:26:38.597 INFO:teuthology.orchestra.run.trial132.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2026-02-26T03:26:38.597 INFO:teuthology.orchestra.run.trial132.stdout: python3-certifi noarch 2023.05.07-4.el9 epel 14 k 2026-02-26T03:26:38.597 INFO:teuthology.orchestra.run.trial132.stdout: python3-cffi x86_64 1.14.5-5.el9 baseos 253 k 2026-02-26T03:26:38.597 INFO:teuthology.orchestra.run.trial132.stdout: python3-cheroot noarch 10.0.1-4.el9 epel 173 k 2026-02-26T03:26:38.597 INFO:teuthology.orchestra.run.trial132.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2026-02-26T03:26:38.597 INFO:teuthology.orchestra.run.trial132.stdout: python3-cryptography x86_64 36.0.1-5.el9 baseos 1.2 M 2026-02-26T03:26:38.597 INFO:teuthology.orchestra.run.trial132.stdout: python3-devel x86_64 3.9.25-3.el9 appstream 244 k 2026-02-26T03:26:38.598 INFO:teuthology.orchestra.run.trial132.stdout: python3-google-auth noarch 1:2.45.0-1.el9 epel 254 k 2026-02-26T03:26:38.598 INFO:teuthology.orchestra.run.trial132.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2026-02-26T03:26:38.598 INFO:teuthology.orchestra.run.trial132.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2026-02-26T03:26:38.598 INFO:teuthology.orchestra.run.trial132.stdout: python3-isodate noarch 0.6.1-3.el9 epel 56 k 2026-02-26T03:26:38.598 INFO:teuthology.orchestra.run.trial132.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2026-02-26T03:26:38.598 INFO:teuthology.orchestra.run.trial132.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2026-02-26T03:26:38.598 INFO:teuthology.orchestra.run.trial132.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2026-02-26T03:26:38.598 INFO:teuthology.orchestra.run.trial132.stdout: python3-jaraco-context noarch 6.0.1-3.el9 epel 20 k 2026-02-26T03:26:38.598 INFO:teuthology.orchestra.run.trial132.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2026-02-26T03:26:38.598 INFO:teuthology.orchestra.run.trial132.stdout: python3-jaraco-text noarch 4.0.0-2.el9 epel 26 k 2026-02-26T03:26:38.598 INFO:teuthology.orchestra.run.trial132.stdout: python3-jinja2 noarch 2.11.3-8.el9 appstream 249 k 2026-02-26T03:26:38.598 INFO:teuthology.orchestra.run.trial132.stdout: python3-jmespath noarch 1.0.1-1.el9 appstream 48 k 2026-02-26T03:26:38.598 INFO:teuthology.orchestra.run.trial132.stdout: python3-kubernetes noarch 1:26.1.0-3.el9 epel 1.0 M 2026-02-26T03:26:38.598 INFO:teuthology.orchestra.run.trial132.stdout: python3-libstoragemgmt x86_64 1.10.1-1.el9 appstream 177 k 2026-02-26T03:26:38.598 INFO:teuthology.orchestra.run.trial132.stdout: python3-lxml x86_64 4.6.5-3.el9 appstream 1.2 M 2026-02-26T03:26:38.599 INFO:teuthology.orchestra.run.trial132.stdout: python3-markupsafe x86_64 1.1.1-12.el9 appstream 35 k 2026-02-26T03:26:38.599 INFO:teuthology.orchestra.run.trial132.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2026-02-26T03:26:38.599 INFO:teuthology.orchestra.run.trial132.stdout: python3-msgpack x86_64 1.0.3-2.el9 epel 86 k 2026-02-26T03:26:38.599 INFO:teuthology.orchestra.run.trial132.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2026-02-26T03:26:38.599 INFO:teuthology.orchestra.run.trial132.stdout: python3-numpy x86_64 1:1.23.5-2.el9 appstream 6.1 M 2026-02-26T03:26:38.599 INFO:teuthology.orchestra.run.trial132.stdout: python3-numpy-f2py x86_64 1:1.23.5-2.el9 appstream 442 k 2026-02-26T03:26:38.599 INFO:teuthology.orchestra.run.trial132.stdout: python3-packaging noarch 20.9-5.el9 appstream 77 k 2026-02-26T03:26:38.599 INFO:teuthology.orchestra.run.trial132.stdout: python3-ply noarch 3.11-14.el9 baseos 106 k 2026-02-26T03:26:38.599 INFO:teuthology.orchestra.run.trial132.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2026-02-26T03:26:38.599 INFO:teuthology.orchestra.run.trial132.stdout: python3-protobuf noarch 3.14.0-17.el9 appstream 267 k 2026-02-26T03:26:38.599 INFO:teuthology.orchestra.run.trial132.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2026-02-26T03:26:38.599 INFO:teuthology.orchestra.run.trial132.stdout: python3-pyasn1 noarch 0.4.8-7.el9 appstream 157 k 2026-02-26T03:26:38.599 INFO:teuthology.orchestra.run.trial132.stdout: python3-pyasn1-modules noarch 0.4.8-7.el9 appstream 277 k 2026-02-26T03:26:38.599 INFO:teuthology.orchestra.run.trial132.stdout: python3-pycparser noarch 2.20-6.el9 baseos 135 k 2026-02-26T03:26:38.599 INFO:teuthology.orchestra.run.trial132.stdout: python3-pyparsing noarch 2.4.7-9.el9 baseos 150 k 2026-02-26T03:26:38.600 INFO:teuthology.orchestra.run.trial132.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2026-02-26T03:26:38.600 INFO:teuthology.orchestra.run.trial132.stdout: python3-requests noarch 2.25.1-10.el9 baseos 126 k 2026-02-26T03:26:38.600 INFO:teuthology.orchestra.run.trial132.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 appstream 54 k 2026-02-26T03:26:38.600 INFO:teuthology.orchestra.run.trial132.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2026-02-26T03:26:38.600 INFO:teuthology.orchestra.run.trial132.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2026-02-26T03:26:38.600 INFO:teuthology.orchestra.run.trial132.stdout: python3-scipy x86_64 1.9.3-2.el9 appstream 19 M 2026-02-26T03:26:38.600 INFO:teuthology.orchestra.run.trial132.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2026-02-26T03:26:38.600 INFO:teuthology.orchestra.run.trial132.stdout: python3-toml noarch 0.10.2-6.el9 appstream 42 k 2026-02-26T03:26:38.600 INFO:teuthology.orchestra.run.trial132.stdout: python3-typing-extensions noarch 4.15.0-1.el9 epel 86 k 2026-02-26T03:26:38.600 INFO:teuthology.orchestra.run.trial132.stdout: python3-urllib3 noarch 1.26.5-7.el9 baseos 218 k 2026-02-26T03:26:38.600 INFO:teuthology.orchestra.run.trial132.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2026-02-26T03:26:38.600 INFO:teuthology.orchestra.run.trial132.stdout: python3-xmlsec x86_64 1.3.13-1.el9 epel 48 k 2026-02-26T03:26:38.600 INFO:teuthology.orchestra.run.trial132.stdout: python3-xmltodict noarch 0.12.0-15.el9 epel 22 k 2026-02-26T03:26:38.600 INFO:teuthology.orchestra.run.trial132.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2026-02-26T03:26:38.601 INFO:teuthology.orchestra.run.trial132.stdout: unzip x86_64 6.0-59.el9 baseos 182 k 2026-02-26T03:26:38.601 INFO:teuthology.orchestra.run.trial132.stdout: xmlsec1 x86_64 1.2.29-13.el9 appstream 189 k 2026-02-26T03:26:38.601 INFO:teuthology.orchestra.run.trial132.stdout: xmlsec1-openssl x86_64 1.2.29-13.el9 appstream 90 k 2026-02-26T03:26:38.601 INFO:teuthology.orchestra.run.trial132.stdout: zip x86_64 3.0-35.el9 baseos 266 k 2026-02-26T03:26:38.601 INFO:teuthology.orchestra.run.trial132.stdout:Installing weak dependencies: 2026-02-26T03:26:38.601 INFO:teuthology.orchestra.run.trial132.stdout: ceph-mgr-cephadm noarch 2:20.3.0-5312.gabffb617.el9 ceph-noarch 184 k 2026-02-26T03:26:38.601 INFO:teuthology.orchestra.run.trial132.stdout: ceph-mgr-dashboard noarch 2:20.3.0-5312.gabffb617.el9 ceph-noarch 51 M 2026-02-26T03:26:38.601 INFO:teuthology.orchestra.run.trial132.stdout: ceph-mgr-diskprediction-local noarch 2:20.3.0-5312.gabffb617.el9 ceph-noarch 7.4 M 2026-02-26T03:26:38.601 INFO:teuthology.orchestra.run.trial132.stdout: ceph-mgr-k8sevents noarch 2:20.3.0-5312.gabffb617.el9 ceph-noarch 22 k 2026-02-26T03:26:38.601 INFO:teuthology.orchestra.run.trial132.stdout: ceph-mgr-rook noarch 2:20.3.0-5312.gabffb617.el9 ceph-noarch 50 k 2026-02-26T03:26:38.601 INFO:teuthology.orchestra.run.trial132.stdout: ceph-volume noarch 2:20.3.0-5312.gabffb617.el9 ceph-noarch 301 k 2026-02-26T03:26:38.601 INFO:teuthology.orchestra.run.trial132.stdout: python3-influxdb noarch 5.3.1-1.el9 epel 139 k 2026-02-26T03:26:38.601 INFO:teuthology.orchestra.run.trial132.stdout: python3-saml noarch 1.16.0-1.el9 epel 125 k 2026-02-26T03:26:38.601 INFO:teuthology.orchestra.run.trial132.stdout: 2026-02-26T03:26:38.602 INFO:teuthology.orchestra.run.trial132.stdout:Transaction Summary 2026-02-26T03:26:38.602 INFO:teuthology.orchestra.run.trial132.stdout:======================================================================================= 2026-02-26T03:26:38.602 INFO:teuthology.orchestra.run.trial132.stdout:Install 96 Packages 2026-02-26T03:26:38.602 INFO:teuthology.orchestra.run.trial132.stdout: 2026-02-26T03:26:38.602 INFO:teuthology.orchestra.run.trial132.stdout:Total download size: 225 M 2026-02-26T03:26:38.602 INFO:teuthology.orchestra.run.trial132.stdout:Installed size: 1.5 G 2026-02-26T03:26:38.603 INFO:teuthology.orchestra.run.trial132.stdout:Downloading Packages: 2026-02-26T03:26:39.704 INFO:teuthology.orchestra.run.trial132.stdout:(1/96): ceph-20.3.0-5312.gabffb617.el9.x86_64.r 22 kB/s | 6.5 kB 00:00 2026-02-26T03:26:40.648 INFO:teuthology.orchestra.run.trial132.stdout:(2/96): ceph-mgr-20.3.0-5312.gabffb617.el9.x86_ 759 kB/s | 937 kB 00:01 2026-02-26T03:26:40.785 INFO:teuthology.orchestra.run.trial132.stdout:(3/96): ceph-osd-20.3.0-5312.gabffb617.el9.x86_ 1.2 MB/s | 163 kB 00:00 2026-02-26T03:26:42.470 INFO:teuthology.orchestra.run.trial132.stdout:(4/96): ceph-mds-20.3.0-5312.gabffb617.el9.x86_ 677 kB/s | 2.0 MB 00:03 2026-02-26T03:26:53.227 INFO:teuthology.orchestra.run.trial132.stdout:(5/96): ceph-mon-20.3.0-5312.gabffb617.el9.x86_ 368 kB/s | 4.9 MB 00:13 2026-02-26T03:26:53.507 INFO:teuthology.orchestra.run.trial132.stdout:(6/96): ceph-grafana-dashboards-20.3.0-5312.gab 205 kB/s | 57 kB 00:00 2026-02-26T03:26:54.196 INFO:teuthology.orchestra.run.trial132.stdout:(7/96): ceph-mgr-cephadm-20.3.0-5312.gabffb617. 267 kB/s | 184 kB 00:00 2026-02-26T03:27:01.606 INFO:teuthology.orchestra.run.trial132.stdout:(8/96): ceph-osd-classic-20.3.0-5312.gabffb617. 726 kB/s | 15 MB 00:20 2026-02-26T03:27:13.697 INFO:teuthology.orchestra.run.trial132.stdout:(9/96): ceph-mgr-diskprediction-local-20.3.0-53 625 kB/s | 7.4 MB 00:12 2026-02-26T03:27:13.763 INFO:teuthology.orchestra.run.trial132.stdout:(10/96): ceph-mgr-k8sevents-20.3.0-5312.gabffb6 334 kB/s | 22 kB 00:00 2026-02-26T03:27:14.322 INFO:teuthology.orchestra.run.trial132.stdout:(11/96): ceph-mgr-modules-core-20.3.0-5312.gabf 531 kB/s | 297 kB 00:00 2026-02-26T03:27:14.458 INFO:teuthology.orchestra.run.trial132.stdout:(12/96): ceph-mgr-rook-20.3.0-5312.gabffb617.el 370 kB/s | 50 kB 00:00 2026-02-26T03:27:14.525 INFO:teuthology.orchestra.run.trial132.stdout:(13/96): ceph-prometheus-alerts-20.3.0-5312.gab 258 kB/s | 17 kB 00:00 2026-02-26T03:27:15.149 INFO:teuthology.orchestra.run.trial132.stdout:(14/96): ceph-volume-20.3.0-5312.gabffb617.el9. 483 kB/s | 301 kB 00:00 2026-02-26T03:27:17.219 INFO:teuthology.orchestra.run.trial132.stdout:(15/96): cephadm-20.3.0-5312.gabffb617.el9.noar 502 kB/s | 1.0 MB 00:02 2026-02-26T03:27:17.392 INFO:teuthology.orchestra.run.trial132.stdout:(16/96): cryptsetup-2.8.1-3.el9.x86_64.rpm 2.0 MB/s | 351 kB 00:00 2026-02-26T03:27:17.418 INFO:teuthology.orchestra.run.trial132.stdout:(17/96): ledmon-libs-1.1.0-3.el9.x86_64.rpm 1.5 MB/s | 40 kB 00:00 2026-02-26T03:27:17.445 INFO:teuthology.orchestra.run.trial132.stdout:(18/96): libconfig-1.7.2-9.el9.x86_64.rpm 2.6 MB/s | 72 kB 00:00 2026-02-26T03:27:17.509 INFO:teuthology.orchestra.run.trial132.stdout:(19/96): libgfortran-11.5.0-14.el9.x86_64.rpm 12 MB/s | 794 kB 00:00 2026-02-26T03:27:17.538 INFO:teuthology.orchestra.run.trial132.stdout:(20/96): libquadmath-11.5.0-14.el9.x86_64.rpm 6.4 MB/s | 184 kB 00:00 2026-02-26T03:27:17.566 INFO:teuthology.orchestra.run.trial132.stdout:(21/96): python3-cffi-1.14.5-5.el9.x86_64.rpm 8.6 MB/s | 253 kB 00:00 2026-02-26T03:27:17.617 INFO:teuthology.orchestra.run.trial132.stdout:(22/96): python3-cryptography-36.0.1-5.el9.x86_ 25 MB/s | 1.2 MB 00:00 2026-02-26T03:27:17.644 INFO:teuthology.orchestra.run.trial132.stdout:(23/96): python3-ply-3.11-14.el9.noarch.rpm 3.9 MB/s | 106 kB 00:00 2026-02-26T03:27:17.671 INFO:teuthology.orchestra.run.trial132.stdout:(24/96): python3-pycparser-2.20-6.el9.noarch.rp 4.9 MB/s | 135 kB 00:00 2026-02-26T03:27:17.699 INFO:teuthology.orchestra.run.trial132.stdout:(25/96): python3-pyparsing-2.4.7-9.el9.noarch.r 5.4 MB/s | 150 kB 00:00 2026-02-26T03:27:17.726 INFO:teuthology.orchestra.run.trial132.stdout:(26/96): python3-requests-2.25.1-10.el9.noarch. 4.5 MB/s | 126 kB 00:00 2026-02-26T03:27:17.754 INFO:teuthology.orchestra.run.trial132.stdout:(27/96): python3-urllib3-1.26.5-7.el9.noarch.rp 7.7 MB/s | 218 kB 00:00 2026-02-26T03:27:17.782 INFO:teuthology.orchestra.run.trial132.stdout:(28/96): unzip-6.0-59.el9.x86_64.rpm 6.4 MB/s | 182 kB 00:00 2026-02-26T03:27:17.810 INFO:teuthology.orchestra.run.trial132.stdout:(29/96): zip-3.0-35.el9.x86_64.rpm 9.4 MB/s | 266 kB 00:00 2026-02-26T03:27:17.873 INFO:teuthology.orchestra.run.trial132.stdout:(30/96): flexiblas-3.0.4-9.el9.x86_64.rpm 469 kB/s | 30 kB 00:00 2026-02-26T03:27:17.930 INFO:teuthology.orchestra.run.trial132.stdout:(31/96): flexiblas-netlib-3.0.4-9.el9.x86_64.rp 52 MB/s | 3.0 MB 00:00 2026-02-26T03:27:17.965 INFO:teuthology.orchestra.run.trial132.stdout:(32/96): flexiblas-openblas-openmp-3.0.4-9.el9. 419 kB/s | 15 kB 00:00 2026-02-26T03:27:17.992 INFO:teuthology.orchestra.run.trial132.stdout:(33/96): libstoragemgmt-1.10.1-1.el9.x86_64.rpm 9.2 MB/s | 246 kB 00:00 2026-02-26T03:27:18.030 INFO:teuthology.orchestra.run.trial132.stdout:(34/96): lua-5.4.4-4.el9.x86_64.rpm 4.8 MB/s | 188 kB 00:00 2026-02-26T03:27:18.057 INFO:teuthology.orchestra.run.trial132.stdout:(35/96): openblas-0.3.29-1.el9.x86_64.rpm 1.6 MB/s | 42 kB 00:00 2026-02-26T03:27:18.122 INFO:teuthology.orchestra.run.trial132.stdout:(36/96): openblas-openmp-0.3.29-1.el9.x86_64.rp 81 MB/s | 5.3 MB 00:00 2026-02-26T03:27:18.196 INFO:teuthology.orchestra.run.trial132.stdout:(37/96): python3-babel-2.9.1-2.el9.noarch.rpm 80 MB/s | 6.0 MB 00:00 2026-02-26T03:27:18.224 INFO:teuthology.orchestra.run.trial132.stdout:(38/96): python3-devel-3.9.25-3.el9.x86_64.rpm 8.5 MB/s | 244 kB 00:00 2026-02-26T03:27:18.255 INFO:teuthology.orchestra.run.trial132.stdout:(39/96): python3-jinja2-2.11.3-8.el9.noarch.rpm 7.9 MB/s | 249 kB 00:00 2026-02-26T03:27:18.283 INFO:teuthology.orchestra.run.trial132.stdout:(40/96): python3-jmespath-1.0.1-1.el9.noarch.rp 1.7 MB/s | 48 kB 00:00 2026-02-26T03:27:18.310 INFO:teuthology.orchestra.run.trial132.stdout:(41/96): python3-libstoragemgmt-1.10.1-1.el9.x8 6.3 MB/s | 177 kB 00:00 2026-02-26T03:27:18.344 INFO:teuthology.orchestra.run.trial132.stdout:(42/96): python3-lxml-4.6.5-3.el9.x86_64.rpm 37 MB/s | 1.2 MB 00:00 2026-02-26T03:27:18.371 INFO:teuthology.orchestra.run.trial132.stdout:(43/96): python3-markupsafe-1.1.1-12.el9.x86_64 1.2 MB/s | 35 kB 00:00 2026-02-26T03:27:18.438 INFO:teuthology.orchestra.run.trial132.stdout:(44/96): python3-numpy-1.23.5-2.el9.x86_64.rpm 93 MB/s | 6.1 MB 00:00 2026-02-26T03:27:18.474 INFO:teuthology.orchestra.run.trial132.stdout:(45/96): python3-numpy-f2py-1.23.5-2.el9.x86_64 12 MB/s | 442 kB 00:00 2026-02-26T03:27:18.504 INFO:teuthology.orchestra.run.trial132.stdout:(46/96): python3-packaging-20.9-5.el9.noarch.rp 2.5 MB/s | 77 kB 00:00 2026-02-26T03:27:18.539 INFO:teuthology.orchestra.run.trial132.stdout:(47/96): python3-protobuf-3.14.0-17.el9.noarch. 7.6 MB/s | 267 kB 00:00 2026-02-26T03:27:18.564 INFO:teuthology.orchestra.run.trial132.stdout:(48/96): python3-pyasn1-0.4.8-7.el9.noarch.rpm 6.0 MB/s | 157 kB 00:00 2026-02-26T03:27:18.594 INFO:teuthology.orchestra.run.trial132.stdout:(49/96): python3-pyasn1-modules-0.4.8-7.el9.noa 9.2 MB/s | 277 kB 00:00 2026-02-26T03:27:18.627 INFO:teuthology.orchestra.run.trial132.stdout:(50/96): python3-requests-oauthlib-1.3.0-12.el9 1.6 MB/s | 54 kB 00:00 2026-02-26T03:27:18.793 INFO:teuthology.orchestra.run.trial132.stdout:(51/96): python3-scipy-1.9.3-2.el9.x86_64.rpm 116 MB/s | 19 MB 00:00 2026-02-26T03:27:18.821 INFO:teuthology.orchestra.run.trial132.stdout:(52/96): python3-toml-0.10.2-6.el9.noarch.rpm 1.4 MB/s | 42 kB 00:00 2026-02-26T03:27:18.866 INFO:teuthology.orchestra.run.trial132.stdout:(53/96): xmlsec1-1.2.29-13.el9.x86_64.rpm 4.2 MB/s | 189 kB 00:00 2026-02-26T03:27:18.895 INFO:teuthology.orchestra.run.trial132.stdout:(54/96): xmlsec1-openssl-1.2.29-13.el9.x86_64.r 3.0 MB/s | 90 kB 00:00 2026-02-26T03:27:19.303 INFO:teuthology.orchestra.run.trial132.stdout:(55/96): lua-devel-5.4.4-4.el9.x86_64.rpm 55 kB/s | 22 kB 00:00 2026-02-26T03:27:20.408 INFO:teuthology.orchestra.run.trial132.stdout:(56/96): protobuf-compiler-3.14.0-17.el9.x86_64 780 kB/s | 862 kB 00:01 2026-02-26T03:27:20.682 INFO:teuthology.orchestra.run.trial132.stdout:(57/96): abseil-cpp-20211102.0-4.el9.x86_64.rpm 2.0 MB/s | 551 kB 00:00 2026-02-26T03:27:20.718 INFO:teuthology.orchestra.run.trial132.stdout:(58/96): grpc-data-1.46.7-10.el9.noarch.rpm 536 kB/s | 19 kB 00:00 2026-02-26T03:27:20.760 INFO:teuthology.orchestra.run.trial132.stdout:(59/96): luarocks-3.9.2-5.el9.noarch.rpm 3.5 MB/s | 151 kB 00:00 2026-02-26T03:27:20.820 INFO:teuthology.orchestra.run.trial132.stdout:(60/96): python3-asyncssh-2.13.2-5.el9.noarch.r 8.9 MB/s | 548 kB 00:00 2026-02-26T03:27:20.857 INFO:teuthology.orchestra.run.trial132.stdout:(61/96): python3-autocommand-2.2.2-8.el9.noarch 800 kB/s | 29 kB 00:00 2026-02-26T03:27:20.895 INFO:teuthology.orchestra.run.trial132.stdout:(62/96): python3-backports-tarfile-1.2.0-1.el9. 1.6 MB/s | 60 kB 00:00 2026-02-26T03:27:20.932 INFO:teuthology.orchestra.run.trial132.stdout:(63/96): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 1.1 MB/s | 43 kB 00:00 2026-02-26T03:27:20.968 INFO:teuthology.orchestra.run.trial132.stdout:(64/96): python3-cachetools-4.2.4-1.el9.noarch. 878 kB/s | 32 kB 00:00 2026-02-26T03:27:21.005 INFO:teuthology.orchestra.run.trial132.stdout:(65/96): python3-certifi-2023.05.07-4.el9.noarc 388 kB/s | 14 kB 00:00 2026-02-26T03:27:21.046 INFO:teuthology.orchestra.run.trial132.stdout:(66/96): python3-cheroot-10.0.1-4.el9.noarch.rp 4.1 MB/s | 173 kB 00:00 2026-02-26T03:27:21.095 INFO:teuthology.orchestra.run.trial132.stdout:(67/96): python3-cherrypy-18.6.1-2.el9.noarch.r 7.2 MB/s | 358 kB 00:00 2026-02-26T03:27:21.140 INFO:teuthology.orchestra.run.trial132.stdout:(68/96): python3-google-auth-2.45.0-1.el9.noarc 5.5 MB/s | 254 kB 00:00 2026-02-26T03:27:21.250 INFO:teuthology.orchestra.run.trial132.stdout:(69/96): python3-grpcio-1.46.7-10.el9.x86_64.rp 19 MB/s | 2.0 MB 00:00 2026-02-26T03:27:21.289 INFO:teuthology.orchestra.run.trial132.stdout:(70/96): python3-grpcio-tools-1.46.7-10.el9.x86 3.7 MB/s | 144 kB 00:00 2026-02-26T03:27:21.328 INFO:teuthology.orchestra.run.trial132.stdout:(71/96): python3-influxdb-5.3.1-1.el9.noarch.rp 3.5 MB/s | 139 kB 00:00 2026-02-26T03:27:21.365 INFO:teuthology.orchestra.run.trial132.stdout:(72/96): python3-isodate-0.6.1-3.el9.noarch.rpm 1.5 MB/s | 56 kB 00:00 2026-02-26T03:27:21.402 INFO:teuthology.orchestra.run.trial132.stdout:(73/96): python3-jaraco-8.2.1-3.el9.noarch.rpm 294 kB/s | 11 kB 00:00 2026-02-26T03:27:21.438 INFO:teuthology.orchestra.run.trial132.stdout:(74/96): python3-jaraco-classes-3.2.1-5.el9.noa 485 kB/s | 18 kB 00:00 2026-02-26T03:27:21.475 INFO:teuthology.orchestra.run.trial132.stdout:(75/96): python3-jaraco-collections-3.0.0-8.el9 635 kB/s | 23 kB 00:00 2026-02-26T03:27:21.511 INFO:teuthology.orchestra.run.trial132.stdout:(76/96): python3-jaraco-context-6.0.1-3.el9.noa 538 kB/s | 20 kB 00:00 2026-02-26T03:27:21.548 INFO:teuthology.orchestra.run.trial132.stdout:(77/96): python3-jaraco-functools-3.5.0-2.el9.n 532 kB/s | 19 kB 00:00 2026-02-26T03:27:21.584 INFO:teuthology.orchestra.run.trial132.stdout:(78/96): python3-jaraco-text-4.0.0-2.el9.noarch 722 kB/s | 26 kB 00:00 2026-02-26T03:27:21.660 INFO:teuthology.orchestra.run.trial132.stdout:(79/96): python3-kubernetes-26.1.0-3.el9.noarch 13 MB/s | 1.0 MB 00:00 2026-02-26T03:27:21.696 INFO:teuthology.orchestra.run.trial132.stdout:(80/96): python3-more-itertools-8.12.0-2.el9.no 2.2 MB/s | 79 kB 00:00 2026-02-26T03:27:21.732 INFO:teuthology.orchestra.run.trial132.stdout:(81/96): python3-msgpack-1.0.3-2.el9.x86_64.rpm 2.4 MB/s | 86 kB 00:00 2026-02-26T03:27:21.803 INFO:teuthology.orchestra.run.trial132.stdout:(82/96): python3-natsort-7.1.1-5.el9.noarch.rpm 819 kB/s | 58 kB 00:00 2026-02-26T03:27:21.838 INFO:teuthology.orchestra.run.trial132.stdout:(83/96): python3-portend-3.1.0-2.el9.noarch.rpm 460 kB/s | 16 kB 00:00 2026-02-26T03:27:21.877 INFO:teuthology.orchestra.run.trial132.stdout:(84/96): python3-pyOpenSSL-21.0.0-1.el9.noarch. 2.3 MB/s | 90 kB 00:00 2026-02-26T03:27:21.913 INFO:teuthology.orchestra.run.trial132.stdout:(85/96): python3-repoze-lru-0.7-16.el9.noarch.r 858 kB/s | 31 kB 00:00 2026-02-26T03:27:21.956 INFO:teuthology.orchestra.run.trial132.stdout:(86/96): python3-routes-2.5.1-5.el9.noarch.rpm 4.3 MB/s | 188 kB 00:00 2026-02-26T03:27:22.043 INFO:teuthology.orchestra.run.trial132.stdout:(87/96): python3-rsa-4.9-2.el9.noarch.rpm 679 kB/s | 59 kB 00:00 2026-02-26T03:27:22.084 INFO:teuthology.orchestra.run.trial132.stdout:(88/96): python3-saml-1.16.0-1.el9.noarch.rpm 3.0 MB/s | 125 kB 00:00 2026-02-26T03:27:22.120 INFO:teuthology.orchestra.run.trial132.stdout:(89/96): python3-tempora-5.0.0-2.el9.noarch.rpm 996 kB/s | 36 kB 00:00 2026-02-26T03:27:22.168 INFO:teuthology.orchestra.run.trial132.stdout:(90/96): python3-typing-extensions-4.15.0-1.el9 1.8 MB/s | 86 kB 00:00 2026-02-26T03:27:22.254 INFO:teuthology.orchestra.run.trial132.stdout:(91/96): python3-websocket-client-1.2.3-2.el9.n 1.0 MB/s | 90 kB 00:00 2026-02-26T03:27:22.318 INFO:teuthology.orchestra.run.trial132.stdout:(92/96): python3-xmlsec-1.3.13-1.el9.x86_64.rpm 755 kB/s | 48 kB 00:00 2026-02-26T03:27:22.401 INFO:teuthology.orchestra.run.trial132.stdout:(93/96): python3-xmltodict-0.12.0-15.el9.noarch 267 kB/s | 22 kB 00:00 2026-02-26T03:27:22.515 INFO:teuthology.orchestra.run.trial132.stdout:(94/96): python3-zc-lockfile-2.0-10.el9.noarch. 176 kB/s | 20 kB 00:00 2026-02-26T03:29:11.678 INFO:teuthology.orchestra.run.trial132.stdout:(95/96): ceph-mgr-dashboard-20.3.0-5312.gabffb6 377 kB/s | 51 MB 02:17 2026-02-26T03:30:18.284 INFO:teuthology.orchestra.run.trial132.stdout:(96/96): ceph-osd-crimson-20.3.0-5312.gabffb617 411 kB/s | 87 MB 03:35 2026-02-26T03:30:18.287 INFO:teuthology.orchestra.run.trial132.stdout:-------------------------------------------------------------------------------- 2026-02-26T03:30:18.287 INFO:teuthology.orchestra.run.trial132.stdout:Total 1.0 MB/s | 225 MB 03:39 2026-02-26T03:30:18.667 INFO:teuthology.orchestra.run.trial132.stdout:Running transaction check 2026-02-26T03:30:18.705 INFO:teuthology.orchestra.run.trial132.stdout:Transaction check succeeded. 2026-02-26T03:30:18.706 INFO:teuthology.orchestra.run.trial132.stdout:Running transaction test 2026-02-26T03:30:19.503 INFO:teuthology.orchestra.run.trial132.stdout:Transaction test succeeded. 2026-02-26T03:30:19.503 INFO:teuthology.orchestra.run.trial132.stdout:Running transaction 2026-02-26T03:30:20.328 INFO:teuthology.orchestra.run.trial132.stdout: Preparing : 1/1 2026-02-26T03:30:20.330 INFO:teuthology.orchestra.run.trial132.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/96 2026-02-26T03:30:20.333 INFO:teuthology.orchestra.run.trial132.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/96 2026-02-26T03:30:20.355 INFO:teuthology.orchestra.run.trial132.stdout: Installing : xmlsec1-1.2.29-13.el9.x86_64 3/96 2026-02-26T03:30:20.388 INFO:teuthology.orchestra.run.trial132.stdout: Installing : python3-pyasn1-0.4.8-7.el9.noarch 4/96 2026-02-26T03:30:20.394 INFO:teuthology.orchestra.run.trial132.stdout: Installing : python3-lxml-4.6.5-3.el9.x86_64 5/96 2026-02-26T03:30:20.397 INFO:teuthology.orchestra.run.trial132.stdout: Installing : lua-5.4.4-4.el9.x86_64 6/96 2026-02-26T03:30:20.401 INFO:teuthology.orchestra.run.trial132.stdout: Installing : flexiblas-3.0.4-9.el9.x86_64 7/96 2026-02-26T03:30:20.424 INFO:teuthology.orchestra.run.trial132.stdout: Installing : unzip-6.0-59.el9.x86_64 8/96 2026-02-26T03:30:20.437 INFO:teuthology.orchestra.run.trial132.stdout: Installing : python3-urllib3-1.26.5-7.el9.noarch 9/96 2026-02-26T03:30:20.440 INFO:teuthology.orchestra.run.trial132.stdout: Installing : python3-requests-2.25.1-10.el9.noarch 10/96 2026-02-26T03:30:20.446 INFO:teuthology.orchestra.run.trial132.stdout: Installing : libquadmath-11.5.0-14.el9.x86_64 11/96 2026-02-26T03:30:20.447 INFO:teuthology.orchestra.run.trial132.stdout: Installing : libgfortran-11.5.0-14.el9.x86_64 12/96 2026-02-26T03:30:20.458 INFO:teuthology.orchestra.run.trial132.stdout: Installing : ledmon-libs-1.1.0-3.el9.x86_64 13/96 2026-02-26T03:30:20.464 INFO:teuthology.orchestra.run.trial132.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 14/96 2026-02-26T03:30:20.488 INFO:teuthology.orchestra.run.trial132.stdout: Installing : zip-3.0-35.el9.x86_64 15/96 2026-02-26T03:30:20.491 INFO:teuthology.orchestra.run.trial132.stdout: Installing : luarocks-3.9.2-5.el9.noarch 16/96 2026-02-26T03:30:20.541 INFO:teuthology.orchestra.run.trial132.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 17/96 2026-02-26T03:30:20.554 INFO:teuthology.orchestra.run.trial132.stdout: Installing : python3-pyasn1-modules-0.4.8-7.el9.noarch 18/96 2026-02-26T03:30:20.556 INFO:teuthology.orchestra.run.trial132.stdout: Installing : python3-rsa-4.9-2.el9.noarch 19/96 2026-02-26T03:30:20.560 INFO:teuthology.orchestra.run.trial132.stdout: Installing : xmlsec1-openssl-1.2.29-13.el9.x86_64 20/96 2026-02-26T03:30:20.565 INFO:teuthology.orchestra.run.trial132.stdout: Installing : python3-xmlsec-1.3.13-1.el9.x86_64 21/96 2026-02-26T03:30:20.570 INFO:teuthology.orchestra.run.trial132.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 22/96 2026-02-26T03:30:20.573 INFO:teuthology.orchestra.run.trial132.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 23/96 2026-02-26T03:30:20.588 INFO:teuthology.orchestra.run.trial132.stdout: Installing : python3-xmltodict-0.12.0-15.el9.noarch 24/96 2026-02-26T03:30:20.592 INFO:teuthology.orchestra.run.trial132.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 25/96 2026-02-26T03:30:20.597 INFO:teuthology.orchestra.run.trial132.stdout: Installing : python3-typing-extensions-4.15.0-1.el9.noarch 26/96 2026-02-26T03:30:20.608 INFO:teuthology.orchestra.run.trial132.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 27/96 2026-02-26T03:30:20.617 INFO:teuthology.orchestra.run.trial132.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 28/96 2026-02-26T03:30:20.623 INFO:teuthology.orchestra.run.trial132.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 29/96 2026-02-26T03:30:20.642 INFO:teuthology.orchestra.run.trial132.stdout: Installing : python3-msgpack-1.0.3-2.el9.x86_64 30/96 2026-02-26T03:30:20.653 INFO:teuthology.orchestra.run.trial132.stdout: Installing : python3-influxdb-5.3.1-1.el9.noarch 31/96 2026-02-26T03:30:20.669 INFO:teuthology.orchestra.run.trial132.stdout: Installing : python3-isodate-0.6.1-3.el9.noarch 32/96 2026-02-26T03:30:20.673 INFO:teuthology.orchestra.run.trial132.stdout: Installing : python3-saml-1.16.0-1.el9.noarch 33/96 2026-02-26T03:30:20.681 INFO:teuthology.orchestra.run.trial132.stdout: Installing : python3-certifi-2023.05.07-4.el9.noarch 34/96 2026-02-26T03:30:20.720 INFO:teuthology.orchestra.run.trial132.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 35/96 2026-02-26T03:30:21.040 INFO:teuthology.orchestra.run.trial132.stdout: Installing : python3-google-auth-1:2.45.0-1.el9.noarch 36/96 2026-02-26T03:30:21.050 INFO:teuthology.orchestra.run.trial132.stdout: Installing : python3-kubernetes-1:26.1.0-3.el9.noarch 37/96 2026-02-26T03:30:21.054 INFO:teuthology.orchestra.run.trial132.stdout: Installing : python3-backports-tarfile-1.2.0-1.el9.noarch 38/96 2026-02-26T03:30:21.060 INFO:teuthology.orchestra.run.trial132.stdout: Installing : python3-jaraco-context-6.0.1-3.el9.noarch 39/96 2026-02-26T03:30:21.062 INFO:teuthology.orchestra.run.trial132.stdout: Installing : python3-autocommand-2.2.2-8.el9.noarch 40/96 2026-02-26T03:30:21.088 INFO:teuthology.orchestra.run.trial132.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 41/96 2026-02-26T03:30:21.131 INFO:teuthology.orchestra.run.trial132.stdout: Installing : abseil-cpp-20211102.0-4.el9.x86_64 42/96 2026-02-26T03:30:21.140 INFO:teuthology.orchestra.run.trial132.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 43/96 2026-02-26T03:30:21.145 INFO:teuthology.orchestra.run.trial132.stdout: Installing : protobuf-compiler-3.14.0-17.el9.x86_64 44/96 2026-02-26T03:30:21.149 INFO:teuthology.orchestra.run.trial132.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 45/96 2026-02-26T03:30:21.154 INFO:teuthology.orchestra.run.trial132.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 46/96 2026-02-26T03:30:21.158 INFO:teuthology.orchestra.run.trial132.stdout: Installing : python3-jaraco-text-4.0.0-2.el9.noarch 47/96 2026-02-26T03:30:21.165 INFO:teuthology.orchestra.run.trial132.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 48/96 2026-02-26T03:30:21.168 INFO:teuthology.orchestra.run.trial132.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 49/96 2026-02-26T03:30:21.196 INFO:teuthology.orchestra.run.trial132.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 50/96 2026-02-26T03:30:21.205 INFO:teuthology.orchestra.run.trial132.stdout: Installing : python3-protobuf-3.14.0-17.el9.noarch 51/96 2026-02-26T03:30:21.211 INFO:teuthology.orchestra.run.trial132.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 52/96 2026-02-26T03:30:21.218 INFO:teuthology.orchestra.run.trial132.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 53/96 2026-02-26T03:30:21.252 INFO:teuthology.orchestra.run.trial132.stdout: Installing : python3-jmespath-1.0.1-1.el9.noarch 54/96 2026-02-26T03:30:21.473 INFO:teuthology.orchestra.run.trial132.stdout: Installing : python3-devel-3.9.25-3.el9.x86_64 55/96 2026-02-26T03:30:21.496 INFO:teuthology.orchestra.run.trial132.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 56/96 2026-02-26T03:30:21.500 INFO:teuthology.orchestra.run.trial132.stdout: Installing : python3-jinja2-2.11.3-8.el9.noarch 57/96 2026-02-26T03:30:21.558 INFO:teuthology.orchestra.run.trial132.stdout: Installing : openblas-0.3.29-1.el9.x86_64 58/96 2026-02-26T03:30:21.559 INFO:teuthology.orchestra.run.trial132.stdout: Installing : openblas-openmp-0.3.29-1.el9.x86_64 59/96 2026-02-26T03:30:21.580 INFO:teuthology.orchestra.run.trial132.stdout: Installing : flexiblas-openblas-openmp-3.0.4-9.el9.x86_64 60/96 2026-02-26T03:30:21.901 INFO:teuthology.orchestra.run.trial132.stdout: Installing : flexiblas-netlib-3.0.4-9.el9.x86_64 61/96 2026-02-26T03:30:21.973 INFO:teuthology.orchestra.run.trial132.stdout: Installing : python3-numpy-1:1.23.5-2.el9.x86_64 62/96 2026-02-26T03:30:22.637 INFO:teuthology.orchestra.run.trial132.stdout: Installing : python3-numpy-f2py-1:1.23.5-2.el9.x86_64 63/96 2026-02-26T03:30:22.654 INFO:teuthology.orchestra.run.trial132.stdout: Installing : python3-scipy-1.9.3-2.el9.x86_64 64/96 2026-02-26T03:30:22.663 INFO:teuthology.orchestra.run.trial132.stdout: Installing : python3-pyparsing-2.4.7-9.el9.noarch 65/96 2026-02-26T03:30:22.670 INFO:teuthology.orchestra.run.trial132.stdout: Installing : python3-packaging-20.9-5.el9.noarch 66/96 2026-02-26T03:30:22.683 INFO:teuthology.orchestra.run.trial132.stdout: Installing : python3-ply-3.11-14.el9.noarch 67/96 2026-02-26T03:30:22.699 INFO:teuthology.orchestra.run.trial132.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 68/96 2026-02-26T03:30:22.773 INFO:teuthology.orchestra.run.trial132.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 69/96 2026-02-26T03:30:22.782 INFO:teuthology.orchestra.run.trial132.stdout: Installing : python3-cryptography-36.0.1-5.el9.x86_64 70/96 2026-02-26T03:30:22.807 INFO:teuthology.orchestra.run.trial132.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 71/96 2026-02-26T03:30:22.836 INFO:teuthology.orchestra.run.trial132.stdout: Installing : python3-cheroot-10.0.1-4.el9.noarch 72/96 2026-02-26T03:30:22.886 INFO:teuthology.orchestra.run.trial132.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 73/96 2026-02-26T03:30:22.892 INFO:teuthology.orchestra.run.trial132.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 74/96 2026-02-26T03:30:22.895 INFO:teuthology.orchestra.run.trial132.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 75/96 2026-02-26T03:30:22.896 INFO:teuthology.orchestra.run.trial132.stdout: Installing : libconfig-1.7.2-9.el9.x86_64 76/96 2026-02-26T03:30:22.914 INFO:teuthology.orchestra.run.trial132.stdout: Running scriptlet: libstoragemgmt-1.10.1-1.el9.x86_64 77/96 2026-02-26T03:30:22.915 INFO:teuthology.orchestra.run.trial132.stdout:Creating group 'libstoragemgmt' with GID 991. 2026-02-26T03:30:22.915 INFO:teuthology.orchestra.run.trial132.stdout:Creating user 'libstoragemgmt' (daemon account for libstoragemgmt) with UID 991 and GID 991. 2026-02-26T03:30:22.915 INFO:teuthology.orchestra.run.trial132.stdout: 2026-02-26T03:30:22.923 INFO:teuthology.orchestra.run.trial132.stdout: Installing : libstoragemgmt-1.10.1-1.el9.x86_64 77/96 2026-02-26T03:30:22.949 INFO:teuthology.orchestra.run.trial132.stdout: Running scriptlet: libstoragemgmt-1.10.1-1.el9.x86_64 77/96 2026-02-26T03:30:22.949 INFO:teuthology.orchestra.run.trial132.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/libstoragemgmt.service → /usr/lib/systemd/system/libstoragemgmt.service. 2026-02-26T03:30:22.950 INFO:teuthology.orchestra.run.trial132.stdout: 2026-02-26T03:30:22.984 INFO:teuthology.orchestra.run.trial132.stdout: Installing : python3-libstoragemgmt-1.10.1-1.el9.x86_64 78/96 2026-02-26T03:30:23.573 INFO:teuthology.orchestra.run.trial132.stdout: Installing : cryptsetup-2.8.1-3.el9.x86_64 79/96 2026-02-26T03:30:28.891 INFO:teuthology.orchestra.run.trial132.stdout: Installing : ceph-osd-classic-2:20.3.0-5312.gabffb617.el9.x86_6 80/96 2026-02-26T03:30:28.954 INFO:teuthology.orchestra.run.trial132.stdout: Installing : ceph-osd-crimson-2:20.3.0-5312.gabffb617.el9.x86_6 81/96 2026-02-26T03:30:28.956 INFO:teuthology.orchestra.run.trial132.stdout: Installing : ceph-volume-2:20.3.0-5312.gabffb617.el9.noarch 82/96 2026-02-26T03:30:28.971 INFO:teuthology.orchestra.run.trial132.stdout: Running scriptlet: ceph-volume-2:20.3.0-5312.gabffb617.el9.noarch 82/96 2026-02-26T03:30:28.971 INFO:teuthology.orchestra.run.trial132.stdout:Glob pattern passed to enable, but globs are not supported for this. 2026-02-26T03:30:28.971 INFO:teuthology.orchestra.run.trial132.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2026-02-26T03:30:28.971 INFO:teuthology.orchestra.run.trial132.stdout: 2026-02-26T03:30:28.980 INFO:teuthology.orchestra.run.trial132.stdout: Installing : ceph-osd-2:20.3.0-5312.gabffb617.el9.x86_64 83/96 2026-02-26T03:30:29.003 INFO:teuthology.orchestra.run.trial132.stdout: Running scriptlet: ceph-osd-2:20.3.0-5312.gabffb617.el9.x86_64 83/96 2026-02-26T03:30:29.003 INFO:teuthology.orchestra.run.trial132.stdout:Glob pattern passed to enable, but globs are not supported for this. 2026-02-26T03:30:29.003 INFO:teuthology.orchestra.run.trial132.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2026-02-26T03:30:29.004 INFO:teuthology.orchestra.run.trial132.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2026-02-26T03:30:29.004 INFO:teuthology.orchestra.run.trial132.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2026-02-26T03:30:29.004 INFO:teuthology.orchestra.run.trial132.stdout: 2026-02-26T03:30:29.064 INFO:teuthology.orchestra.run.trial132.stdout: Running scriptlet: cephadm-2:20.3.0-5312.gabffb617.el9.noarch 84/96 2026-02-26T03:30:29.066 INFO:teuthology.orchestra.run.trial132.stdout: Installing : cephadm-2:20.3.0-5312.gabffb617.el9.noarch 84/96 2026-02-26T03:30:29.075 INFO:teuthology.orchestra.run.trial132.stdout: Installing : ceph-prometheus-alerts-2:20.3.0-5312.gabffb617.el9 85/96 2026-02-26T03:30:29.099 INFO:teuthology.orchestra.run.trial132.stdout: Installing : ceph-grafana-dashboards-2:20.3.0-5312.gabffb617.el 86/96 2026-02-26T03:30:29.100 INFO:teuthology.orchestra.run.trial132.stdout: Installing : ceph-mgr-cephadm-2:20.3.0-5312.gabffb617.el9.noarc 87/96 2026-02-26T03:30:31.974 INFO:teuthology.orchestra.run.trial132.stdout: Running scriptlet: ceph-mgr-cephadm-2:20.3.0-5312.gabffb617.el9.noarc 87/96 2026-02-26T03:30:31.981 INFO:teuthology.orchestra.run.trial132.stdout: Installing : ceph-mgr-dashboard-2:20.3.0-5312.gabffb617.el9.noa 88/96 2026-02-26T03:30:32.432 INFO:teuthology.orchestra.run.trial132.stdout: Running scriptlet: ceph-mgr-dashboard-2:20.3.0-5312.gabffb617.el9.noa 88/96 2026-02-26T03:30:32.433 INFO:teuthology.orchestra.run.trial132.stdout: Installing : ceph-mgr-diskprediction-local-2:20.3.0-5312.gabffb 89/96 2026-02-26T03:30:32.445 INFO:teuthology.orchestra.run.trial132.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:20.3.0-5312.gabffb 89/96 2026-02-26T03:30:32.445 INFO:teuthology.orchestra.run.trial132.stdout: Installing : ceph-mgr-k8sevents-2:20.3.0-5312.gabffb617.el9.noa 90/96 2026-02-26T03:30:32.504 INFO:teuthology.orchestra.run.trial132.stdout: Running scriptlet: ceph-mgr-k8sevents-2:20.3.0-5312.gabffb617.el9.noa 90/96 2026-02-26T03:30:32.544 INFO:teuthology.orchestra.run.trial132.stdout: Installing : ceph-mgr-modules-core-2:20.3.0-5312.gabffb617.el9. 91/96 2026-02-26T03:30:32.545 INFO:teuthology.orchestra.run.trial132.stdout: Installing : ceph-mgr-2:20.3.0-5312.gabffb617.el9.x86_64 92/96 2026-02-26T03:30:32.565 INFO:teuthology.orchestra.run.trial132.stdout: Running scriptlet: ceph-mgr-2:20.3.0-5312.gabffb617.el9.x86_64 92/96 2026-02-26T03:30:32.565 INFO:teuthology.orchestra.run.trial132.stdout:Glob pattern passed to enable, but globs are not supported for this. 2026-02-26T03:30:32.565 INFO:teuthology.orchestra.run.trial132.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2026-02-26T03:30:32.565 INFO:teuthology.orchestra.run.trial132.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2026-02-26T03:30:32.566 INFO:teuthology.orchestra.run.trial132.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2026-02-26T03:30:32.566 INFO:teuthology.orchestra.run.trial132.stdout: 2026-02-26T03:30:32.577 INFO:teuthology.orchestra.run.trial132.stdout: Installing : ceph-mgr-rook-2:20.3.0-5312.gabffb617.el9.noarch 93/96 2026-02-26T03:30:32.788 INFO:teuthology.orchestra.run.trial132.stdout: Running scriptlet: ceph-mgr-rook-2:20.3.0-5312.gabffb617.el9.noarch 93/96 2026-02-26T03:30:32.789 INFO:teuthology.orchestra.run.trial132.stdout: Installing : ceph-mon-2:20.3.0-5312.gabffb617.el9.x86_64 94/96 2026-02-26T03:30:32.810 INFO:teuthology.orchestra.run.trial132.stdout: Running scriptlet: ceph-mon-2:20.3.0-5312.gabffb617.el9.x86_64 94/96 2026-02-26T03:30:32.810 INFO:teuthology.orchestra.run.trial132.stdout:Glob pattern passed to enable, but globs are not supported for this. 2026-02-26T03:30:32.810 INFO:teuthology.orchestra.run.trial132.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2026-02-26T03:30:32.810 INFO:teuthology.orchestra.run.trial132.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2026-02-26T03:30:32.811 INFO:teuthology.orchestra.run.trial132.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2026-02-26T03:30:32.811 INFO:teuthology.orchestra.run.trial132.stdout: 2026-02-26T03:30:32.893 INFO:teuthology.orchestra.run.trial132.stdout: Installing : ceph-mds-2:20.3.0-5312.gabffb617.el9.x86_64 95/96 2026-02-26T03:30:32.913 INFO:teuthology.orchestra.run.trial132.stdout: Running scriptlet: ceph-mds-2:20.3.0-5312.gabffb617.el9.x86_64 95/96 2026-02-26T03:30:32.913 INFO:teuthology.orchestra.run.trial132.stdout:Glob pattern passed to enable, but globs are not supported for this. 2026-02-26T03:30:32.913 INFO:teuthology.orchestra.run.trial132.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2026-02-26T03:30:32.913 INFO:teuthology.orchestra.run.trial132.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2026-02-26T03:30:32.913 INFO:teuthology.orchestra.run.trial132.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2026-02-26T03:30:32.913 INFO:teuthology.orchestra.run.trial132.stdout: 2026-02-26T03:30:32.914 INFO:teuthology.orchestra.run.trial132.stdout: Installing : ceph-2:20.3.0-5312.gabffb617.el9.x86_64 96/96 2026-02-26T03:30:32.920 INFO:teuthology.orchestra.run.trial132.stdout: Running scriptlet: ceph-osd-classic-2:20.3.0-5312.gabffb617.el9.x86_6 96/96 2026-02-26T03:30:32.944 INFO:teuthology.orchestra.run.trial132.stdout: Running scriptlet: ceph-osd-crimson-2:20.3.0-5312.gabffb617.el9.x86_6 96/96 2026-02-26T03:30:34.281 INFO:teuthology.orchestra.run.trial132.stdout: Running scriptlet: ceph-2:20.3.0-5312.gabffb617.el9.x86_64 96/96 2026-02-26T03:30:34.281 INFO:teuthology.orchestra.run.trial132.stdout: Verifying : ceph-2:20.3.0-5312.gabffb617.el9.x86_64 1/96 2026-02-26T03:30:34.281 INFO:teuthology.orchestra.run.trial132.stdout: Verifying : ceph-mds-2:20.3.0-5312.gabffb617.el9.x86_64 2/96 2026-02-26T03:30:34.281 INFO:teuthology.orchestra.run.trial132.stdout: Verifying : ceph-mgr-2:20.3.0-5312.gabffb617.el9.x86_64 3/96 2026-02-26T03:30:34.282 INFO:teuthology.orchestra.run.trial132.stdout: Verifying : ceph-mon-2:20.3.0-5312.gabffb617.el9.x86_64 4/96 2026-02-26T03:30:34.282 INFO:teuthology.orchestra.run.trial132.stdout: Verifying : ceph-osd-2:20.3.0-5312.gabffb617.el9.x86_64 5/96 2026-02-26T03:30:34.282 INFO:teuthology.orchestra.run.trial132.stdout: Verifying : ceph-osd-classic-2:20.3.0-5312.gabffb617.el9.x86_6 6/96 2026-02-26T03:30:34.282 INFO:teuthology.orchestra.run.trial132.stdout: Verifying : ceph-osd-crimson-2:20.3.0-5312.gabffb617.el9.x86_6 7/96 2026-02-26T03:30:34.282 INFO:teuthology.orchestra.run.trial132.stdout: Verifying : ceph-grafana-dashboards-2:20.3.0-5312.gabffb617.el 8/96 2026-02-26T03:30:34.282 INFO:teuthology.orchestra.run.trial132.stdout: Verifying : ceph-mgr-cephadm-2:20.3.0-5312.gabffb617.el9.noarc 9/96 2026-02-26T03:30:34.282 INFO:teuthology.orchestra.run.trial132.stdout: Verifying : ceph-mgr-dashboard-2:20.3.0-5312.gabffb617.el9.noa 10/96 2026-02-26T03:30:34.282 INFO:teuthology.orchestra.run.trial132.stdout: Verifying : ceph-mgr-diskprediction-local-2:20.3.0-5312.gabffb 11/96 2026-02-26T03:30:34.282 INFO:teuthology.orchestra.run.trial132.stdout: Verifying : ceph-mgr-k8sevents-2:20.3.0-5312.gabffb617.el9.noa 12/96 2026-02-26T03:30:34.282 INFO:teuthology.orchestra.run.trial132.stdout: Verifying : ceph-mgr-modules-core-2:20.3.0-5312.gabffb617.el9. 13/96 2026-02-26T03:30:34.282 INFO:teuthology.orchestra.run.trial132.stdout: Verifying : ceph-mgr-rook-2:20.3.0-5312.gabffb617.el9.noarch 14/96 2026-02-26T03:30:34.282 INFO:teuthology.orchestra.run.trial132.stdout: Verifying : ceph-prometheus-alerts-2:20.3.0-5312.gabffb617.el9 15/96 2026-02-26T03:30:34.282 INFO:teuthology.orchestra.run.trial132.stdout: Verifying : ceph-volume-2:20.3.0-5312.gabffb617.el9.noarch 16/96 2026-02-26T03:30:34.283 INFO:teuthology.orchestra.run.trial132.stdout: Verifying : cephadm-2:20.3.0-5312.gabffb617.el9.noarch 17/96 2026-02-26T03:30:34.283 INFO:teuthology.orchestra.run.trial132.stdout: Verifying : cryptsetup-2.8.1-3.el9.x86_64 18/96 2026-02-26T03:30:34.283 INFO:teuthology.orchestra.run.trial132.stdout: Verifying : ledmon-libs-1.1.0-3.el9.x86_64 19/96 2026-02-26T03:30:34.283 INFO:teuthology.orchestra.run.trial132.stdout: Verifying : libconfig-1.7.2-9.el9.x86_64 20/96 2026-02-26T03:30:34.283 INFO:teuthology.orchestra.run.trial132.stdout: Verifying : libgfortran-11.5.0-14.el9.x86_64 21/96 2026-02-26T03:30:34.283 INFO:teuthology.orchestra.run.trial132.stdout: Verifying : libquadmath-11.5.0-14.el9.x86_64 22/96 2026-02-26T03:30:34.283 INFO:teuthology.orchestra.run.trial132.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 23/96 2026-02-26T03:30:34.283 INFO:teuthology.orchestra.run.trial132.stdout: Verifying : python3-cryptography-36.0.1-5.el9.x86_64 24/96 2026-02-26T03:30:34.283 INFO:teuthology.orchestra.run.trial132.stdout: Verifying : python3-ply-3.11-14.el9.noarch 25/96 2026-02-26T03:30:34.283 INFO:teuthology.orchestra.run.trial132.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 26/96 2026-02-26T03:30:34.283 INFO:teuthology.orchestra.run.trial132.stdout: Verifying : python3-pyparsing-2.4.7-9.el9.noarch 27/96 2026-02-26T03:30:34.284 INFO:teuthology.orchestra.run.trial132.stdout: Verifying : python3-requests-2.25.1-10.el9.noarch 28/96 2026-02-26T03:30:34.284 INFO:teuthology.orchestra.run.trial132.stdout: Verifying : python3-urllib3-1.26.5-7.el9.noarch 29/96 2026-02-26T03:30:34.284 INFO:teuthology.orchestra.run.trial132.stdout: Verifying : unzip-6.0-59.el9.x86_64 30/96 2026-02-26T03:30:34.284 INFO:teuthology.orchestra.run.trial132.stdout: Verifying : zip-3.0-35.el9.x86_64 31/96 2026-02-26T03:30:34.284 INFO:teuthology.orchestra.run.trial132.stdout: Verifying : flexiblas-3.0.4-9.el9.x86_64 32/96 2026-02-26T03:30:34.284 INFO:teuthology.orchestra.run.trial132.stdout: Verifying : flexiblas-netlib-3.0.4-9.el9.x86_64 33/96 2026-02-26T03:30:34.284 INFO:teuthology.orchestra.run.trial132.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-9.el9.x86_64 34/96 2026-02-26T03:30:34.284 INFO:teuthology.orchestra.run.trial132.stdout: Verifying : libstoragemgmt-1.10.1-1.el9.x86_64 35/96 2026-02-26T03:30:34.284 INFO:teuthology.orchestra.run.trial132.stdout: Verifying : lua-5.4.4-4.el9.x86_64 36/96 2026-02-26T03:30:34.284 INFO:teuthology.orchestra.run.trial132.stdout: Verifying : openblas-0.3.29-1.el9.x86_64 37/96 2026-02-26T03:30:34.284 INFO:teuthology.orchestra.run.trial132.stdout: Verifying : openblas-openmp-0.3.29-1.el9.x86_64 38/96 2026-02-26T03:30:34.284 INFO:teuthology.orchestra.run.trial132.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 39/96 2026-02-26T03:30:34.284 INFO:teuthology.orchestra.run.trial132.stdout: Verifying : python3-devel-3.9.25-3.el9.x86_64 40/96 2026-02-26T03:30:34.285 INFO:teuthology.orchestra.run.trial132.stdout: Verifying : python3-jinja2-2.11.3-8.el9.noarch 41/96 2026-02-26T03:30:34.285 INFO:teuthology.orchestra.run.trial132.stdout: Verifying : python3-jmespath-1.0.1-1.el9.noarch 42/96 2026-02-26T03:30:34.285 INFO:teuthology.orchestra.run.trial132.stdout: Verifying : python3-libstoragemgmt-1.10.1-1.el9.x86_64 43/96 2026-02-26T03:30:34.285 INFO:teuthology.orchestra.run.trial132.stdout: Verifying : python3-lxml-4.6.5-3.el9.x86_64 44/96 2026-02-26T03:30:34.285 INFO:teuthology.orchestra.run.trial132.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 45/96 2026-02-26T03:30:34.285 INFO:teuthology.orchestra.run.trial132.stdout: Verifying : python3-numpy-1:1.23.5-2.el9.x86_64 46/96 2026-02-26T03:30:34.285 INFO:teuthology.orchestra.run.trial132.stdout: Verifying : python3-numpy-f2py-1:1.23.5-2.el9.x86_64 47/96 2026-02-26T03:30:34.285 INFO:teuthology.orchestra.run.trial132.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 48/96 2026-02-26T03:30:34.285 INFO:teuthology.orchestra.run.trial132.stdout: Verifying : python3-protobuf-3.14.0-17.el9.noarch 49/96 2026-02-26T03:30:34.285 INFO:teuthology.orchestra.run.trial132.stdout: Verifying : python3-pyasn1-0.4.8-7.el9.noarch 50/96 2026-02-26T03:30:34.285 INFO:teuthology.orchestra.run.trial132.stdout: Verifying : python3-pyasn1-modules-0.4.8-7.el9.noarch 51/96 2026-02-26T03:30:34.285 INFO:teuthology.orchestra.run.trial132.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 52/96 2026-02-26T03:30:34.286 INFO:teuthology.orchestra.run.trial132.stdout: Verifying : python3-scipy-1.9.3-2.el9.x86_64 53/96 2026-02-26T03:30:34.286 INFO:teuthology.orchestra.run.trial132.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 54/96 2026-02-26T03:30:34.286 INFO:teuthology.orchestra.run.trial132.stdout: Verifying : xmlsec1-1.2.29-13.el9.x86_64 55/96 2026-02-26T03:30:34.286 INFO:teuthology.orchestra.run.trial132.stdout: Verifying : xmlsec1-openssl-1.2.29-13.el9.x86_64 56/96 2026-02-26T03:30:34.286 INFO:teuthology.orchestra.run.trial132.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 57/96 2026-02-26T03:30:34.286 INFO:teuthology.orchestra.run.trial132.stdout: Verifying : protobuf-compiler-3.14.0-17.el9.x86_64 58/96 2026-02-26T03:30:34.286 INFO:teuthology.orchestra.run.trial132.stdout: Verifying : abseil-cpp-20211102.0-4.el9.x86_64 59/96 2026-02-26T03:30:34.286 INFO:teuthology.orchestra.run.trial132.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 60/96 2026-02-26T03:30:34.286 INFO:teuthology.orchestra.run.trial132.stdout: Verifying : luarocks-3.9.2-5.el9.noarch 61/96 2026-02-26T03:30:34.286 INFO:teuthology.orchestra.run.trial132.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 62/96 2026-02-26T03:30:34.286 INFO:teuthology.orchestra.run.trial132.stdout: Verifying : python3-autocommand-2.2.2-8.el9.noarch 63/96 2026-02-26T03:30:34.286 INFO:teuthology.orchestra.run.trial132.stdout: Verifying : python3-backports-tarfile-1.2.0-1.el9.noarch 64/96 2026-02-26T03:30:34.286 INFO:teuthology.orchestra.run.trial132.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 65/96 2026-02-26T03:30:34.287 INFO:teuthology.orchestra.run.trial132.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 66/96 2026-02-26T03:30:34.287 INFO:teuthology.orchestra.run.trial132.stdout: Verifying : python3-certifi-2023.05.07-4.el9.noarch 67/96 2026-02-26T03:30:34.287 INFO:teuthology.orchestra.run.trial132.stdout: Verifying : python3-cheroot-10.0.1-4.el9.noarch 68/96 2026-02-26T03:30:34.287 INFO:teuthology.orchestra.run.trial132.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 69/96 2026-02-26T03:30:34.287 INFO:teuthology.orchestra.run.trial132.stdout: Verifying : python3-google-auth-1:2.45.0-1.el9.noarch 70/96 2026-02-26T03:30:34.287 INFO:teuthology.orchestra.run.trial132.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 71/96 2026-02-26T03:30:34.287 INFO:teuthology.orchestra.run.trial132.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 72/96 2026-02-26T03:30:34.287 INFO:teuthology.orchestra.run.trial132.stdout: Verifying : python3-influxdb-5.3.1-1.el9.noarch 73/96 2026-02-26T03:30:34.287 INFO:teuthology.orchestra.run.trial132.stdout: Verifying : python3-isodate-0.6.1-3.el9.noarch 74/96 2026-02-26T03:30:34.287 INFO:teuthology.orchestra.run.trial132.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 75/96 2026-02-26T03:30:34.287 INFO:teuthology.orchestra.run.trial132.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 76/96 2026-02-26T03:30:34.287 INFO:teuthology.orchestra.run.trial132.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 77/96 2026-02-26T03:30:34.287 INFO:teuthology.orchestra.run.trial132.stdout: Verifying : python3-jaraco-context-6.0.1-3.el9.noarch 78/96 2026-02-26T03:30:34.288 INFO:teuthology.orchestra.run.trial132.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 79/96 2026-02-26T03:30:34.288 INFO:teuthology.orchestra.run.trial132.stdout: Verifying : python3-jaraco-text-4.0.0-2.el9.noarch 80/96 2026-02-26T03:30:34.288 INFO:teuthology.orchestra.run.trial132.stdout: Verifying : python3-kubernetes-1:26.1.0-3.el9.noarch 81/96 2026-02-26T03:30:34.288 INFO:teuthology.orchestra.run.trial132.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 82/96 2026-02-26T03:30:34.288 INFO:teuthology.orchestra.run.trial132.stdout: Verifying : python3-msgpack-1.0.3-2.el9.x86_64 83/96 2026-02-26T03:30:34.288 INFO:teuthology.orchestra.run.trial132.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 84/96 2026-02-26T03:30:34.288 INFO:teuthology.orchestra.run.trial132.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 85/96 2026-02-26T03:30:34.288 INFO:teuthology.orchestra.run.trial132.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 86/96 2026-02-26T03:30:34.288 INFO:teuthology.orchestra.run.trial132.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 87/96 2026-02-26T03:30:34.288 INFO:teuthology.orchestra.run.trial132.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 88/96 2026-02-26T03:30:34.288 INFO:teuthology.orchestra.run.trial132.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 89/96 2026-02-26T03:30:34.288 INFO:teuthology.orchestra.run.trial132.stdout: Verifying : python3-saml-1.16.0-1.el9.noarch 90/96 2026-02-26T03:30:34.288 INFO:teuthology.orchestra.run.trial132.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 91/96 2026-02-26T03:30:34.289 INFO:teuthology.orchestra.run.trial132.stdout: Verifying : python3-typing-extensions-4.15.0-1.el9.noarch 92/96 2026-02-26T03:30:34.289 INFO:teuthology.orchestra.run.trial132.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 93/96 2026-02-26T03:30:34.289 INFO:teuthology.orchestra.run.trial132.stdout: Verifying : python3-xmlsec-1.3.13-1.el9.x86_64 94/96 2026-02-26T03:30:34.289 INFO:teuthology.orchestra.run.trial132.stdout: Verifying : python3-xmltodict-0.12.0-15.el9.noarch 95/96 2026-02-26T03:30:34.349 INFO:teuthology.orchestra.run.trial132.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 96/96 2026-02-26T03:30:34.350 INFO:teuthology.orchestra.run.trial132.stdout: 2026-02-26T03:30:34.350 INFO:teuthology.orchestra.run.trial132.stdout:Installed: 2026-02-26T03:30:34.350 INFO:teuthology.orchestra.run.trial132.stdout: abseil-cpp-20211102.0-4.el9.x86_64 2026-02-26T03:30:34.350 INFO:teuthology.orchestra.run.trial132.stdout: ceph-2:20.3.0-5312.gabffb617.el9.x86_64 2026-02-26T03:30:34.350 INFO:teuthology.orchestra.run.trial132.stdout: ceph-grafana-dashboards-2:20.3.0-5312.gabffb617.el9.noarch 2026-02-26T03:30:34.350 INFO:teuthology.orchestra.run.trial132.stdout: ceph-mds-2:20.3.0-5312.gabffb617.el9.x86_64 2026-02-26T03:30:34.350 INFO:teuthology.orchestra.run.trial132.stdout: ceph-mgr-2:20.3.0-5312.gabffb617.el9.x86_64 2026-02-26T03:30:34.350 INFO:teuthology.orchestra.run.trial132.stdout: ceph-mgr-cephadm-2:20.3.0-5312.gabffb617.el9.noarch 2026-02-26T03:30:34.351 INFO:teuthology.orchestra.run.trial132.stdout: ceph-mgr-dashboard-2:20.3.0-5312.gabffb617.el9.noarch 2026-02-26T03:30:34.351 INFO:teuthology.orchestra.run.trial132.stdout: ceph-mgr-diskprediction-local-2:20.3.0-5312.gabffb617.el9.noarch 2026-02-26T03:30:34.351 INFO:teuthology.orchestra.run.trial132.stdout: ceph-mgr-k8sevents-2:20.3.0-5312.gabffb617.el9.noarch 2026-02-26T03:30:34.351 INFO:teuthology.orchestra.run.trial132.stdout: ceph-mgr-modules-core-2:20.3.0-5312.gabffb617.el9.noarch 2026-02-26T03:30:34.351 INFO:teuthology.orchestra.run.trial132.stdout: ceph-mgr-rook-2:20.3.0-5312.gabffb617.el9.noarch 2026-02-26T03:30:34.351 INFO:teuthology.orchestra.run.trial132.stdout: ceph-mon-2:20.3.0-5312.gabffb617.el9.x86_64 2026-02-26T03:30:34.351 INFO:teuthology.orchestra.run.trial132.stdout: ceph-osd-2:20.3.0-5312.gabffb617.el9.x86_64 2026-02-26T03:30:34.351 INFO:teuthology.orchestra.run.trial132.stdout: ceph-osd-classic-2:20.3.0-5312.gabffb617.el9.x86_64 2026-02-26T03:30:34.351 INFO:teuthology.orchestra.run.trial132.stdout: ceph-osd-crimson-2:20.3.0-5312.gabffb617.el9.x86_64 2026-02-26T03:30:34.351 INFO:teuthology.orchestra.run.trial132.stdout: ceph-prometheus-alerts-2:20.3.0-5312.gabffb617.el9.noarch 2026-02-26T03:30:34.351 INFO:teuthology.orchestra.run.trial132.stdout: ceph-volume-2:20.3.0-5312.gabffb617.el9.noarch 2026-02-26T03:30:34.351 INFO:teuthology.orchestra.run.trial132.stdout: cephadm-2:20.3.0-5312.gabffb617.el9.noarch 2026-02-26T03:30:34.352 INFO:teuthology.orchestra.run.trial132.stdout: cryptsetup-2.8.1-3.el9.x86_64 2026-02-26T03:30:34.352 INFO:teuthology.orchestra.run.trial132.stdout: flexiblas-3.0.4-9.el9.x86_64 2026-02-26T03:30:34.352 INFO:teuthology.orchestra.run.trial132.stdout: flexiblas-netlib-3.0.4-9.el9.x86_64 2026-02-26T03:30:34.352 INFO:teuthology.orchestra.run.trial132.stdout: flexiblas-openblas-openmp-3.0.4-9.el9.x86_64 2026-02-26T03:30:34.352 INFO:teuthology.orchestra.run.trial132.stdout: grpc-data-1.46.7-10.el9.noarch 2026-02-26T03:30:34.352 INFO:teuthology.orchestra.run.trial132.stdout: ledmon-libs-1.1.0-3.el9.x86_64 2026-02-26T03:30:34.352 INFO:teuthology.orchestra.run.trial132.stdout: libconfig-1.7.2-9.el9.x86_64 2026-02-26T03:30:34.352 INFO:teuthology.orchestra.run.trial132.stdout: libgfortran-11.5.0-14.el9.x86_64 2026-02-26T03:30:34.352 INFO:teuthology.orchestra.run.trial132.stdout: libquadmath-11.5.0-14.el9.x86_64 2026-02-26T03:30:34.352 INFO:teuthology.orchestra.run.trial132.stdout: libstoragemgmt-1.10.1-1.el9.x86_64 2026-02-26T03:30:34.352 INFO:teuthology.orchestra.run.trial132.stdout: lua-5.4.4-4.el9.x86_64 2026-02-26T03:30:34.352 INFO:teuthology.orchestra.run.trial132.stdout: lua-devel-5.4.4-4.el9.x86_64 2026-02-26T03:30:34.353 INFO:teuthology.orchestra.run.trial132.stdout: luarocks-3.9.2-5.el9.noarch 2026-02-26T03:30:34.353 INFO:teuthology.orchestra.run.trial132.stdout: openblas-0.3.29-1.el9.x86_64 2026-02-26T03:30:34.353 INFO:teuthology.orchestra.run.trial132.stdout: openblas-openmp-0.3.29-1.el9.x86_64 2026-02-26T03:30:34.353 INFO:teuthology.orchestra.run.trial132.stdout: protobuf-compiler-3.14.0-17.el9.x86_64 2026-02-26T03:30:34.353 INFO:teuthology.orchestra.run.trial132.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2026-02-26T03:30:34.353 INFO:teuthology.orchestra.run.trial132.stdout: python3-autocommand-2.2.2-8.el9.noarch 2026-02-26T03:30:34.353 INFO:teuthology.orchestra.run.trial132.stdout: python3-babel-2.9.1-2.el9.noarch 2026-02-26T03:30:34.353 INFO:teuthology.orchestra.run.trial132.stdout: python3-backports-tarfile-1.2.0-1.el9.noarch 2026-02-26T03:30:34.353 INFO:teuthology.orchestra.run.trial132.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2026-02-26T03:30:34.353 INFO:teuthology.orchestra.run.trial132.stdout: python3-cachetools-4.2.4-1.el9.noarch 2026-02-26T03:30:34.353 INFO:teuthology.orchestra.run.trial132.stdout: python3-certifi-2023.05.07-4.el9.noarch 2026-02-26T03:30:34.353 INFO:teuthology.orchestra.run.trial132.stdout: python3-cffi-1.14.5-5.el9.x86_64 2026-02-26T03:30:34.354 INFO:teuthology.orchestra.run.trial132.stdout: python3-cheroot-10.0.1-4.el9.noarch 2026-02-26T03:30:34.354 INFO:teuthology.orchestra.run.trial132.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2026-02-26T03:30:34.354 INFO:teuthology.orchestra.run.trial132.stdout: python3-cryptography-36.0.1-5.el9.x86_64 2026-02-26T03:30:34.354 INFO:teuthology.orchestra.run.trial132.stdout: python3-devel-3.9.25-3.el9.x86_64 2026-02-26T03:30:34.354 INFO:teuthology.orchestra.run.trial132.stdout: python3-google-auth-1:2.45.0-1.el9.noarch 2026-02-26T03:30:34.354 INFO:teuthology.orchestra.run.trial132.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2026-02-26T03:30:34.354 INFO:teuthology.orchestra.run.trial132.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2026-02-26T03:30:34.354 INFO:teuthology.orchestra.run.trial132.stdout: python3-influxdb-5.3.1-1.el9.noarch 2026-02-26T03:30:34.354 INFO:teuthology.orchestra.run.trial132.stdout: python3-isodate-0.6.1-3.el9.noarch 2026-02-26T03:30:34.354 INFO:teuthology.orchestra.run.trial132.stdout: python3-jaraco-8.2.1-3.el9.noarch 2026-02-26T03:30:34.354 INFO:teuthology.orchestra.run.trial132.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2026-02-26T03:30:34.354 INFO:teuthology.orchestra.run.trial132.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2026-02-26T03:30:34.355 INFO:teuthology.orchestra.run.trial132.stdout: python3-jaraco-context-6.0.1-3.el9.noarch 2026-02-26T03:30:34.355 INFO:teuthology.orchestra.run.trial132.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2026-02-26T03:30:34.355 INFO:teuthology.orchestra.run.trial132.stdout: python3-jaraco-text-4.0.0-2.el9.noarch 2026-02-26T03:30:34.355 INFO:teuthology.orchestra.run.trial132.stdout: python3-jinja2-2.11.3-8.el9.noarch 2026-02-26T03:30:34.355 INFO:teuthology.orchestra.run.trial132.stdout: python3-jmespath-1.0.1-1.el9.noarch 2026-02-26T03:30:34.355 INFO:teuthology.orchestra.run.trial132.stdout: python3-kubernetes-1:26.1.0-3.el9.noarch 2026-02-26T03:30:34.355 INFO:teuthology.orchestra.run.trial132.stdout: python3-libstoragemgmt-1.10.1-1.el9.x86_64 2026-02-26T03:30:34.355 INFO:teuthology.orchestra.run.trial132.stdout: python3-lxml-4.6.5-3.el9.x86_64 2026-02-26T03:30:34.355 INFO:teuthology.orchestra.run.trial132.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2026-02-26T03:30:34.355 INFO:teuthology.orchestra.run.trial132.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2026-02-26T03:30:34.355 INFO:teuthology.orchestra.run.trial132.stdout: python3-msgpack-1.0.3-2.el9.x86_64 2026-02-26T03:30:34.355 INFO:teuthology.orchestra.run.trial132.stdout: python3-natsort-7.1.1-5.el9.noarch 2026-02-26T03:30:34.356 INFO:teuthology.orchestra.run.trial132.stdout: python3-numpy-1:1.23.5-2.el9.x86_64 2026-02-26T03:30:34.356 INFO:teuthology.orchestra.run.trial132.stdout: python3-numpy-f2py-1:1.23.5-2.el9.x86_64 2026-02-26T03:30:34.356 INFO:teuthology.orchestra.run.trial132.stdout: python3-packaging-20.9-5.el9.noarch 2026-02-26T03:30:34.356 INFO:teuthology.orchestra.run.trial132.stdout: python3-ply-3.11-14.el9.noarch 2026-02-26T03:30:34.356 INFO:teuthology.orchestra.run.trial132.stdout: python3-portend-3.1.0-2.el9.noarch 2026-02-26T03:30:34.356 INFO:teuthology.orchestra.run.trial132.stdout: python3-protobuf-3.14.0-17.el9.noarch 2026-02-26T03:30:34.356 INFO:teuthology.orchestra.run.trial132.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2026-02-26T03:30:34.356 INFO:teuthology.orchestra.run.trial132.stdout: python3-pyasn1-0.4.8-7.el9.noarch 2026-02-26T03:30:34.356 INFO:teuthology.orchestra.run.trial132.stdout: python3-pyasn1-modules-0.4.8-7.el9.noarch 2026-02-26T03:30:34.356 INFO:teuthology.orchestra.run.trial132.stdout: python3-pycparser-2.20-6.el9.noarch 2026-02-26T03:30:34.356 INFO:teuthology.orchestra.run.trial132.stdout: python3-pyparsing-2.4.7-9.el9.noarch 2026-02-26T03:30:34.356 INFO:teuthology.orchestra.run.trial132.stdout: python3-repoze-lru-0.7-16.el9.noarch 2026-02-26T03:30:34.356 INFO:teuthology.orchestra.run.trial132.stdout: python3-requests-2.25.1-10.el9.noarch 2026-02-26T03:30:34.357 INFO:teuthology.orchestra.run.trial132.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2026-02-26T03:30:34.357 INFO:teuthology.orchestra.run.trial132.stdout: python3-routes-2.5.1-5.el9.noarch 2026-02-26T03:30:34.357 INFO:teuthology.orchestra.run.trial132.stdout: python3-rsa-4.9-2.el9.noarch 2026-02-26T03:30:34.357 INFO:teuthology.orchestra.run.trial132.stdout: python3-saml-1.16.0-1.el9.noarch 2026-02-26T03:30:34.357 INFO:teuthology.orchestra.run.trial132.stdout: python3-scipy-1.9.3-2.el9.x86_64 2026-02-26T03:30:34.357 INFO:teuthology.orchestra.run.trial132.stdout: python3-tempora-5.0.0-2.el9.noarch 2026-02-26T03:30:34.357 INFO:teuthology.orchestra.run.trial132.stdout: python3-toml-0.10.2-6.el9.noarch 2026-02-26T03:30:34.357 INFO:teuthology.orchestra.run.trial132.stdout: python3-typing-extensions-4.15.0-1.el9.noarch 2026-02-26T03:30:34.357 INFO:teuthology.orchestra.run.trial132.stdout: python3-urllib3-1.26.5-7.el9.noarch 2026-02-26T03:30:34.357 INFO:teuthology.orchestra.run.trial132.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2026-02-26T03:30:34.357 INFO:teuthology.orchestra.run.trial132.stdout: python3-xmlsec-1.3.13-1.el9.x86_64 2026-02-26T03:30:34.357 INFO:teuthology.orchestra.run.trial132.stdout: python3-xmltodict-0.12.0-15.el9.noarch 2026-02-26T03:30:34.357 INFO:teuthology.orchestra.run.trial132.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2026-02-26T03:30:34.358 INFO:teuthology.orchestra.run.trial132.stdout: unzip-6.0-59.el9.x86_64 2026-02-26T03:30:34.358 INFO:teuthology.orchestra.run.trial132.stdout: xmlsec1-1.2.29-13.el9.x86_64 2026-02-26T03:30:34.358 INFO:teuthology.orchestra.run.trial132.stdout: xmlsec1-openssl-1.2.29-13.el9.x86_64 2026-02-26T03:30:34.358 INFO:teuthology.orchestra.run.trial132.stdout: zip-3.0-35.el9.x86_64 2026-02-26T03:30:34.358 INFO:teuthology.orchestra.run.trial132.stdout: 2026-02-26T03:30:34.358 INFO:teuthology.orchestra.run.trial132.stdout:Complete! 2026-02-26T03:30:34.415 DEBUG:teuthology.orchestra.run.trial132:> sudo yum -y install ceph-base 2026-02-26T03:30:34.730 INFO:teuthology.orchestra.run.trial132.stdout:Last metadata expiration check: 0:09:14 ago on Thu 26 Feb 2026 03:21:20 AM UTC. 2026-02-26T03:30:34.795 INFO:teuthology.orchestra.run.trial132.stdout:Package ceph-base-2:20.3.0-5312.gabffb617.el9.x86_64 is already installed. 2026-02-26T03:30:34.814 INFO:teuthology.orchestra.run.trial132.stdout:Dependencies resolved. 2026-02-26T03:30:34.814 INFO:teuthology.orchestra.run.trial132.stdout:Nothing to do. 2026-02-26T03:30:34.814 INFO:teuthology.orchestra.run.trial132.stdout:Complete! 2026-02-26T03:30:34.835 DEBUG:teuthology.orchestra.run.trial132:> sudo yum -y install cephadm 2026-02-26T03:30:35.150 INFO:teuthology.orchestra.run.trial132.stdout:Last metadata expiration check: 0:09:15 ago on Thu 26 Feb 2026 03:21:20 AM UTC. 2026-02-26T03:30:35.218 INFO:teuthology.orchestra.run.trial132.stdout:Package cephadm-2:20.3.0-5312.gabffb617.el9.noarch is already installed. 2026-02-26T03:30:35.236 INFO:teuthology.orchestra.run.trial132.stdout:Dependencies resolved. 2026-02-26T03:30:35.237 INFO:teuthology.orchestra.run.trial132.stdout:Nothing to do. 2026-02-26T03:30:35.237 INFO:teuthology.orchestra.run.trial132.stdout:Complete! 2026-02-26T03:30:35.258 DEBUG:teuthology.orchestra.run.trial132:> sudo yum -y install ceph-immutable-object-cache 2026-02-26T03:30:35.577 INFO:teuthology.orchestra.run.trial132.stdout:Last metadata expiration check: 0:09:15 ago on Thu 26 Feb 2026 03:21:20 AM UTC. 2026-02-26T03:30:35.661 INFO:teuthology.orchestra.run.trial132.stdout:Dependencies resolved. 2026-02-26T03:30:35.661 INFO:teuthology.orchestra.run.trial132.stdout:================================================================================ 2026-02-26T03:30:35.662 INFO:teuthology.orchestra.run.trial132.stdout: Package Arch Version Repo Size 2026-02-26T03:30:35.662 INFO:teuthology.orchestra.run.trial132.stdout:================================================================================ 2026-02-26T03:30:35.662 INFO:teuthology.orchestra.run.trial132.stdout:Installing: 2026-02-26T03:30:35.662 INFO:teuthology.orchestra.run.trial132.stdout: ceph-immutable-object-cache x86_64 2:20.3.0-5312.gabffb617.el9 ceph 149 k 2026-02-26T03:30:35.662 INFO:teuthology.orchestra.run.trial132.stdout: 2026-02-26T03:30:35.662 INFO:teuthology.orchestra.run.trial132.stdout:Transaction Summary 2026-02-26T03:30:35.662 INFO:teuthology.orchestra.run.trial132.stdout:================================================================================ 2026-02-26T03:30:35.663 INFO:teuthology.orchestra.run.trial132.stdout:Install 1 Package 2026-02-26T03:30:35.663 INFO:teuthology.orchestra.run.trial132.stdout: 2026-02-26T03:30:35.663 INFO:teuthology.orchestra.run.trial132.stdout:Total download size: 149 k 2026-02-26T03:30:35.663 INFO:teuthology.orchestra.run.trial132.stdout:Installed size: 455 k 2026-02-26T03:30:35.663 INFO:teuthology.orchestra.run.trial132.stdout:Downloading Packages: 2026-02-26T03:30:36.234 INFO:teuthology.orchestra.run.trial132.stdout:ceph-immutable-object-cache-20.3.0-5312.gabffb6 261 kB/s | 149 kB 00:00 2026-02-26T03:30:36.235 INFO:teuthology.orchestra.run.trial132.stdout:-------------------------------------------------------------------------------- 2026-02-26T03:30:36.235 INFO:teuthology.orchestra.run.trial132.stdout:Total 260 kB/s | 149 kB 00:00 2026-02-26T03:30:36.235 INFO:teuthology.orchestra.run.trial132.stdout:Running transaction check 2026-02-26T03:30:36.239 INFO:teuthology.orchestra.run.trial132.stdout:Transaction check succeeded. 2026-02-26T03:30:36.239 INFO:teuthology.orchestra.run.trial132.stdout:Running transaction test 2026-02-26T03:30:36.270 INFO:teuthology.orchestra.run.trial132.stdout:Transaction test succeeded. 2026-02-26T03:30:36.270 INFO:teuthology.orchestra.run.trial132.stdout:Running transaction 2026-02-26T03:30:36.327 INFO:teuthology.orchestra.run.trial132.stdout: Preparing : 1/1 2026-02-26T03:30:36.328 INFO:teuthology.orchestra.run.trial132.stdout: Installing : ceph-immutable-object-cache-2:20.3.0-5312.gabffb617. 1/1 2026-02-26T03:30:36.351 INFO:teuthology.orchestra.run.trial132.stdout: Running scriptlet: ceph-immutable-object-cache-2:20.3.0-5312.gabffb617. 1/1 2026-02-26T03:30:36.351 INFO:teuthology.orchestra.run.trial132.stdout:Glob pattern passed to enable, but globs are not supported for this. 2026-02-26T03:30:36.351 INFO:teuthology.orchestra.run.trial132.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2026-02-26T03:30:36.351 INFO:teuthology.orchestra.run.trial132.stdout: 2026-02-26T03:30:36.666 INFO:teuthology.orchestra.run.trial132.stdout: Verifying : ceph-immutable-object-cache-2:20.3.0-5312.gabffb617. 1/1 2026-02-26T03:30:36.666 INFO:teuthology.orchestra.run.trial132.stdout: 2026-02-26T03:30:36.666 INFO:teuthology.orchestra.run.trial132.stdout:Installed: 2026-02-26T03:30:36.666 INFO:teuthology.orchestra.run.trial132.stdout: ceph-immutable-object-cache-2:20.3.0-5312.gabffb617.el9.x86_64 2026-02-26T03:30:36.666 INFO:teuthology.orchestra.run.trial132.stdout: 2026-02-26T03:30:36.666 INFO:teuthology.orchestra.run.trial132.stdout:Complete! 2026-02-26T03:30:36.694 DEBUG:teuthology.orchestra.run.trial132:> sudo yum -y install ceph-mgr 2026-02-26T03:30:37.013 INFO:teuthology.orchestra.run.trial132.stdout:Last metadata expiration check: 0:09:17 ago on Thu 26 Feb 2026 03:21:20 AM UTC. 2026-02-26T03:30:37.079 INFO:teuthology.orchestra.run.trial132.stdout:Package ceph-mgr-2:20.3.0-5312.gabffb617.el9.x86_64 is already installed. 2026-02-26T03:30:37.097 INFO:teuthology.orchestra.run.trial132.stdout:Dependencies resolved. 2026-02-26T03:30:37.098 INFO:teuthology.orchestra.run.trial132.stdout:Nothing to do. 2026-02-26T03:30:37.098 INFO:teuthology.orchestra.run.trial132.stdout:Complete! 2026-02-26T03:30:37.119 DEBUG:teuthology.orchestra.run.trial132:> sudo yum -y install ceph-mgr-dashboard 2026-02-26T03:30:37.436 INFO:teuthology.orchestra.run.trial132.stdout:Last metadata expiration check: 0:09:17 ago on Thu 26 Feb 2026 03:21:20 AM UTC. 2026-02-26T03:30:37.501 INFO:teuthology.orchestra.run.trial132.stdout:Package ceph-mgr-dashboard-2:20.3.0-5312.gabffb617.el9.noarch is already installed. 2026-02-26T03:30:37.520 INFO:teuthology.orchestra.run.trial132.stdout:Dependencies resolved. 2026-02-26T03:30:37.521 INFO:teuthology.orchestra.run.trial132.stdout:Nothing to do. 2026-02-26T03:30:37.521 INFO:teuthology.orchestra.run.trial132.stdout:Complete! 2026-02-26T03:30:37.542 DEBUG:teuthology.orchestra.run.trial132:> sudo yum -y install ceph-mgr-diskprediction-local 2026-02-26T03:30:37.855 INFO:teuthology.orchestra.run.trial132.stdout:Last metadata expiration check: 0:09:17 ago on Thu 26 Feb 2026 03:21:20 AM UTC. 2026-02-26T03:30:37.921 INFO:teuthology.orchestra.run.trial132.stdout:Package ceph-mgr-diskprediction-local-2:20.3.0-5312.gabffb617.el9.noarch is already installed. 2026-02-26T03:30:37.939 INFO:teuthology.orchestra.run.trial132.stdout:Dependencies resolved. 2026-02-26T03:30:37.940 INFO:teuthology.orchestra.run.trial132.stdout:Nothing to do. 2026-02-26T03:30:37.940 INFO:teuthology.orchestra.run.trial132.stdout:Complete! 2026-02-26T03:30:37.961 DEBUG:teuthology.orchestra.run.trial132:> sudo yum -y install ceph-mgr-rook 2026-02-26T03:30:38.277 INFO:teuthology.orchestra.run.trial132.stdout:Last metadata expiration check: 0:09:18 ago on Thu 26 Feb 2026 03:21:20 AM UTC. 2026-02-26T03:30:38.343 INFO:teuthology.orchestra.run.trial132.stdout:Package ceph-mgr-rook-2:20.3.0-5312.gabffb617.el9.noarch is already installed. 2026-02-26T03:30:38.361 INFO:teuthology.orchestra.run.trial132.stdout:Dependencies resolved. 2026-02-26T03:30:38.362 INFO:teuthology.orchestra.run.trial132.stdout:Nothing to do. 2026-02-26T03:30:38.362 INFO:teuthology.orchestra.run.trial132.stdout:Complete! 2026-02-26T03:30:38.383 DEBUG:teuthology.orchestra.run.trial132:> sudo yum -y install ceph-mgr-cephadm 2026-02-26T03:30:38.701 INFO:teuthology.orchestra.run.trial132.stdout:Last metadata expiration check: 0:09:18 ago on Thu 26 Feb 2026 03:21:20 AM UTC. 2026-02-26T03:30:38.767 INFO:teuthology.orchestra.run.trial132.stdout:Package ceph-mgr-cephadm-2:20.3.0-5312.gabffb617.el9.noarch is already installed. 2026-02-26T03:30:38.786 INFO:teuthology.orchestra.run.trial132.stdout:Dependencies resolved. 2026-02-26T03:30:38.787 INFO:teuthology.orchestra.run.trial132.stdout:Nothing to do. 2026-02-26T03:30:38.787 INFO:teuthology.orchestra.run.trial132.stdout:Complete! 2026-02-26T03:30:38.808 DEBUG:teuthology.orchestra.run.trial132:> sudo yum -y install ceph-osd 2026-02-26T03:30:39.128 INFO:teuthology.orchestra.run.trial132.stdout:Last metadata expiration check: 0:09:19 ago on Thu 26 Feb 2026 03:21:20 AM UTC. 2026-02-26T03:30:39.195 INFO:teuthology.orchestra.run.trial132.stdout:Package ceph-osd-2:20.3.0-5312.gabffb617.el9.x86_64 is already installed. 2026-02-26T03:30:39.214 INFO:teuthology.orchestra.run.trial132.stdout:Dependencies resolved. 2026-02-26T03:30:39.215 INFO:teuthology.orchestra.run.trial132.stdout:Nothing to do. 2026-02-26T03:30:39.215 INFO:teuthology.orchestra.run.trial132.stdout:Complete! 2026-02-26T03:30:39.237 DEBUG:teuthology.orchestra.run.trial132:> sudo yum -y install ceph-osd-classic 2026-02-26T03:30:39.556 INFO:teuthology.orchestra.run.trial132.stdout:Last metadata expiration check: 0:09:19 ago on Thu 26 Feb 2026 03:21:20 AM UTC. 2026-02-26T03:30:39.621 INFO:teuthology.orchestra.run.trial132.stdout:Package ceph-osd-classic-2:20.3.0-5312.gabffb617.el9.x86_64 is already installed. 2026-02-26T03:30:39.640 INFO:teuthology.orchestra.run.trial132.stdout:Dependencies resolved. 2026-02-26T03:30:39.640 INFO:teuthology.orchestra.run.trial132.stdout:Nothing to do. 2026-02-26T03:30:39.640 INFO:teuthology.orchestra.run.trial132.stdout:Complete! 2026-02-26T03:30:39.663 DEBUG:teuthology.orchestra.run.trial132:> sudo yum -y install ceph-fuse 2026-02-26T03:30:39.977 INFO:teuthology.orchestra.run.trial132.stdout:Last metadata expiration check: 0:09:19 ago on Thu 26 Feb 2026 03:21:20 AM UTC. 2026-02-26T03:30:40.064 INFO:teuthology.orchestra.run.trial132.stdout:Dependencies resolved. 2026-02-26T03:30:40.065 INFO:teuthology.orchestra.run.trial132.stdout:================================================================================ 2026-02-26T03:30:40.065 INFO:teuthology.orchestra.run.trial132.stdout: Package Arch Version Repo Size 2026-02-26T03:30:40.065 INFO:teuthology.orchestra.run.trial132.stdout:================================================================================ 2026-02-26T03:30:40.065 INFO:teuthology.orchestra.run.trial132.stdout:Installing: 2026-02-26T03:30:40.065 INFO:teuthology.orchestra.run.trial132.stdout: ceph-fuse x86_64 2:20.3.0-5312.gabffb617.el9 ceph 914 k 2026-02-26T03:30:40.065 INFO:teuthology.orchestra.run.trial132.stdout: 2026-02-26T03:30:40.065 INFO:teuthology.orchestra.run.trial132.stdout:Transaction Summary 2026-02-26T03:30:40.065 INFO:teuthology.orchestra.run.trial132.stdout:================================================================================ 2026-02-26T03:30:40.066 INFO:teuthology.orchestra.run.trial132.stdout:Install 1 Package 2026-02-26T03:30:40.066 INFO:teuthology.orchestra.run.trial132.stdout: 2026-02-26T03:30:40.066 INFO:teuthology.orchestra.run.trial132.stdout:Total download size: 914 k 2026-02-26T03:30:40.066 INFO:teuthology.orchestra.run.trial132.stdout:Installed size: 2.9 M 2026-02-26T03:30:40.066 INFO:teuthology.orchestra.run.trial132.stdout:Downloading Packages: 2026-02-26T03:30:40.969 INFO:teuthology.orchestra.run.trial132.stdout:ceph-fuse-20.3.0-5312.gabffb617.el9.x86_64.rpm 1.0 MB/s | 914 kB 00:00 2026-02-26T03:30:40.969 INFO:teuthology.orchestra.run.trial132.stdout:-------------------------------------------------------------------------------- 2026-02-26T03:30:40.970 INFO:teuthology.orchestra.run.trial132.stdout:Total 1.0 MB/s | 914 kB 00:00 2026-02-26T03:30:40.970 INFO:teuthology.orchestra.run.trial132.stdout:Running transaction check 2026-02-26T03:30:40.974 INFO:teuthology.orchestra.run.trial132.stdout:Transaction check succeeded. 2026-02-26T03:30:40.974 INFO:teuthology.orchestra.run.trial132.stdout:Running transaction test 2026-02-26T03:30:41.008 INFO:teuthology.orchestra.run.trial132.stdout:Transaction test succeeded. 2026-02-26T03:30:41.008 INFO:teuthology.orchestra.run.trial132.stdout:Running transaction 2026-02-26T03:30:41.101 INFO:teuthology.orchestra.run.trial132.stdout: Preparing : 1/1 2026-02-26T03:30:41.119 INFO:teuthology.orchestra.run.trial132.stdout: Installing : ceph-fuse-2:20.3.0-5312.gabffb617.el9.x86_64 1/1 2026-02-26T03:30:41.366 INFO:teuthology.orchestra.run.trial132.stdout: Running scriptlet: ceph-fuse-2:20.3.0-5312.gabffb617.el9.x86_64 1/1 2026-02-26T03:30:41.429 INFO:teuthology.orchestra.run.trial132.stdout: Verifying : ceph-fuse-2:20.3.0-5312.gabffb617.el9.x86_64 1/1 2026-02-26T03:30:41.430 INFO:teuthology.orchestra.run.trial132.stdout: 2026-02-26T03:30:41.430 INFO:teuthology.orchestra.run.trial132.stdout:Installed: 2026-02-26T03:30:41.430 INFO:teuthology.orchestra.run.trial132.stdout: ceph-fuse-2:20.3.0-5312.gabffb617.el9.x86_64 2026-02-26T03:30:41.430 INFO:teuthology.orchestra.run.trial132.stdout: 2026-02-26T03:30:41.430 INFO:teuthology.orchestra.run.trial132.stdout:Complete! 2026-02-26T03:30:41.458 DEBUG:teuthology.orchestra.run.trial132:> sudo yum -y install ceph-volume 2026-02-26T03:30:41.774 INFO:teuthology.orchestra.run.trial132.stdout:Last metadata expiration check: 0:09:21 ago on Thu 26 Feb 2026 03:21:20 AM UTC. 2026-02-26T03:30:41.840 INFO:teuthology.orchestra.run.trial132.stdout:Package ceph-volume-2:20.3.0-5312.gabffb617.el9.noarch is already installed. 2026-02-26T03:30:41.859 INFO:teuthology.orchestra.run.trial132.stdout:Dependencies resolved. 2026-02-26T03:30:41.859 INFO:teuthology.orchestra.run.trial132.stdout:Nothing to do. 2026-02-26T03:30:41.860 INFO:teuthology.orchestra.run.trial132.stdout:Complete! 2026-02-26T03:30:41.881 DEBUG:teuthology.orchestra.run.trial132:> sudo yum -y install librados-devel 2026-02-26T03:30:42.201 INFO:teuthology.orchestra.run.trial132.stdout:Last metadata expiration check: 0:09:22 ago on Thu 26 Feb 2026 03:21:20 AM UTC. 2026-02-26T03:30:42.284 INFO:teuthology.orchestra.run.trial132.stdout:Dependencies resolved. 2026-02-26T03:30:42.285 INFO:teuthology.orchestra.run.trial132.stdout:================================================================================ 2026-02-26T03:30:42.286 INFO:teuthology.orchestra.run.trial132.stdout: Package Arch Version Repo Size 2026-02-26T03:30:42.286 INFO:teuthology.orchestra.run.trial132.stdout:================================================================================ 2026-02-26T03:30:42.286 INFO:teuthology.orchestra.run.trial132.stdout:Installing: 2026-02-26T03:30:42.286 INFO:teuthology.orchestra.run.trial132.stdout: librados-devel x86_64 2:20.3.0-5312.gabffb617.el9 ceph 124 k 2026-02-26T03:30:42.286 INFO:teuthology.orchestra.run.trial132.stdout: 2026-02-26T03:30:42.286 INFO:teuthology.orchestra.run.trial132.stdout:Transaction Summary 2026-02-26T03:30:42.286 INFO:teuthology.orchestra.run.trial132.stdout:================================================================================ 2026-02-26T03:30:42.286 INFO:teuthology.orchestra.run.trial132.stdout:Install 1 Package 2026-02-26T03:30:42.286 INFO:teuthology.orchestra.run.trial132.stdout: 2026-02-26T03:30:42.286 INFO:teuthology.orchestra.run.trial132.stdout:Total download size: 124 k 2026-02-26T03:30:42.286 INFO:teuthology.orchestra.run.trial132.stdout:Installed size: 434 k 2026-02-26T03:30:42.287 INFO:teuthology.orchestra.run.trial132.stdout:Downloading Packages: 2026-02-26T03:30:42.778 INFO:teuthology.orchestra.run.trial132.stdout:librados-devel-20.3.0-5312.gabffb617.el9.x86_64 251 kB/s | 124 kB 00:00 2026-02-26T03:30:42.778 INFO:teuthology.orchestra.run.trial132.stdout:-------------------------------------------------------------------------------- 2026-02-26T03:30:42.779 INFO:teuthology.orchestra.run.trial132.stdout:Total 250 kB/s | 124 kB 00:00 2026-02-26T03:30:42.779 INFO:teuthology.orchestra.run.trial132.stdout:Running transaction check 2026-02-26T03:30:42.782 INFO:teuthology.orchestra.run.trial132.stdout:Transaction check succeeded. 2026-02-26T03:30:42.782 INFO:teuthology.orchestra.run.trial132.stdout:Running transaction test 2026-02-26T03:30:42.813 INFO:teuthology.orchestra.run.trial132.stdout:Transaction test succeeded. 2026-02-26T03:30:42.813 INFO:teuthology.orchestra.run.trial132.stdout:Running transaction 2026-02-26T03:30:42.868 INFO:teuthology.orchestra.run.trial132.stdout: Preparing : 1/1 2026-02-26T03:30:42.885 INFO:teuthology.orchestra.run.trial132.stdout: Installing : librados-devel-2:20.3.0-5312.gabffb617.el9.x86_64 1/1 2026-02-26T03:30:42.972 INFO:teuthology.orchestra.run.trial132.stdout: Running scriptlet: librados-devel-2:20.3.0-5312.gabffb617.el9.x86_64 1/1 2026-02-26T03:30:43.034 INFO:teuthology.orchestra.run.trial132.stdout: Verifying : librados-devel-2:20.3.0-5312.gabffb617.el9.x86_64 1/1 2026-02-26T03:30:43.034 INFO:teuthology.orchestra.run.trial132.stdout: 2026-02-26T03:30:43.034 INFO:teuthology.orchestra.run.trial132.stdout:Installed: 2026-02-26T03:30:43.034 INFO:teuthology.orchestra.run.trial132.stdout: librados-devel-2:20.3.0-5312.gabffb617.el9.x86_64 2026-02-26T03:30:43.034 INFO:teuthology.orchestra.run.trial132.stdout: 2026-02-26T03:30:43.034 INFO:teuthology.orchestra.run.trial132.stdout:Complete! 2026-02-26T03:30:43.062 DEBUG:teuthology.orchestra.run.trial132:> sudo yum -y install libcephfs2 2026-02-26T03:30:43.383 INFO:teuthology.orchestra.run.trial132.stdout:Last metadata expiration check: 0:09:23 ago on Thu 26 Feb 2026 03:21:20 AM UTC. 2026-02-26T03:30:43.448 INFO:teuthology.orchestra.run.trial132.stdout:Package libcephfs2-2:20.3.0-5312.gabffb617.el9.x86_64 is already installed. 2026-02-26T03:30:43.468 INFO:teuthology.orchestra.run.trial132.stdout:Dependencies resolved. 2026-02-26T03:30:43.468 INFO:teuthology.orchestra.run.trial132.stdout:Nothing to do. 2026-02-26T03:30:43.468 INFO:teuthology.orchestra.run.trial132.stdout:Complete! 2026-02-26T03:30:43.489 DEBUG:teuthology.orchestra.run.trial132:> sudo yum -y install libcephfs-devel 2026-02-26T03:30:43.805 INFO:teuthology.orchestra.run.trial132.stdout:Last metadata expiration check: 0:09:23 ago on Thu 26 Feb 2026 03:21:20 AM UTC. 2026-02-26T03:30:43.889 INFO:teuthology.orchestra.run.trial132.stdout:Dependencies resolved. 2026-02-26T03:30:43.889 INFO:teuthology.orchestra.run.trial132.stdout:================================================================================ 2026-02-26T03:30:43.889 INFO:teuthology.orchestra.run.trial132.stdout: Package Arch Version Repo Size 2026-02-26T03:30:43.889 INFO:teuthology.orchestra.run.trial132.stdout:================================================================================ 2026-02-26T03:30:43.889 INFO:teuthology.orchestra.run.trial132.stdout:Installing: 2026-02-26T03:30:43.890 INFO:teuthology.orchestra.run.trial132.stdout: libcephfs-devel x86_64 2:20.3.0-5312.gabffb617.el9 ceph 37 k 2026-02-26T03:30:43.890 INFO:teuthology.orchestra.run.trial132.stdout: 2026-02-26T03:30:43.890 INFO:teuthology.orchestra.run.trial132.stdout:Transaction Summary 2026-02-26T03:30:43.890 INFO:teuthology.orchestra.run.trial132.stdout:================================================================================ 2026-02-26T03:30:43.890 INFO:teuthology.orchestra.run.trial132.stdout:Install 1 Package 2026-02-26T03:30:43.890 INFO:teuthology.orchestra.run.trial132.stdout: 2026-02-26T03:30:43.890 INFO:teuthology.orchestra.run.trial132.stdout:Total download size: 37 k 2026-02-26T03:30:43.890 INFO:teuthology.orchestra.run.trial132.stdout:Installed size: 171 k 2026-02-26T03:30:43.890 INFO:teuthology.orchestra.run.trial132.stdout:Downloading Packages: 2026-02-26T03:30:44.243 INFO:teuthology.orchestra.run.trial132.stdout:libcephfs-devel-20.3.0-5312.gabffb617.el9.x86_6 106 kB/s | 37 kB 00:00 2026-02-26T03:30:44.243 INFO:teuthology.orchestra.run.trial132.stdout:-------------------------------------------------------------------------------- 2026-02-26T03:30:44.243 INFO:teuthology.orchestra.run.trial132.stdout:Total 106 kB/s | 37 kB 00:00 2026-02-26T03:30:44.243 INFO:teuthology.orchestra.run.trial132.stdout:Running transaction check 2026-02-26T03:30:44.245 INFO:teuthology.orchestra.run.trial132.stdout:Transaction check succeeded. 2026-02-26T03:30:44.245 INFO:teuthology.orchestra.run.trial132.stdout:Running transaction test 2026-02-26T03:30:44.248 INFO:teuthology.orchestra.run.trial132.stdout:Transaction test succeeded. 2026-02-26T03:30:44.248 INFO:teuthology.orchestra.run.trial132.stdout:Running transaction 2026-02-26T03:30:44.269 INFO:teuthology.orchestra.run.trial132.stdout: Preparing : 1/1 2026-02-26T03:30:44.287 INFO:teuthology.orchestra.run.trial132.stdout: Installing : libcephfs-devel-2:20.3.0-5312.gabffb617.el9.x86_64 1/1 2026-02-26T03:30:44.350 INFO:teuthology.orchestra.run.trial132.stdout: Running scriptlet: libcephfs-devel-2:20.3.0-5312.gabffb617.el9.x86_64 1/1 2026-02-26T03:30:44.413 INFO:teuthology.orchestra.run.trial132.stdout: Verifying : libcephfs-devel-2:20.3.0-5312.gabffb617.el9.x86_64 1/1 2026-02-26T03:30:44.413 INFO:teuthology.orchestra.run.trial132.stdout: 2026-02-26T03:30:44.414 INFO:teuthology.orchestra.run.trial132.stdout:Installed: 2026-02-26T03:30:44.414 INFO:teuthology.orchestra.run.trial132.stdout: libcephfs-devel-2:20.3.0-5312.gabffb617.el9.x86_64 2026-02-26T03:30:44.414 INFO:teuthology.orchestra.run.trial132.stdout: 2026-02-26T03:30:44.414 INFO:teuthology.orchestra.run.trial132.stdout:Complete! 2026-02-26T03:30:44.442 DEBUG:teuthology.orchestra.run.trial132:> sudo yum -y install librados2 2026-02-26T03:30:44.766 INFO:teuthology.orchestra.run.trial132.stdout:Last metadata expiration check: 0:09:24 ago on Thu 26 Feb 2026 03:21:20 AM UTC. 2026-02-26T03:30:44.832 INFO:teuthology.orchestra.run.trial132.stdout:Package librados2-2:20.3.0-5312.gabffb617.el9.x86_64 is already installed. 2026-02-26T03:30:44.849 INFO:teuthology.orchestra.run.trial132.stdout:Dependencies resolved. 2026-02-26T03:30:44.850 INFO:teuthology.orchestra.run.trial132.stdout:Nothing to do. 2026-02-26T03:30:44.850 INFO:teuthology.orchestra.run.trial132.stdout:Complete! 2026-02-26T03:30:44.872 DEBUG:teuthology.orchestra.run.trial132:> sudo yum -y install librbd1 2026-02-26T03:30:45.185 INFO:teuthology.orchestra.run.trial132.stdout:Last metadata expiration check: 0:09:25 ago on Thu 26 Feb 2026 03:21:20 AM UTC. 2026-02-26T03:30:45.250 INFO:teuthology.orchestra.run.trial132.stdout:Package librbd1-2:20.3.0-5312.gabffb617.el9.x86_64 is already installed. 2026-02-26T03:30:45.268 INFO:teuthology.orchestra.run.trial132.stdout:Dependencies resolved. 2026-02-26T03:30:45.268 INFO:teuthology.orchestra.run.trial132.stdout:Nothing to do. 2026-02-26T03:30:45.268 INFO:teuthology.orchestra.run.trial132.stdout:Complete! 2026-02-26T03:30:45.290 DEBUG:teuthology.orchestra.run.trial132:> sudo yum -y install python3-rados 2026-02-26T03:30:45.606 INFO:teuthology.orchestra.run.trial132.stdout:Last metadata expiration check: 0:09:25 ago on Thu 26 Feb 2026 03:21:20 AM UTC. 2026-02-26T03:30:45.671 INFO:teuthology.orchestra.run.trial132.stdout:Package python3-rados-2:20.3.0-5312.gabffb617.el9.x86_64 is already installed. 2026-02-26T03:30:45.689 INFO:teuthology.orchestra.run.trial132.stdout:Dependencies resolved. 2026-02-26T03:30:45.689 INFO:teuthology.orchestra.run.trial132.stdout:Nothing to do. 2026-02-26T03:30:45.689 INFO:teuthology.orchestra.run.trial132.stdout:Complete! 2026-02-26T03:30:45.710 DEBUG:teuthology.orchestra.run.trial132:> sudo yum -y install python3-rgw 2026-02-26T03:30:46.029 INFO:teuthology.orchestra.run.trial132.stdout:Last metadata expiration check: 0:09:26 ago on Thu 26 Feb 2026 03:21:20 AM UTC. 2026-02-26T03:30:46.094 INFO:teuthology.orchestra.run.trial132.stdout:Package python3-rgw-2:20.3.0-5312.gabffb617.el9.x86_64 is already installed. 2026-02-26T03:30:46.112 INFO:teuthology.orchestra.run.trial132.stdout:Dependencies resolved. 2026-02-26T03:30:46.113 INFO:teuthology.orchestra.run.trial132.stdout:Nothing to do. 2026-02-26T03:30:46.113 INFO:teuthology.orchestra.run.trial132.stdout:Complete! 2026-02-26T03:30:46.134 DEBUG:teuthology.orchestra.run.trial132:> sudo yum -y install python3-cephfs 2026-02-26T03:30:46.446 INFO:teuthology.orchestra.run.trial132.stdout:Last metadata expiration check: 0:09:26 ago on Thu 26 Feb 2026 03:21:20 AM UTC. 2026-02-26T03:30:46.511 INFO:teuthology.orchestra.run.trial132.stdout:Package python3-cephfs-2:20.3.0-5312.gabffb617.el9.x86_64 is already installed. 2026-02-26T03:30:46.529 INFO:teuthology.orchestra.run.trial132.stdout:Dependencies resolved. 2026-02-26T03:30:46.529 INFO:teuthology.orchestra.run.trial132.stdout:Nothing to do. 2026-02-26T03:30:46.530 INFO:teuthology.orchestra.run.trial132.stdout:Complete! 2026-02-26T03:30:46.551 DEBUG:teuthology.orchestra.run.trial132:> sudo yum -y install python3-rbd 2026-02-26T03:30:46.870 INFO:teuthology.orchestra.run.trial132.stdout:Last metadata expiration check: 0:09:26 ago on Thu 26 Feb 2026 03:21:20 AM UTC. 2026-02-26T03:30:46.935 INFO:teuthology.orchestra.run.trial132.stdout:Package python3-rbd-2:20.3.0-5312.gabffb617.el9.x86_64 is already installed. 2026-02-26T03:30:46.953 INFO:teuthology.orchestra.run.trial132.stdout:Dependencies resolved. 2026-02-26T03:30:46.954 INFO:teuthology.orchestra.run.trial132.stdout:Nothing to do. 2026-02-26T03:30:46.954 INFO:teuthology.orchestra.run.trial132.stdout:Complete! 2026-02-26T03:30:46.976 DEBUG:teuthology.orchestra.run.trial132:> sudo yum -y install rbd-fuse 2026-02-26T03:30:47.288 INFO:teuthology.orchestra.run.trial132.stdout:Last metadata expiration check: 0:09:27 ago on Thu 26 Feb 2026 03:21:20 AM UTC. 2026-02-26T03:30:47.371 INFO:teuthology.orchestra.run.trial132.stdout:Dependencies resolved. 2026-02-26T03:30:47.372 INFO:teuthology.orchestra.run.trial132.stdout:================================================================================ 2026-02-26T03:30:47.372 INFO:teuthology.orchestra.run.trial132.stdout: Package Architecture Version Repository Size 2026-02-26T03:30:47.372 INFO:teuthology.orchestra.run.trial132.stdout:================================================================================ 2026-02-26T03:30:47.373 INFO:teuthology.orchestra.run.trial132.stdout:Installing: 2026-02-26T03:30:47.373 INFO:teuthology.orchestra.run.trial132.stdout: rbd-fuse x86_64 2:20.3.0-5312.gabffb617.el9 ceph 87 k 2026-02-26T03:30:47.373 INFO:teuthology.orchestra.run.trial132.stdout: 2026-02-26T03:30:47.373 INFO:teuthology.orchestra.run.trial132.stdout:Transaction Summary 2026-02-26T03:30:47.373 INFO:teuthology.orchestra.run.trial132.stdout:================================================================================ 2026-02-26T03:30:47.373 INFO:teuthology.orchestra.run.trial132.stdout:Install 1 Package 2026-02-26T03:30:47.373 INFO:teuthology.orchestra.run.trial132.stdout: 2026-02-26T03:30:47.373 INFO:teuthology.orchestra.run.trial132.stdout:Total download size: 87 k 2026-02-26T03:30:47.373 INFO:teuthology.orchestra.run.trial132.stdout:Installed size: 236 k 2026-02-26T03:30:47.373 INFO:teuthology.orchestra.run.trial132.stdout:Downloading Packages: 2026-02-26T03:30:47.815 INFO:teuthology.orchestra.run.trial132.stdout:rbd-fuse-20.3.0-5312.gabffb617.el9.x86_64.rpm 196 kB/s | 87 kB 00:00 2026-02-26T03:30:47.816 INFO:teuthology.orchestra.run.trial132.stdout:-------------------------------------------------------------------------------- 2026-02-26T03:30:47.816 INFO:teuthology.orchestra.run.trial132.stdout:Total 195 kB/s | 87 kB 00:00 2026-02-26T03:30:47.816 INFO:teuthology.orchestra.run.trial132.stdout:Running transaction check 2026-02-26T03:30:47.820 INFO:teuthology.orchestra.run.trial132.stdout:Transaction check succeeded. 2026-02-26T03:30:47.820 INFO:teuthology.orchestra.run.trial132.stdout:Running transaction test 2026-02-26T03:30:47.851 INFO:teuthology.orchestra.run.trial132.stdout:Transaction test succeeded. 2026-02-26T03:30:47.851 INFO:teuthology.orchestra.run.trial132.stdout:Running transaction 2026-02-26T03:30:47.904 INFO:teuthology.orchestra.run.trial132.stdout: Preparing : 1/1 2026-02-26T03:30:47.923 INFO:teuthology.orchestra.run.trial132.stdout: Installing : rbd-fuse-2:20.3.0-5312.gabffb617.el9.x86_64 1/1 2026-02-26T03:30:48.011 INFO:teuthology.orchestra.run.trial132.stdout: Running scriptlet: rbd-fuse-2:20.3.0-5312.gabffb617.el9.x86_64 1/1 2026-02-26T03:30:48.073 INFO:teuthology.orchestra.run.trial132.stdout: Verifying : rbd-fuse-2:20.3.0-5312.gabffb617.el9.x86_64 1/1 2026-02-26T03:30:48.073 INFO:teuthology.orchestra.run.trial132.stdout: 2026-02-26T03:30:48.073 INFO:teuthology.orchestra.run.trial132.stdout:Installed: 2026-02-26T03:30:48.073 INFO:teuthology.orchestra.run.trial132.stdout: rbd-fuse-2:20.3.0-5312.gabffb617.el9.x86_64 2026-02-26T03:30:48.074 INFO:teuthology.orchestra.run.trial132.stdout: 2026-02-26T03:30:48.074 INFO:teuthology.orchestra.run.trial132.stdout:Complete! 2026-02-26T03:30:48.102 DEBUG:teuthology.orchestra.run.trial132:> sudo yum -y install rbd-mirror 2026-02-26T03:30:48.414 INFO:teuthology.orchestra.run.trial132.stdout:Last metadata expiration check: 0:09:28 ago on Thu 26 Feb 2026 03:21:20 AM UTC. 2026-02-26T03:30:48.498 INFO:teuthology.orchestra.run.trial132.stdout:Dependencies resolved. 2026-02-26T03:30:48.498 INFO:teuthology.orchestra.run.trial132.stdout:================================================================================ 2026-02-26T03:30:48.498 INFO:teuthology.orchestra.run.trial132.stdout: Package Arch Version Repo Size 2026-02-26T03:30:48.498 INFO:teuthology.orchestra.run.trial132.stdout:================================================================================ 2026-02-26T03:30:48.498 INFO:teuthology.orchestra.run.trial132.stdout:Installing: 2026-02-26T03:30:48.498 INFO:teuthology.orchestra.run.trial132.stdout: rbd-mirror x86_64 2:20.3.0-5312.gabffb617.el9 ceph 2.6 M 2026-02-26T03:30:48.498 INFO:teuthology.orchestra.run.trial132.stdout: 2026-02-26T03:30:48.498 INFO:teuthology.orchestra.run.trial132.stdout:Transaction Summary 2026-02-26T03:30:48.499 INFO:teuthology.orchestra.run.trial132.stdout:================================================================================ 2026-02-26T03:30:48.499 INFO:teuthology.orchestra.run.trial132.stdout:Install 1 Package 2026-02-26T03:30:48.499 INFO:teuthology.orchestra.run.trial132.stdout: 2026-02-26T03:30:48.499 INFO:teuthology.orchestra.run.trial132.stdout:Total download size: 2.6 M 2026-02-26T03:30:48.499 INFO:teuthology.orchestra.run.trial132.stdout:Installed size: 11 M 2026-02-26T03:30:48.499 INFO:teuthology.orchestra.run.trial132.stdout:Downloading Packages: 2026-02-26T03:30:50.404 INFO:teuthology.orchestra.run.trial132.stdout:rbd-mirror-20.3.0-5312.gabffb617.el9.x86_64.rpm 1.4 MB/s | 2.6 MB 00:01 2026-02-26T03:30:50.404 INFO:teuthology.orchestra.run.trial132.stdout:-------------------------------------------------------------------------------- 2026-02-26T03:30:50.404 INFO:teuthology.orchestra.run.trial132.stdout:Total 1.4 MB/s | 2.6 MB 00:01 2026-02-26T03:30:50.404 INFO:teuthology.orchestra.run.trial132.stdout:Running transaction check 2026-02-26T03:30:50.409 INFO:teuthology.orchestra.run.trial132.stdout:Transaction check succeeded. 2026-02-26T03:30:50.409 INFO:teuthology.orchestra.run.trial132.stdout:Running transaction test 2026-02-26T03:30:50.447 INFO:teuthology.orchestra.run.trial132.stdout:Transaction test succeeded. 2026-02-26T03:30:50.448 INFO:teuthology.orchestra.run.trial132.stdout:Running transaction 2026-02-26T03:30:50.613 INFO:teuthology.orchestra.run.trial132.stdout: Preparing : 1/1 2026-02-26T03:30:50.615 INFO:teuthology.orchestra.run.trial132.stdout: Installing : rbd-mirror-2:20.3.0-5312.gabffb617.el9.x86_64 1/1 2026-02-26T03:30:50.638 INFO:teuthology.orchestra.run.trial132.stdout: Running scriptlet: rbd-mirror-2:20.3.0-5312.gabffb617.el9.x86_64 1/1 2026-02-26T03:30:50.638 INFO:teuthology.orchestra.run.trial132.stdout:Glob pattern passed to enable, but globs are not supported for this. 2026-02-26T03:30:50.638 INFO:teuthology.orchestra.run.trial132.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2026-02-26T03:30:50.638 INFO:teuthology.orchestra.run.trial132.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2026-02-26T03:30:50.638 INFO:teuthology.orchestra.run.trial132.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2026-02-26T03:30:50.638 INFO:teuthology.orchestra.run.trial132.stdout: 2026-02-26T03:30:50.980 INFO:teuthology.orchestra.run.trial132.stdout: Verifying : rbd-mirror-2:20.3.0-5312.gabffb617.el9.x86_64 1/1 2026-02-26T03:30:50.980 INFO:teuthology.orchestra.run.trial132.stdout: 2026-02-26T03:30:50.980 INFO:teuthology.orchestra.run.trial132.stdout:Installed: 2026-02-26T03:30:50.980 INFO:teuthology.orchestra.run.trial132.stdout: rbd-mirror-2:20.3.0-5312.gabffb617.el9.x86_64 2026-02-26T03:30:50.980 INFO:teuthology.orchestra.run.trial132.stdout: 2026-02-26T03:30:50.980 INFO:teuthology.orchestra.run.trial132.stdout:Complete! 2026-02-26T03:30:51.009 DEBUG:teuthology.orchestra.run.trial132:> sudo yum -y install rbd-nbd 2026-02-26T03:30:51.331 INFO:teuthology.orchestra.run.trial132.stdout:Last metadata expiration check: 0:09:31 ago on Thu 26 Feb 2026 03:21:20 AM UTC. 2026-02-26T03:30:51.414 INFO:teuthology.orchestra.run.trial132.stdout:Dependencies resolved. 2026-02-26T03:30:51.414 INFO:teuthology.orchestra.run.trial132.stdout:================================================================================ 2026-02-26T03:30:51.414 INFO:teuthology.orchestra.run.trial132.stdout: Package Architecture Version Repository Size 2026-02-26T03:30:51.414 INFO:teuthology.orchestra.run.trial132.stdout:================================================================================ 2026-02-26T03:30:51.414 INFO:teuthology.orchestra.run.trial132.stdout:Installing: 2026-02-26T03:30:51.414 INFO:teuthology.orchestra.run.trial132.stdout: rbd-nbd x86_64 2:20.3.0-5312.gabffb617.el9 ceph 166 k 2026-02-26T03:30:51.414 INFO:teuthology.orchestra.run.trial132.stdout: 2026-02-26T03:30:51.414 INFO:teuthology.orchestra.run.trial132.stdout:Transaction Summary 2026-02-26T03:30:51.414 INFO:teuthology.orchestra.run.trial132.stdout:================================================================================ 2026-02-26T03:30:51.414 INFO:teuthology.orchestra.run.trial132.stdout:Install 1 Package 2026-02-26T03:30:51.414 INFO:teuthology.orchestra.run.trial132.stdout: 2026-02-26T03:30:51.415 INFO:teuthology.orchestra.run.trial132.stdout:Total download size: 166 k 2026-02-26T03:30:51.415 INFO:teuthology.orchestra.run.trial132.stdout:Installed size: 496 k 2026-02-26T03:30:51.415 INFO:teuthology.orchestra.run.trial132.stdout:Downloading Packages: 2026-02-26T03:30:52.621 INFO:teuthology.orchestra.run.trial132.stdout:rbd-nbd-20.3.0-5312.gabffb617.el9.x86_64.rpm 138 kB/s | 166 kB 00:01 2026-02-26T03:30:52.621 INFO:teuthology.orchestra.run.trial132.stdout:-------------------------------------------------------------------------------- 2026-02-26T03:30:52.621 INFO:teuthology.orchestra.run.trial132.stdout:Total 138 kB/s | 166 kB 00:01 2026-02-26T03:30:52.621 INFO:teuthology.orchestra.run.trial132.stdout:Running transaction check 2026-02-26T03:30:52.626 INFO:teuthology.orchestra.run.trial132.stdout:Transaction check succeeded. 2026-02-26T03:30:52.626 INFO:teuthology.orchestra.run.trial132.stdout:Running transaction test 2026-02-26T03:30:52.658 INFO:teuthology.orchestra.run.trial132.stdout:Transaction test succeeded. 2026-02-26T03:30:52.658 INFO:teuthology.orchestra.run.trial132.stdout:Running transaction 2026-02-26T03:30:52.715 INFO:teuthology.orchestra.run.trial132.stdout: Preparing : 1/1 2026-02-26T03:30:52.732 INFO:teuthology.orchestra.run.trial132.stdout: Installing : rbd-nbd-2:20.3.0-5312.gabffb617.el9.x86_64 1/1 2026-02-26T03:30:52.818 INFO:teuthology.orchestra.run.trial132.stdout: Running scriptlet: rbd-nbd-2:20.3.0-5312.gabffb617.el9.x86_64 1/1 2026-02-26T03:30:52.881 INFO:teuthology.orchestra.run.trial132.stdout: Verifying : rbd-nbd-2:20.3.0-5312.gabffb617.el9.x86_64 1/1 2026-02-26T03:30:52.881 INFO:teuthology.orchestra.run.trial132.stdout: 2026-02-26T03:30:52.881 INFO:teuthology.orchestra.run.trial132.stdout:Installed: 2026-02-26T03:30:52.881 INFO:teuthology.orchestra.run.trial132.stdout: rbd-nbd-2:20.3.0-5312.gabffb617.el9.x86_64 2026-02-26T03:30:52.881 INFO:teuthology.orchestra.run.trial132.stdout: 2026-02-26T03:30:52.881 INFO:teuthology.orchestra.run.trial132.stdout:Complete! 2026-02-26T03:30:52.910 DEBUG:teuthology.orchestra.run.trial132:> sudo yum -y install ceph-osd-crimson 2026-02-26T03:30:53.227 INFO:teuthology.orchestra.run.trial132.stdout:Last metadata expiration check: 0:09:33 ago on Thu 26 Feb 2026 03:21:20 AM UTC. 2026-02-26T03:30:53.292 INFO:teuthology.orchestra.run.trial132.stdout:Package ceph-osd-crimson-2:20.3.0-5312.gabffb617.el9.x86_64 is already installed. 2026-02-26T03:30:53.310 INFO:teuthology.orchestra.run.trial132.stdout:Dependencies resolved. 2026-02-26T03:30:53.310 INFO:teuthology.orchestra.run.trial132.stdout:Nothing to do. 2026-02-26T03:30:53.310 INFO:teuthology.orchestra.run.trial132.stdout:Complete! 2026-02-26T03:30:53.332 DEBUG:teuthology.orchestra.run.trial132:> sudo yum -y install ceph-osd-crimson-debuginfo 2026-02-26T03:30:53.648 INFO:teuthology.orchestra.run.trial132.stdout:Last metadata expiration check: 0:09:33 ago on Thu 26 Feb 2026 03:21:20 AM UTC. 2026-02-26T03:30:53.731 INFO:teuthology.orchestra.run.trial132.stdout:Dependencies resolved. 2026-02-26T03:30:53.731 INFO:teuthology.orchestra.run.trial132.stdout:================================================================================ 2026-02-26T03:30:53.731 INFO:teuthology.orchestra.run.trial132.stdout: Package Arch Version Repo Size 2026-02-26T03:30:53.731 INFO:teuthology.orchestra.run.trial132.stdout:================================================================================ 2026-02-26T03:30:53.732 INFO:teuthology.orchestra.run.trial132.stdout:Installing: 2026-02-26T03:30:53.732 INFO:teuthology.orchestra.run.trial132.stdout: ceph-osd-crimson-debuginfo x86_64 2:20.3.0-5312.gabffb617.el9 ceph 429 M 2026-02-26T03:30:53.732 INFO:teuthology.orchestra.run.trial132.stdout:Installing weak dependencies: 2026-02-26T03:30:53.732 INFO:teuthology.orchestra.run.trial132.stdout: ceph-debugsource x86_64 2:20.3.0-5312.gabffb617.el9 ceph 12 M 2026-02-26T03:30:53.732 INFO:teuthology.orchestra.run.trial132.stdout: 2026-02-26T03:30:53.732 INFO:teuthology.orchestra.run.trial132.stdout:Transaction Summary 2026-02-26T03:30:53.732 INFO:teuthology.orchestra.run.trial132.stdout:================================================================================ 2026-02-26T03:30:53.732 INFO:teuthology.orchestra.run.trial132.stdout:Install 2 Packages 2026-02-26T03:30:53.732 INFO:teuthology.orchestra.run.trial132.stdout: 2026-02-26T03:30:53.732 INFO:teuthology.orchestra.run.trial132.stdout:Total download size: 441 M 2026-02-26T03:30:53.732 INFO:teuthology.orchestra.run.trial132.stdout:Installed size: 2.7 G 2026-02-26T03:30:53.732 INFO:teuthology.orchestra.run.trial132.stdout:Downloading Packages: 2026-02-26T03:31:06.596 INFO:teuthology.orchestra.run.trial132.stdout:(1/2): ceph-debugsource-20.3.0-5312.gabffb617.e 958 kB/s | 12 MB 00:12 2026-02-26T03:42:41.664 INFO:teuthology.orchestra.run.trial132.stdout:(2/2): ceph-osd-crimson-debuginfo-20.3.0-5312.g 620 kB/s | 429 MB 11:47 2026-02-26T03:42:41.681 INFO:teuthology.orchestra.run.trial132.stdout:-------------------------------------------------------------------------------- 2026-02-26T03:42:41.682 INFO:teuthology.orchestra.run.trial132.stdout:Total 638 kB/s | 441 MB 11:47 2026-02-26T03:42:41.682 INFO:teuthology.orchestra.run.trial132.stdout:Running transaction check 2026-02-26T03:42:41.682 INFO:teuthology.orchestra.run.trial132.stdout:Transaction check succeeded. 2026-02-26T03:42:41.682 INFO:teuthology.orchestra.run.trial132.stdout:Running transaction test 2026-02-26T03:42:43.038 INFO:teuthology.orchestra.run.trial132.stdout:Transaction test succeeded. 2026-02-26T03:42:43.038 INFO:teuthology.orchestra.run.trial132.stdout:Running transaction 2026-02-26T03:42:46.632 INFO:teuthology.orchestra.run.trial132.stdout: Preparing : 1/1 2026-02-26T03:43:08.670 INFO:teuthology.orchestra.run.trial132.stdout: Installing : ceph-debugsource-2:20.3.0-5312.gabffb617.el9.x86_64 1/2 2026-02-26T03:43:08.688 INFO:teuthology.orchestra.run.trial132.stdout: Installing : ceph-osd-crimson-debuginfo-2:20.3.0-5312.gabffb617.e 2/2 2026-02-26T03:43:09.018 INFO:teuthology.orchestra.run.trial132.stdout: Running scriptlet: ceph-osd-crimson-debuginfo-2:20.3.0-5312.gabffb617.e 2/2 2026-02-26T03:43:09.018 INFO:teuthology.orchestra.run.trial132.stdout: Verifying : ceph-debugsource-2:20.3.0-5312.gabffb617.el9.x86_64 1/2 2026-02-26T03:43:09.082 INFO:teuthology.orchestra.run.trial132.stdout: Verifying : ceph-osd-crimson-debuginfo-2:20.3.0-5312.gabffb617.e 2/2 2026-02-26T03:43:09.083 INFO:teuthology.orchestra.run.trial132.stdout: 2026-02-26T03:43:09.083 INFO:teuthology.orchestra.run.trial132.stdout:Installed: 2026-02-26T03:43:09.083 INFO:teuthology.orchestra.run.trial132.stdout: ceph-debugsource-2:20.3.0-5312.gabffb617.el9.x86_64 2026-02-26T03:43:09.083 INFO:teuthology.orchestra.run.trial132.stdout: ceph-osd-crimson-debuginfo-2:20.3.0-5312.gabffb617.el9.x86_64 2026-02-26T03:43:09.083 INFO:teuthology.orchestra.run.trial132.stdout: 2026-02-26T03:43:09.083 INFO:teuthology.orchestra.run.trial132.stdout:Complete! 2026-02-26T03:43:09.157 DEBUG:teuthology.parallel:result is None 2026-02-26T03:43:09.158 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=debug&distros=centos%2F9%2Fx86_64&sha1=abffb617683951511fccf3c3cab6e219f2ec83a8 2026-02-26T03:43:09.327 DEBUG:teuthology.orchestra.run.trial132:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2026-02-26T03:43:09.349 INFO:teuthology.orchestra.run.trial132.stdout:20.3.0-5312.gabffb617.el9 2026-02-26T03:43:09.350 INFO:teuthology.packaging:The installed version of ceph is 20.3.0-5312.gabffb617.el9 2026-02-26T03:43:09.350 INFO:teuthology.task.install:The correct ceph version 20.3.0-5312.gabffb617 is installed. 2026-02-26T03:43:09.352 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2026-02-26T03:43:09.353 DEBUG:teuthology.orchestra.run.trial132:> set -ex 2026-02-26T03:43:09.353 DEBUG:teuthology.orchestra.run.trial132:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2026-02-26T03:43:09.415 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2026-02-26T03:43:09.416 DEBUG:teuthology.orchestra.run.trial132:> set -ex 2026-02-26T03:43:09.416 DEBUG:teuthology.orchestra.run.trial132:> sudo dd of=/usr/bin/daemon-helper 2026-02-26T03:43:09.479 DEBUG:teuthology.orchestra.run.trial132:> sudo chmod a=rx -- /usr/bin/daemon-helper 2026-02-26T03:43:09.543 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2026-02-26T03:43:09.543 DEBUG:teuthology.orchestra.run.trial132:> set -ex 2026-02-26T03:43:09.543 DEBUG:teuthology.orchestra.run.trial132:> sudo dd of=/usr/bin/adjust-ulimits 2026-02-26T03:43:09.608 DEBUG:teuthology.orchestra.run.trial132:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2026-02-26T03:43:09.671 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2026-02-26T03:43:09.672 DEBUG:teuthology.orchestra.run.trial132:> set -ex 2026-02-26T03:43:09.672 DEBUG:teuthology.orchestra.run.trial132:> sudo dd of=/usr/bin/stdin-killer 2026-02-26T03:43:09.735 DEBUG:teuthology.orchestra.run.trial132:> sudo chmod a=rx -- /usr/bin/stdin-killer 2026-02-26T03:43:09.799 INFO:teuthology.run_tasks:Running task pexec... 2026-02-26T03:43:09.816 INFO:teuthology.task.pexec:Executing custom commands... 2026-02-26T03:43:09.817 DEBUG:teuthology.orchestra.run.trial132:> TESTDIR=/home/ubuntu/cephtest bash -s 2026-02-26T03:43:09.842 INFO:teuthology.task.pexec:Running commands on host ubuntu@trial132.front.sepia.ceph.com 2026-02-26T03:43:09.842 INFO:teuthology.task.pexec:sudo update-alternatives --set ceph-osd /usr/bin/ceph-osd-crimson 2026-02-26T03:43:09.892 DEBUG:teuthology.parallel:result is None 2026-02-26T03:43:09.892 INFO:teuthology.run_tasks:Running task install... 2026-02-26T03:43:09.906 DEBUG:teuthology.task.install:project ceph 2026-02-26T03:43:09.906 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'debug', 'sha1': 'abffb617683951511fccf3c3cab6e219f2ec83a8'}} 2026-02-26T03:43:09.906 DEBUG:teuthology.task.install:config {'flavor': 'debug', 'sha1': 'abffb617683951511fccf3c3cab6e219f2ec83a8'} 2026-02-26T03:43:09.906 INFO:teuthology.task.install:Using flavor: debug 2026-02-26T03:43:09.915 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-osd', 'ceph-osd-classic', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-osd', 'ceph-osd-classic', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2026-02-26T03:43:09.915 INFO:teuthology.task.install:extra packages: [] 2026-02-26T03:43:09.915 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'debug', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': 'abffb617683951511fccf3c3cab6e219f2ec83a8', 'tag': None, 'wait_for_package': False} 2026-02-26T03:43:09.916 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=debug&distros=centos%2F9%2Fx86_64&sha1=abffb617683951511fccf3c3cab6e219f2ec83a8 2026-02-26T03:43:10.076 INFO:teuthology.task.install.rpm:Pulling from https://1.chacra.ceph.com/r/ceph/wip-shraddhaag-peering-bug/abffb617683951511fccf3c3cab6e219f2ec83a8/centos/9/flavors/debug/ 2026-02-26T03:43:10.076 INFO:teuthology.task.install.rpm:Package version is 20.3.0-5312.gabffb617 2026-02-26T03:43:10.432 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://1.chacra.ceph.com/r/ceph/wip-shraddhaag-peering-bug/abffb617683951511fccf3c3cab6e219f2ec83a8/centos/9/flavors/debug/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://1.chacra.ceph.com/r/ceph/wip-shraddhaag-peering-bug/abffb617683951511fccf3c3cab6e219f2ec83a8/centos/9/flavors/debug/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://1.chacra.ceph.com/r/ceph/wip-shraddhaag-peering-bug/abffb617683951511fccf3c3cab6e219f2ec83a8/centos/9/flavors/debug/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2026-02-26T03:43:10.432 DEBUG:teuthology.orchestra.run.trial132:> set -ex 2026-02-26T03:43:10.432 DEBUG:teuthology.orchestra.run.trial132:> sudo dd of=/etc/yum.repos.d/ceph.repo 2026-02-26T03:43:10.458 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-osd, ceph-osd-classic, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2026-02-26T03:43:10.458 DEBUG:teuthology.orchestra.run.trial132:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' /etc/yum.repos.d/ceph.repo ; fi 2026-02-26T03:43:10.524 DEBUG:teuthology.orchestra.run.trial132:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2026-02-26T03:43:10.588 DEBUG:teuthology.orchestra.run.trial132:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2026-02-26T03:43:10.644 INFO:teuthology.orchestra.run.trial132.stdout:check_obsoletes = 1 2026-02-26T03:43:10.653 DEBUG:teuthology.orchestra.run.trial132:> sudo yum clean all 2026-02-26T03:43:10.813 INFO:teuthology.orchestra.run.trial132.stdout:56 files removed 2026-02-26T03:43:10.828 DEBUG:teuthology.orchestra.run.trial132:> sudo yum -y install ceph-radosgw 2026-02-26T03:43:11.763 INFO:teuthology.orchestra.run.trial132.stdout:ceph packages for x86_64 120 kB/s | 94 kB 00:00 2026-02-26T03:43:12.420 INFO:teuthology.orchestra.run.trial132.stdout:ceph noarch packages 39 kB/s | 24 kB 00:00 2026-02-26T03:43:13.019 INFO:teuthology.orchestra.run.trial132.stdout:ceph source packages 3.6 kB/s | 2.1 kB 00:00 2026-02-26T03:43:14.385 INFO:teuthology.orchestra.run.trial132.stdout:CentOS Stream 9 - BaseOS 6.6 MB/s | 8.9 MB 00:01 2026-02-26T03:43:17.686 INFO:teuthology.orchestra.run.trial132.stdout:CentOS Stream 9 - AppStream 9.7 MB/s | 27 MB 00:02 2026-02-26T03:43:22.161 INFO:teuthology.orchestra.run.trial132.stdout:CentOS Stream 9 - CRB 3.8 MB/s | 8.0 MB 00:02 2026-02-26T03:43:24.052 INFO:teuthology.orchestra.run.trial132.stdout:CentOS Stream 9 - Extras packages 18 kB/s | 20 kB 00:01 2026-02-26T03:43:25.453 INFO:teuthology.orchestra.run.trial132.stdout:Extra Packages for Enterprise Linux 15 MB/s | 20 MB 00:01 2026-02-26T03:43:28.630 INFO:teuthology.orchestra.run.trial132.stdout:lab-extras 1.8 MB/s | 50 kB 00:00 2026-02-26T03:43:29.834 INFO:teuthology.orchestra.run.trial132.stdout:Package ceph-radosgw-2:20.3.0-5312.gabffb617.el9.x86_64 is already installed. 2026-02-26T03:43:29.852 INFO:teuthology.orchestra.run.trial132.stdout:Dependencies resolved. 2026-02-26T03:43:29.853 INFO:teuthology.orchestra.run.trial132.stdout:Nothing to do. 2026-02-26T03:43:29.853 INFO:teuthology.orchestra.run.trial132.stdout:Complete! 2026-02-26T03:43:29.877 DEBUG:teuthology.orchestra.run.trial132:> sudo yum -y install ceph-test 2026-02-26T03:43:30.196 INFO:teuthology.orchestra.run.trial132.stdout:Last metadata expiration check: 0:00:02 ago on Thu 26 Feb 2026 03:43:28 AM UTC. 2026-02-26T03:43:30.260 INFO:teuthology.orchestra.run.trial132.stdout:Package ceph-test-2:20.3.0-5312.gabffb617.el9.x86_64 is already installed. 2026-02-26T03:43:30.278 INFO:teuthology.orchestra.run.trial132.stdout:Dependencies resolved. 2026-02-26T03:43:30.278 INFO:teuthology.orchestra.run.trial132.stdout:Nothing to do. 2026-02-26T03:43:30.279 INFO:teuthology.orchestra.run.trial132.stdout:Complete! 2026-02-26T03:43:30.299 DEBUG:teuthology.orchestra.run.trial132:> sudo yum -y install ceph 2026-02-26T03:43:30.617 INFO:teuthology.orchestra.run.trial132.stdout:Last metadata expiration check: 0:00:02 ago on Thu 26 Feb 2026 03:43:28 AM UTC. 2026-02-26T03:43:30.682 INFO:teuthology.orchestra.run.trial132.stdout:Package ceph-2:20.3.0-5312.gabffb617.el9.x86_64 is already installed. 2026-02-26T03:43:30.700 INFO:teuthology.orchestra.run.trial132.stdout:Dependencies resolved. 2026-02-26T03:43:30.700 INFO:teuthology.orchestra.run.trial132.stdout:Nothing to do. 2026-02-26T03:43:30.701 INFO:teuthology.orchestra.run.trial132.stdout:Complete! 2026-02-26T03:43:30.722 DEBUG:teuthology.orchestra.run.trial132:> sudo yum -y install ceph-base 2026-02-26T03:43:31.042 INFO:teuthology.orchestra.run.trial132.stdout:Last metadata expiration check: 0:00:03 ago on Thu 26 Feb 2026 03:43:28 AM UTC. 2026-02-26T03:43:31.107 INFO:teuthology.orchestra.run.trial132.stdout:Package ceph-base-2:20.3.0-5312.gabffb617.el9.x86_64 is already installed. 2026-02-26T03:43:31.124 INFO:teuthology.orchestra.run.trial132.stdout:Dependencies resolved. 2026-02-26T03:43:31.125 INFO:teuthology.orchestra.run.trial132.stdout:Nothing to do. 2026-02-26T03:43:31.125 INFO:teuthology.orchestra.run.trial132.stdout:Complete! 2026-02-26T03:43:31.146 DEBUG:teuthology.orchestra.run.trial132:> sudo yum -y install cephadm 2026-02-26T03:43:31.464 INFO:teuthology.orchestra.run.trial132.stdout:Last metadata expiration check: 0:00:03 ago on Thu 26 Feb 2026 03:43:28 AM UTC. 2026-02-26T03:43:31.529 INFO:teuthology.orchestra.run.trial132.stdout:Package cephadm-2:20.3.0-5312.gabffb617.el9.noarch is already installed. 2026-02-26T03:43:31.547 INFO:teuthology.orchestra.run.trial132.stdout:Dependencies resolved. 2026-02-26T03:43:31.548 INFO:teuthology.orchestra.run.trial132.stdout:Nothing to do. 2026-02-26T03:43:31.548 INFO:teuthology.orchestra.run.trial132.stdout:Complete! 2026-02-26T03:43:31.569 DEBUG:teuthology.orchestra.run.trial132:> sudo yum -y install ceph-immutable-object-cache 2026-02-26T03:43:31.889 INFO:teuthology.orchestra.run.trial132.stdout:Last metadata expiration check: 0:00:03 ago on Thu 26 Feb 2026 03:43:28 AM UTC. 2026-02-26T03:43:31.955 INFO:teuthology.orchestra.run.trial132.stdout:Package ceph-immutable-object-cache-2:20.3.0-5312.gabffb617.el9.x86_64 is already installed. 2026-02-26T03:43:31.973 INFO:teuthology.orchestra.run.trial132.stdout:Dependencies resolved. 2026-02-26T03:43:31.974 INFO:teuthology.orchestra.run.trial132.stdout:Nothing to do. 2026-02-26T03:43:31.974 INFO:teuthology.orchestra.run.trial132.stdout:Complete! 2026-02-26T03:43:31.995 DEBUG:teuthology.orchestra.run.trial132:> sudo yum -y install ceph-mgr 2026-02-26T03:43:32.317 INFO:teuthology.orchestra.run.trial132.stdout:Last metadata expiration check: 0:00:04 ago on Thu 26 Feb 2026 03:43:28 AM UTC. 2026-02-26T03:43:32.381 INFO:teuthology.orchestra.run.trial132.stdout:Package ceph-mgr-2:20.3.0-5312.gabffb617.el9.x86_64 is already installed. 2026-02-26T03:43:32.398 INFO:teuthology.orchestra.run.trial132.stdout:Dependencies resolved. 2026-02-26T03:43:32.399 INFO:teuthology.orchestra.run.trial132.stdout:Nothing to do. 2026-02-26T03:43:32.399 INFO:teuthology.orchestra.run.trial132.stdout:Complete! 2026-02-26T03:43:32.421 DEBUG:teuthology.orchestra.run.trial132:> sudo yum -y install ceph-mgr-dashboard 2026-02-26T03:43:32.738 INFO:teuthology.orchestra.run.trial132.stdout:Last metadata expiration check: 0:00:04 ago on Thu 26 Feb 2026 03:43:28 AM UTC. 2026-02-26T03:43:32.803 INFO:teuthology.orchestra.run.trial132.stdout:Package ceph-mgr-dashboard-2:20.3.0-5312.gabffb617.el9.noarch is already installed. 2026-02-26T03:43:32.820 INFO:teuthology.orchestra.run.trial132.stdout:Dependencies resolved. 2026-02-26T03:43:32.821 INFO:teuthology.orchestra.run.trial132.stdout:Nothing to do. 2026-02-26T03:43:32.821 INFO:teuthology.orchestra.run.trial132.stdout:Complete! 2026-02-26T03:43:32.842 DEBUG:teuthology.orchestra.run.trial132:> sudo yum -y install ceph-mgr-diskprediction-local 2026-02-26T03:43:33.163 INFO:teuthology.orchestra.run.trial132.stdout:Last metadata expiration check: 0:00:05 ago on Thu 26 Feb 2026 03:43:28 AM UTC. 2026-02-26T03:43:33.228 INFO:teuthology.orchestra.run.trial132.stdout:Package ceph-mgr-diskprediction-local-2:20.3.0-5312.gabffb617.el9.noarch is already installed. 2026-02-26T03:43:33.246 INFO:teuthology.orchestra.run.trial132.stdout:Dependencies resolved. 2026-02-26T03:43:33.247 INFO:teuthology.orchestra.run.trial132.stdout:Nothing to do. 2026-02-26T03:43:33.247 INFO:teuthology.orchestra.run.trial132.stdout:Complete! 2026-02-26T03:43:33.269 DEBUG:teuthology.orchestra.run.trial132:> sudo yum -y install ceph-mgr-rook 2026-02-26T03:43:33.590 INFO:teuthology.orchestra.run.trial132.stdout:Last metadata expiration check: 0:00:05 ago on Thu 26 Feb 2026 03:43:28 AM UTC. 2026-02-26T03:43:33.655 INFO:teuthology.orchestra.run.trial132.stdout:Package ceph-mgr-rook-2:20.3.0-5312.gabffb617.el9.noarch is already installed. 2026-02-26T03:43:33.673 INFO:teuthology.orchestra.run.trial132.stdout:Dependencies resolved. 2026-02-26T03:43:33.673 INFO:teuthology.orchestra.run.trial132.stdout:Nothing to do. 2026-02-26T03:43:33.673 INFO:teuthology.orchestra.run.trial132.stdout:Complete! 2026-02-26T03:43:33.695 DEBUG:teuthology.orchestra.run.trial132:> sudo yum -y install ceph-mgr-cephadm 2026-02-26T03:43:34.018 INFO:teuthology.orchestra.run.trial132.stdout:Last metadata expiration check: 0:00:06 ago on Thu 26 Feb 2026 03:43:28 AM UTC. 2026-02-26T03:43:34.083 INFO:teuthology.orchestra.run.trial132.stdout:Package ceph-mgr-cephadm-2:20.3.0-5312.gabffb617.el9.noarch is already installed. 2026-02-26T03:43:34.101 INFO:teuthology.orchestra.run.trial132.stdout:Dependencies resolved. 2026-02-26T03:43:34.102 INFO:teuthology.orchestra.run.trial132.stdout:Nothing to do. 2026-02-26T03:43:34.102 INFO:teuthology.orchestra.run.trial132.stdout:Complete! 2026-02-26T03:43:34.123 DEBUG:teuthology.orchestra.run.trial132:> sudo yum -y install ceph-osd 2026-02-26T03:43:34.437 INFO:teuthology.orchestra.run.trial132.stdout:Last metadata expiration check: 0:00:06 ago on Thu 26 Feb 2026 03:43:28 AM UTC. 2026-02-26T03:43:34.501 INFO:teuthology.orchestra.run.trial132.stdout:Package ceph-osd-2:20.3.0-5312.gabffb617.el9.x86_64 is already installed. 2026-02-26T03:43:34.519 INFO:teuthology.orchestra.run.trial132.stdout:Dependencies resolved. 2026-02-26T03:43:34.520 INFO:teuthology.orchestra.run.trial132.stdout:Nothing to do. 2026-02-26T03:43:34.520 INFO:teuthology.orchestra.run.trial132.stdout:Complete! 2026-02-26T03:43:34.541 DEBUG:teuthology.orchestra.run.trial132:> sudo yum -y install ceph-osd-classic 2026-02-26T03:43:34.858 INFO:teuthology.orchestra.run.trial132.stdout:Last metadata expiration check: 0:00:06 ago on Thu 26 Feb 2026 03:43:28 AM UTC. 2026-02-26T03:43:34.923 INFO:teuthology.orchestra.run.trial132.stdout:Package ceph-osd-classic-2:20.3.0-5312.gabffb617.el9.x86_64 is already installed. 2026-02-26T03:43:34.940 INFO:teuthology.orchestra.run.trial132.stdout:Dependencies resolved. 2026-02-26T03:43:34.941 INFO:teuthology.orchestra.run.trial132.stdout:Nothing to do. 2026-02-26T03:43:34.941 INFO:teuthology.orchestra.run.trial132.stdout:Complete! 2026-02-26T03:43:34.963 DEBUG:teuthology.orchestra.run.trial132:> sudo yum -y install ceph-fuse 2026-02-26T03:43:35.283 INFO:teuthology.orchestra.run.trial132.stdout:Last metadata expiration check: 0:00:07 ago on Thu 26 Feb 2026 03:43:28 AM UTC. 2026-02-26T03:43:35.347 INFO:teuthology.orchestra.run.trial132.stdout:Package ceph-fuse-2:20.3.0-5312.gabffb617.el9.x86_64 is already installed. 2026-02-26T03:43:35.365 INFO:teuthology.orchestra.run.trial132.stdout:Dependencies resolved. 2026-02-26T03:43:35.366 INFO:teuthology.orchestra.run.trial132.stdout:Nothing to do. 2026-02-26T03:43:35.366 INFO:teuthology.orchestra.run.trial132.stdout:Complete! 2026-02-26T03:43:35.388 DEBUG:teuthology.orchestra.run.trial132:> sudo yum -y install ceph-volume 2026-02-26T03:43:35.709 INFO:teuthology.orchestra.run.trial132.stdout:Last metadata expiration check: 0:00:07 ago on Thu 26 Feb 2026 03:43:28 AM UTC. 2026-02-26T03:43:35.774 INFO:teuthology.orchestra.run.trial132.stdout:Package ceph-volume-2:20.3.0-5312.gabffb617.el9.noarch is already installed. 2026-02-26T03:43:35.792 INFO:teuthology.orchestra.run.trial132.stdout:Dependencies resolved. 2026-02-26T03:43:35.793 INFO:teuthology.orchestra.run.trial132.stdout:Nothing to do. 2026-02-26T03:43:35.793 INFO:teuthology.orchestra.run.trial132.stdout:Complete! 2026-02-26T03:43:35.814 DEBUG:teuthology.orchestra.run.trial132:> sudo yum -y install librados-devel 2026-02-26T03:43:36.130 INFO:teuthology.orchestra.run.trial132.stdout:Last metadata expiration check: 0:00:08 ago on Thu 26 Feb 2026 03:43:28 AM UTC. 2026-02-26T03:43:36.195 INFO:teuthology.orchestra.run.trial132.stdout:Package librados-devel-2:20.3.0-5312.gabffb617.el9.x86_64 is already installed. 2026-02-26T03:43:36.213 INFO:teuthology.orchestra.run.trial132.stdout:Dependencies resolved. 2026-02-26T03:43:36.214 INFO:teuthology.orchestra.run.trial132.stdout:Nothing to do. 2026-02-26T03:43:36.214 INFO:teuthology.orchestra.run.trial132.stdout:Complete! 2026-02-26T03:43:36.236 DEBUG:teuthology.orchestra.run.trial132:> sudo yum -y install libcephfs2 2026-02-26T03:43:36.558 INFO:teuthology.orchestra.run.trial132.stdout:Last metadata expiration check: 0:00:08 ago on Thu 26 Feb 2026 03:43:28 AM UTC. 2026-02-26T03:43:36.624 INFO:teuthology.orchestra.run.trial132.stdout:Package libcephfs2-2:20.3.0-5312.gabffb617.el9.x86_64 is already installed. 2026-02-26T03:43:36.641 INFO:teuthology.orchestra.run.trial132.stdout:Dependencies resolved. 2026-02-26T03:43:36.642 INFO:teuthology.orchestra.run.trial132.stdout:Nothing to do. 2026-02-26T03:43:36.642 INFO:teuthology.orchestra.run.trial132.stdout:Complete! 2026-02-26T03:43:36.663 DEBUG:teuthology.orchestra.run.trial132:> sudo yum -y install libcephfs-devel 2026-02-26T03:43:36.977 INFO:teuthology.orchestra.run.trial132.stdout:Last metadata expiration check: 0:00:08 ago on Thu 26 Feb 2026 03:43:28 AM UTC. 2026-02-26T03:43:37.042 INFO:teuthology.orchestra.run.trial132.stdout:Package libcephfs-devel-2:20.3.0-5312.gabffb617.el9.x86_64 is already installed. 2026-02-26T03:43:37.060 INFO:teuthology.orchestra.run.trial132.stdout:Dependencies resolved. 2026-02-26T03:43:37.060 INFO:teuthology.orchestra.run.trial132.stdout:Nothing to do. 2026-02-26T03:43:37.060 INFO:teuthology.orchestra.run.trial132.stdout:Complete! 2026-02-26T03:43:37.083 DEBUG:teuthology.orchestra.run.trial132:> sudo yum -y install librados2 2026-02-26T03:43:37.403 INFO:teuthology.orchestra.run.trial132.stdout:Last metadata expiration check: 0:00:09 ago on Thu 26 Feb 2026 03:43:28 AM UTC. 2026-02-26T03:43:37.467 INFO:teuthology.orchestra.run.trial132.stdout:Package librados2-2:20.3.0-5312.gabffb617.el9.x86_64 is already installed. 2026-02-26T03:43:37.485 INFO:teuthology.orchestra.run.trial132.stdout:Dependencies resolved. 2026-02-26T03:43:37.486 INFO:teuthology.orchestra.run.trial132.stdout:Nothing to do. 2026-02-26T03:43:37.486 INFO:teuthology.orchestra.run.trial132.stdout:Complete! 2026-02-26T03:43:37.508 DEBUG:teuthology.orchestra.run.trial132:> sudo yum -y install librbd1 2026-02-26T03:43:37.825 INFO:teuthology.orchestra.run.trial132.stdout:Last metadata expiration check: 0:00:09 ago on Thu 26 Feb 2026 03:43:28 AM UTC. 2026-02-26T03:43:37.889 INFO:teuthology.orchestra.run.trial132.stdout:Package librbd1-2:20.3.0-5312.gabffb617.el9.x86_64 is already installed. 2026-02-26T03:43:37.907 INFO:teuthology.orchestra.run.trial132.stdout:Dependencies resolved. 2026-02-26T03:43:37.908 INFO:teuthology.orchestra.run.trial132.stdout:Nothing to do. 2026-02-26T03:43:37.908 INFO:teuthology.orchestra.run.trial132.stdout:Complete! 2026-02-26T03:43:37.930 DEBUG:teuthology.orchestra.run.trial132:> sudo yum -y install python3-rados 2026-02-26T03:43:38.247 INFO:teuthology.orchestra.run.trial132.stdout:Last metadata expiration check: 0:00:10 ago on Thu 26 Feb 2026 03:43:28 AM UTC. 2026-02-26T03:43:38.312 INFO:teuthology.orchestra.run.trial132.stdout:Package python3-rados-2:20.3.0-5312.gabffb617.el9.x86_64 is already installed. 2026-02-26T03:43:38.330 INFO:teuthology.orchestra.run.trial132.stdout:Dependencies resolved. 2026-02-26T03:43:38.330 INFO:teuthology.orchestra.run.trial132.stdout:Nothing to do. 2026-02-26T03:43:38.331 INFO:teuthology.orchestra.run.trial132.stdout:Complete! 2026-02-26T03:43:38.353 DEBUG:teuthology.orchestra.run.trial132:> sudo yum -y install python3-rgw 2026-02-26T03:43:38.673 INFO:teuthology.orchestra.run.trial132.stdout:Last metadata expiration check: 0:00:10 ago on Thu 26 Feb 2026 03:43:28 AM UTC. 2026-02-26T03:43:38.739 INFO:teuthology.orchestra.run.trial132.stdout:Package python3-rgw-2:20.3.0-5312.gabffb617.el9.x86_64 is already installed. 2026-02-26T03:43:38.757 INFO:teuthology.orchestra.run.trial132.stdout:Dependencies resolved. 2026-02-26T03:43:38.758 INFO:teuthology.orchestra.run.trial132.stdout:Nothing to do. 2026-02-26T03:43:38.758 INFO:teuthology.orchestra.run.trial132.stdout:Complete! 2026-02-26T03:43:38.780 DEBUG:teuthology.orchestra.run.trial132:> sudo yum -y install python3-cephfs 2026-02-26T03:43:39.099 INFO:teuthology.orchestra.run.trial132.stdout:Last metadata expiration check: 0:00:11 ago on Thu 26 Feb 2026 03:43:28 AM UTC. 2026-02-26T03:43:39.164 INFO:teuthology.orchestra.run.trial132.stdout:Package python3-cephfs-2:20.3.0-5312.gabffb617.el9.x86_64 is already installed. 2026-02-26T03:43:39.181 INFO:teuthology.orchestra.run.trial132.stdout:Dependencies resolved. 2026-02-26T03:43:39.182 INFO:teuthology.orchestra.run.trial132.stdout:Nothing to do. 2026-02-26T03:43:39.182 INFO:teuthology.orchestra.run.trial132.stdout:Complete! 2026-02-26T03:43:39.204 DEBUG:teuthology.orchestra.run.trial132:> sudo yum -y install python3-rbd 2026-02-26T03:43:39.522 INFO:teuthology.orchestra.run.trial132.stdout:Last metadata expiration check: 0:00:11 ago on Thu 26 Feb 2026 03:43:28 AM UTC. 2026-02-26T03:43:39.587 INFO:teuthology.orchestra.run.trial132.stdout:Package python3-rbd-2:20.3.0-5312.gabffb617.el9.x86_64 is already installed. 2026-02-26T03:43:39.605 INFO:teuthology.orchestra.run.trial132.stdout:Dependencies resolved. 2026-02-26T03:43:39.605 INFO:teuthology.orchestra.run.trial132.stdout:Nothing to do. 2026-02-26T03:43:39.605 INFO:teuthology.orchestra.run.trial132.stdout:Complete! 2026-02-26T03:43:39.627 DEBUG:teuthology.orchestra.run.trial132:> sudo yum -y install rbd-fuse 2026-02-26T03:43:39.943 INFO:teuthology.orchestra.run.trial132.stdout:Last metadata expiration check: 0:00:11 ago on Thu 26 Feb 2026 03:43:28 AM UTC. 2026-02-26T03:43:40.007 INFO:teuthology.orchestra.run.trial132.stdout:Package rbd-fuse-2:20.3.0-5312.gabffb617.el9.x86_64 is already installed. 2026-02-26T03:43:40.025 INFO:teuthology.orchestra.run.trial132.stdout:Dependencies resolved. 2026-02-26T03:43:40.025 INFO:teuthology.orchestra.run.trial132.stdout:Nothing to do. 2026-02-26T03:43:40.025 INFO:teuthology.orchestra.run.trial132.stdout:Complete! 2026-02-26T03:43:40.047 DEBUG:teuthology.orchestra.run.trial132:> sudo yum -y install rbd-mirror 2026-02-26T03:43:40.363 INFO:teuthology.orchestra.run.trial132.stdout:Last metadata expiration check: 0:00:12 ago on Thu 26 Feb 2026 03:43:28 AM UTC. 2026-02-26T03:43:40.428 INFO:teuthology.orchestra.run.trial132.stdout:Package rbd-mirror-2:20.3.0-5312.gabffb617.el9.x86_64 is already installed. 2026-02-26T03:43:40.446 INFO:teuthology.orchestra.run.trial132.stdout:Dependencies resolved. 2026-02-26T03:43:40.447 INFO:teuthology.orchestra.run.trial132.stdout:Nothing to do. 2026-02-26T03:43:40.447 INFO:teuthology.orchestra.run.trial132.stdout:Complete! 2026-02-26T03:43:40.469 DEBUG:teuthology.orchestra.run.trial132:> sudo yum -y install rbd-nbd 2026-02-26T03:43:40.785 INFO:teuthology.orchestra.run.trial132.stdout:Last metadata expiration check: 0:00:12 ago on Thu 26 Feb 2026 03:43:28 AM UTC. 2026-02-26T03:43:40.849 INFO:teuthology.orchestra.run.trial132.stdout:Package rbd-nbd-2:20.3.0-5312.gabffb617.el9.x86_64 is already installed. 2026-02-26T03:43:40.867 INFO:teuthology.orchestra.run.trial132.stdout:Dependencies resolved. 2026-02-26T03:43:40.868 INFO:teuthology.orchestra.run.trial132.stdout:Nothing to do. 2026-02-26T03:43:40.868 INFO:teuthology.orchestra.run.trial132.stdout:Complete! 2026-02-26T03:43:40.893 DEBUG:teuthology.parallel:result is None 2026-02-26T03:43:40.894 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=debug&distros=centos%2F9%2Fx86_64&sha1=abffb617683951511fccf3c3cab6e219f2ec83a8 2026-02-26T03:43:41.059 DEBUG:teuthology.orchestra.run.trial132:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2026-02-26T03:43:41.080 INFO:teuthology.orchestra.run.trial132.stdout:20.3.0-5312.gabffb617.el9 2026-02-26T03:43:41.080 INFO:teuthology.packaging:The installed version of ceph is 20.3.0-5312.gabffb617.el9 2026-02-26T03:43:41.081 INFO:teuthology.task.install:The correct ceph version 20.3.0-5312.gabffb617 is installed. 2026-02-26T03:43:41.083 INFO:teuthology.task.install.util:Utilities already shipped, skip it... 2026-02-26T03:43:41.083 INFO:teuthology.run_tasks:Running task ceph... 2026-02-26T03:43:41.185 INFO:tasks.ceph:Making ceph log dir writeable by non-root... 2026-02-26T03:43:41.185 DEBUG:teuthology.orchestra.run.trial132:> sudo chmod 777 /var/log/ceph 2026-02-26T03:43:41.210 INFO:tasks.ceph:Disabling ceph logrotate... 2026-02-26T03:43:41.210 DEBUG:teuthology.orchestra.run.trial132:> sudo rm -f -- /etc/logrotate.d/ceph 2026-02-26T03:43:41.273 INFO:tasks.ceph:Creating extra log directories... 2026-02-26T03:43:41.273 DEBUG:teuthology.orchestra.run.trial132:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2026-02-26T03:43:41.348 INFO:tasks.ceph:Creating ceph cluster ceph... 2026-02-26T03:43:41.349 INFO:tasks.ceph:config {'conf': {'mon': {'mon min osdmap epochs': 50, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'paxos service trim min': 10, 'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'osd pool default crimson': True}, 'osd': {'debug monc': 20, 'crimson cpu num': 4, 'crimson osd obc lru size': 10, 'debug ms': 20, 'debug osd': 20, 'debug seastore': 20, 'debug seastore backref': 20, 'debug seastore cache': 20, 'debug seastore cleaner': 20, 'debug seastore device': 20, 'debug seastore epm': 20, 'debug seastore fixedkv tree': 20, 'debug seastore journal': 20, 'debug seastore lba': 20, 'debug seastore odata': 20, 'debug seastore omap': 20, 'debug seastore onode': 20, 'debug seastore t': 20, 'debug seastore tm': 20, 'osd objectstore': 'seastore', 'osd shutdown pgref assert': True, 'seastore cachepin size pershard': '64M', 'seastore cachepin type': '2Q', 'seastore max concurrent transactions': 8}, 'global': {'enable experimental unrecoverable data corrupting features': 'crimson', 'ms client mode': 'crc', 'ms cluster mode': 'crc', 'ms mon client mode': 'crc', 'ms mon cluster mode': 'crc', 'ms mon service mode': 'crc', 'ms service mode': 'crc'}, 'mgr': {'debug mgr': 20, 'debug ms': 1}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(POOL_APP_NOT_ENABLED\\)'], 'cpu_profile': set(), 'cluster': 'ceph', 'mon_bind_msgr2': True, 'mon_bind_addrvec': True} 2026-02-26T03:43:41.349 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/mnt/teuthology/shraddhaag-2026-02-26_02:22:56-crimson-rados-main-distro-debug-trial/71215', 'branch': 'main', 'description': 'crimson-rados/rbd/{0-crimson_install clusters/crimson-fixed-1 crimson-supported-all-distro/centos_latest crimson_qa_overrides deploy/ceph objectstore/seastore/segmented$/{crimson_seastore_segmented_2q} tasks/rbd_lock_and_fence}', 'email': None, 'first_in_suite': False, 'flavor': 'debug', 'job_id': '71215', 'kernel': {'branch': 'distro', 'kdb': True, 'sha1': 'distro'}, 'last_in_suite': False, 'machine_type': 'trial', 'name': 'shraddhaag-2026-02-26_02:22:56-crimson-rados-main-distro-debug-trial', 'no_nested_subset': False, 'os_type': 'centos', 'os_version': '9.stream', 'overrides': {'admin_socket': {'branch': 'main'}, 'ceph': {'conf': {'global': {'enable experimental unrecoverable data corrupting features': 'crimson', 'ms client mode': 'crc', 'ms cluster mode': 'crc', 'ms mon client mode': 'crc', 'ms mon cluster mode': 'crc', 'ms mon service mode': 'crc', 'ms service mode': 'crc'}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'osd pool default crimson': True}, 'osd': {'crimson cpu num': 4, 'crimson osd obc lru size': 10, 'debug ms': 20, 'debug osd': 20, 'debug seastore': 20, 'debug seastore backref': 20, 'debug seastore cache': 20, 'debug seastore cleaner': 20, 'debug seastore device': 20, 'debug seastore epm': 20, 'debug seastore fixedkv tree': 20, 'debug seastore journal': 20, 'debug seastore lba': 20, 'debug seastore odata': 20, 'debug seastore omap': 20, 'debug seastore onode': 20, 'debug seastore t': 20, 'debug seastore tm': 20, 'osd objectstore': 'seastore', 'osd shutdown pgref assert': True, 'seastore cachepin size pershard': '64M', 'seastore cachepin type': '2Q', 'seastore max concurrent transactions': 8}}, 'crimson_compat': True, 'flavor': 'debug', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(POOL_APP_NOT_ENABLED\\)'], 'sha1': 'abffb617683951511fccf3c3cab6e219f2ec83a8'}, 'ceph-deploy': {'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'global': {'osd crush chooseleaf type': 0, 'osd pool default pg num': 128, 'osd pool default pgp num': 128, 'osd pool default size': 2}, 'mon': {}}}, 'install': {'ceph': {'flavor': 'debug', 'sha1': 'abffb617683951511fccf3c3cab6e219f2ec83a8'}}, 'workunit': {'branch': 'main', 'env': {'CRIMSON_COMPAT': '1'}, 'sha1': '1f87b69b1a8041112e26ae0ea3c0a2a0fd14d996'}}, 'owner': 'scheduled_shraddhaag@soko04.front.sepia.ceph.com', 'priority': 101, 'repo': 'https://git.ceph.com/ceph-ci.git', 'roles': [['mon.a', 'mgr.x', 'osd.0', 'osd.1', 'osd.2', 'client.0']], 'seed': 5117, 'sha1': 'abffb617683951511fccf3c3cab6e219f2ec83a8', 'sleep_before_teardown': 0, 'suite': 'crimson-rados', 'suite_branch': 'main', 'suite_path': '/home/teuthworker/src/github.com_ceph_ceph_1f87b69b1a8041112e26ae0ea3c0a2a0fd14d996/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://github.com/ceph/ceph.git', 'suite_sha1': '1f87b69b1a8041112e26ae0ea3c0a2a0fd14d996', 'targets': {'trial132.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBJIsuoPOk/pW9C2qwYD17/s6eAz2ubvSBrM1JvHl7ScRFJU0FMVfQGaVW5S283uWQnOTSubNmVvvqJmKYuwXDGg='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'branch': 'distro', 'kdb': True, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': {'extra_packages': {'deb': ['ceph-osd-crimson', 'ceph-osd-crimson-dbg'], 'rpm': ['ceph-osd-crimson', 'ceph-osd-crimson-debuginfo']}, 'flavor': 'debug', 'sha1': 'abffb617683951511fccf3c3cab6e219f2ec83a8'}}, {'pexec': {'all': ['sudo update-alternatives --set ceph-osd /usr/bin/ceph-osd-crimson']}}, {'install': None}, {'ceph': {'conf': {'mon': {'mon min osdmap epochs': 50, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'paxos service trim min': 10, 'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'osd pool default crimson': True}, 'osd': {'debug monc': 20, 'crimson cpu num': 4, 'crimson osd obc lru size': 10, 'debug ms': 20, 'debug osd': 20, 'debug seastore': 20, 'debug seastore backref': 20, 'debug seastore cache': 20, 'debug seastore cleaner': 20, 'debug seastore device': 20, 'debug seastore epm': 20, 'debug seastore fixedkv tree': 20, 'debug seastore journal': 20, 'debug seastore lba': 20, 'debug seastore odata': 20, 'debug seastore omap': 20, 'debug seastore onode': 20, 'debug seastore t': 20, 'debug seastore tm': 20, 'osd objectstore': 'seastore', 'osd shutdown pgref assert': True, 'seastore cachepin size pershard': '64M', 'seastore cachepin type': '2Q', 'seastore max concurrent transactions': 8}, 'global': {'enable experimental unrecoverable data corrupting features': 'crimson', 'ms client mode': 'crc', 'ms cluster mode': 'crc', 'ms mon client mode': 'crc', 'ms mon cluster mode': 'crc', 'ms mon service mode': 'crc', 'ms service mode': 'crc'}, 'mgr': {'debug mgr': 20, 'debug ms': 1}}, 'crimson_compat': True, 'flavor': 'debug', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(POOL_APP_NOT_ENABLED\\)'], 'sha1': 'abffb617683951511fccf3c3cab6e219f2ec83a8', 'cluster': 'ceph'}}, {'workunit': {'clients': {'client.0': ['rbd/test_lock_fence.sh']}}}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': []}, 'teuthology_branch': 'main', 'teuthology_sha1': '6e3d0419ccca0e75a03e08feb708946e198236f4', 'timestamp': '2026-02-26_02:22:56', 'tube': 'trial', 'user': 'shraddhaag', 'verbose': False, 'worker_log': '/home/teuthworker/mnt/teuthology/worker_logs/dispatcher.trial.904699'} 2026-02-26T03:43:41.350 DEBUG:teuthology.orchestra.run.trial132:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2026-02-26T03:43:41.393 DEBUG:teuthology.orchestra.run.trial132:> sudo install -d -m0777 -- /var/run/ceph 2026-02-26T03:43:41.456 DEBUG:teuthology.orchestra.run.trial132:> set -ex 2026-02-26T03:43:41.457 DEBUG:teuthology.orchestra.run.trial132:> dd if=/scratch_devs of=/dev/stdout 2026-02-26T03:43:41.512 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2026-02-26T03:43:41.513 DEBUG:teuthology.orchestra.run.trial132:> stat /dev/vg_nvme/lv_1 2026-02-26T03:43:41.569 INFO:teuthology.orchestra.run.trial132.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2026-02-26T03:43:41.569 INFO:teuthology.orchestra.run.trial132.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2026-02-26T03:43:41.569 INFO:teuthology.orchestra.run.trial132.stdout:Device: 6h/6d Inode: 1293 Links: 1 2026-02-26T03:43:41.569 INFO:teuthology.orchestra.run.trial132.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2026-02-26T03:43:41.569 INFO:teuthology.orchestra.run.trial132.stdout:Context: system_u:object_r:device_t:s0 2026-02-26T03:43:41.569 INFO:teuthology.orchestra.run.trial132.stdout:Access: 2026-02-26 03:30:50.793711957 +0000 2026-02-26T03:43:41.569 INFO:teuthology.orchestra.run.trial132.stdout:Modify: 2026-02-26 03:22:04.017759309 +0000 2026-02-26T03:43:41.570 INFO:teuthology.orchestra.run.trial132.stdout:Change: 2026-02-26 03:22:04.017759309 +0000 2026-02-26T03:43:41.570 INFO:teuthology.orchestra.run.trial132.stdout: Birth: 2026-02-26 03:22:04.017759309 +0000 2026-02-26T03:43:41.570 DEBUG:teuthology.orchestra.run.trial132:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2026-02-26T03:43:41.632 INFO:teuthology.orchestra.run.trial132.stderr:1+0 records in 2026-02-26T03:43:41.632 INFO:teuthology.orchestra.run.trial132.stderr:1+0 records out 2026-02-26T03:43:41.632 INFO:teuthology.orchestra.run.trial132.stderr:512 bytes copied, 0.000167924 s, 3.0 MB/s 2026-02-26T03:43:41.633 DEBUG:teuthology.orchestra.run.trial132:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2026-02-26T03:43:41.690 DEBUG:teuthology.orchestra.run.trial132:> stat /dev/vg_nvme/lv_2 2026-02-26T03:43:41.746 INFO:teuthology.orchestra.run.trial132.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2026-02-26T03:43:41.746 INFO:teuthology.orchestra.run.trial132.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2026-02-26T03:43:41.747 INFO:teuthology.orchestra.run.trial132.stdout:Device: 6h/6d Inode: 1297 Links: 1 2026-02-26T03:43:41.747 INFO:teuthology.orchestra.run.trial132.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2026-02-26T03:43:41.747 INFO:teuthology.orchestra.run.trial132.stdout:Context: system_u:object_r:device_t:s0 2026-02-26T03:43:41.747 INFO:teuthology.orchestra.run.trial132.stdout:Access: 2026-02-26 03:30:50.793711957 +0000 2026-02-26T03:43:41.747 INFO:teuthology.orchestra.run.trial132.stdout:Modify: 2026-02-26 03:22:04.017759309 +0000 2026-02-26T03:43:41.747 INFO:teuthology.orchestra.run.trial132.stdout:Change: 2026-02-26 03:22:04.017759309 +0000 2026-02-26T03:43:41.747 INFO:teuthology.orchestra.run.trial132.stdout: Birth: 2026-02-26 03:22:04.017759309 +0000 2026-02-26T03:43:41.747 DEBUG:teuthology.orchestra.run.trial132:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2026-02-26T03:43:41.809 INFO:teuthology.orchestra.run.trial132.stderr:1+0 records in 2026-02-26T03:43:41.809 INFO:teuthology.orchestra.run.trial132.stderr:1+0 records out 2026-02-26T03:43:41.809 INFO:teuthology.orchestra.run.trial132.stderr:512 bytes copied, 0.000133881 s, 3.8 MB/s 2026-02-26T03:43:41.810 DEBUG:teuthology.orchestra.run.trial132:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2026-02-26T03:43:41.867 DEBUG:teuthology.orchestra.run.trial132:> stat /dev/vg_nvme/lv_3 2026-02-26T03:43:41.923 INFO:teuthology.orchestra.run.trial132.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2026-02-26T03:43:41.923 INFO:teuthology.orchestra.run.trial132.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2026-02-26T03:43:41.924 INFO:teuthology.orchestra.run.trial132.stdout:Device: 6h/6d Inode: 1306 Links: 1 2026-02-26T03:43:41.924 INFO:teuthology.orchestra.run.trial132.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2026-02-26T03:43:41.924 INFO:teuthology.orchestra.run.trial132.stdout:Context: system_u:object_r:device_t:s0 2026-02-26T03:43:41.924 INFO:teuthology.orchestra.run.trial132.stdout:Access: 2026-02-26 03:30:50.793711957 +0000 2026-02-26T03:43:41.924 INFO:teuthology.orchestra.run.trial132.stdout:Modify: 2026-02-26 03:22:04.017759309 +0000 2026-02-26T03:43:41.924 INFO:teuthology.orchestra.run.trial132.stdout:Change: 2026-02-26 03:22:04.017759309 +0000 2026-02-26T03:43:41.924 INFO:teuthology.orchestra.run.trial132.stdout: Birth: 2026-02-26 03:22:04.017759309 +0000 2026-02-26T03:43:41.924 DEBUG:teuthology.orchestra.run.trial132:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2026-02-26T03:43:41.986 INFO:teuthology.orchestra.run.trial132.stderr:1+0 records in 2026-02-26T03:43:41.986 INFO:teuthology.orchestra.run.trial132.stderr:1+0 records out 2026-02-26T03:43:41.986 INFO:teuthology.orchestra.run.trial132.stderr:512 bytes copied, 0.000157265 s, 3.3 MB/s 2026-02-26T03:43:41.987 DEBUG:teuthology.orchestra.run.trial132:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2026-02-26T03:43:42.044 DEBUG:teuthology.orchestra.run.trial132:> stat /dev/vg_nvme/lv_4 2026-02-26T03:43:42.100 INFO:teuthology.orchestra.run.trial132.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2026-02-26T03:43:42.100 INFO:teuthology.orchestra.run.trial132.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2026-02-26T03:43:42.101 INFO:teuthology.orchestra.run.trial132.stdout:Device: 6h/6d Inode: 1288 Links: 1 2026-02-26T03:43:42.101 INFO:teuthology.orchestra.run.trial132.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2026-02-26T03:43:42.101 INFO:teuthology.orchestra.run.trial132.stdout:Context: system_u:object_r:device_t:s0 2026-02-26T03:43:42.101 INFO:teuthology.orchestra.run.trial132.stdout:Access: 2026-02-26 03:30:50.793711957 +0000 2026-02-26T03:43:42.101 INFO:teuthology.orchestra.run.trial132.stdout:Modify: 2026-02-26 03:22:04.017759309 +0000 2026-02-26T03:43:42.101 INFO:teuthology.orchestra.run.trial132.stdout:Change: 2026-02-26 03:22:04.017759309 +0000 2026-02-26T03:43:42.101 INFO:teuthology.orchestra.run.trial132.stdout: Birth: 2026-02-26 03:22:04.017759309 +0000 2026-02-26T03:43:42.101 DEBUG:teuthology.orchestra.run.trial132:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2026-02-26T03:43:42.163 INFO:teuthology.orchestra.run.trial132.stderr:1+0 records in 2026-02-26T03:43:42.163 INFO:teuthology.orchestra.run.trial132.stderr:1+0 records out 2026-02-26T03:43:42.163 INFO:teuthology.orchestra.run.trial132.stderr:512 bytes copied, 0.00014964 s, 3.4 MB/s 2026-02-26T03:43:42.164 DEBUG:teuthology.orchestra.run.trial132:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2026-02-26T03:43:42.221 INFO:tasks.ceph:osd dev map: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2026-02-26T03:43:42.221 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@trial132.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'}} 2026-02-26T03:43:42.222 INFO:tasks.ceph:Generating config... 2026-02-26T03:43:42.224 INFO:tasks.ceph:[mon] mon min osdmap epochs = 50 2026-02-26T03:43:42.224 INFO:tasks.ceph:[mon] mon osdmap full prune interval = 2 2026-02-26T03:43:42.224 INFO:tasks.ceph:[mon] mon osdmap full prune min = 15 2026-02-26T03:43:42.224 INFO:tasks.ceph:[mon] mon osdmap full prune txsize = 2 2026-02-26T03:43:42.224 INFO:tasks.ceph:[mon] paxos service trim min = 10 2026-02-26T03:43:42.224 INFO:tasks.ceph:[mon] debug mon = 20 2026-02-26T03:43:42.225 INFO:tasks.ceph:[mon] debug ms = 1 2026-02-26T03:43:42.225 INFO:tasks.ceph:[mon] debug paxos = 20 2026-02-26T03:43:42.225 INFO:tasks.ceph:[mon] osd pool default crimson = True 2026-02-26T03:43:42.225 INFO:tasks.ceph:[osd] debug monc = 20 2026-02-26T03:43:42.225 INFO:tasks.ceph:[osd] crimson cpu num = 4 2026-02-26T03:43:42.225 INFO:tasks.ceph:[osd] crimson osd obc lru size = 10 2026-02-26T03:43:42.225 INFO:tasks.ceph:[osd] debug ms = 20 2026-02-26T03:43:42.225 INFO:tasks.ceph:[osd] debug osd = 20 2026-02-26T03:43:42.225 INFO:tasks.ceph:[osd] debug seastore = 20 2026-02-26T03:43:42.225 INFO:tasks.ceph:[osd] debug seastore backref = 20 2026-02-26T03:43:42.225 INFO:tasks.ceph:[osd] debug seastore cache = 20 2026-02-26T03:43:42.225 INFO:tasks.ceph:[osd] debug seastore cleaner = 20 2026-02-26T03:43:42.225 INFO:tasks.ceph:[osd] debug seastore device = 20 2026-02-26T03:43:42.226 INFO:tasks.ceph:[osd] debug seastore epm = 20 2026-02-26T03:43:42.226 INFO:tasks.ceph:[osd] debug seastore fixedkv tree = 20 2026-02-26T03:43:42.226 INFO:tasks.ceph:[osd] debug seastore journal = 20 2026-02-26T03:43:42.226 INFO:tasks.ceph:[osd] debug seastore lba = 20 2026-02-26T03:43:42.226 INFO:tasks.ceph:[osd] debug seastore odata = 20 2026-02-26T03:43:42.226 INFO:tasks.ceph:[osd] debug seastore omap = 20 2026-02-26T03:43:42.226 INFO:tasks.ceph:[osd] debug seastore onode = 20 2026-02-26T03:43:42.226 INFO:tasks.ceph:[osd] debug seastore t = 20 2026-02-26T03:43:42.226 INFO:tasks.ceph:[osd] debug seastore tm = 20 2026-02-26T03:43:42.226 INFO:tasks.ceph:[osd] osd objectstore = seastore 2026-02-26T03:43:42.226 INFO:tasks.ceph:[osd] osd shutdown pgref assert = True 2026-02-26T03:43:42.226 INFO:tasks.ceph:[osd] seastore cachepin size pershard = 64M 2026-02-26T03:43:42.226 INFO:tasks.ceph:[osd] seastore cachepin type = 2Q 2026-02-26T03:43:42.227 INFO:tasks.ceph:[osd] seastore max concurrent transactions = 8 2026-02-26T03:43:42.227 INFO:tasks.ceph:[global] enable experimental unrecoverable data corrupting features = crimson 2026-02-26T03:43:42.227 INFO:tasks.ceph:[global] ms client mode = crc 2026-02-26T03:43:42.227 INFO:tasks.ceph:[global] ms cluster mode = crc 2026-02-26T03:43:42.227 INFO:tasks.ceph:[global] ms mon client mode = crc 2026-02-26T03:43:42.227 INFO:tasks.ceph:[global] ms mon cluster mode = crc 2026-02-26T03:43:42.227 INFO:tasks.ceph:[global] ms mon service mode = crc 2026-02-26T03:43:42.227 INFO:tasks.ceph:[global] ms service mode = crc 2026-02-26T03:43:42.227 INFO:tasks.ceph:[mgr] debug mgr = 20 2026-02-26T03:43:42.227 INFO:tasks.ceph:[mgr] debug ms = 1 2026-02-26T03:43:42.227 INFO:tasks.ceph:Setting up mon.a... 2026-02-26T03:43:42.228 DEBUG:teuthology.orchestra.run.trial132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/ceph.keyring 2026-02-26T03:43:42.301 INFO:teuthology.orchestra.run.trial132.stdout:creating /etc/ceph/ceph.keyring 2026-02-26T03:43:42.304 DEBUG:teuthology.orchestra.run.trial132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/ceph.keyring 2026-02-26T03:43:42.387 DEBUG:teuthology.orchestra.run.trial132:> sudo chmod 0644 /etc/ceph/ceph.keyring 2026-02-26T03:43:42.450 DEBUG:tasks.ceph:Ceph mon addresses: [('mon.a', '10.20.193.132')] 2026-02-26T03:43:42.451 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'cephx', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=isa technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '10.20.193.132', 'enable experimental unrecoverable data corrupting features': 'crimson', 'ms client mode': 'crc', 'ms cluster mode': 'crc', 'ms mon client mode': 'crc', 'ms mon cluster mode': 'crc', 'ms mon service mode': 'crc', 'ms service mode': 'crc'}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd mclock profile': 'high_recovery_ops', 'osd mclock skip benchmark': 'true', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd debug trim objects': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'debug monc': 20, 'crimson cpu num': 4, 'crimson osd obc lru size': 10, 'debug ms': 20, 'debug osd': 20, 'debug seastore': 20, 'debug seastore backref': 20, 'debug seastore cache': 20, 'debug seastore cleaner': 20, 'debug seastore device': 20, 'debug seastore epm': 20, 'debug seastore fixedkv tree': 20, 'debug seastore journal': 20, 'debug seastore lba': 20, 'debug seastore odata': 20, 'debug seastore omap': 20, 'debug seastore onode': 20, 'debug seastore t': 20, 'debug seastore tm': 20, 'osd objectstore': 'seastore', 'osd shutdown pgref assert': True, 'seastore cachepin size pershard': '64M', 'seastore cachepin type': '2Q', 'seastore max concurrent transactions': 8}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false'}, 'mon': {'debug ms': 1, 'debug mon': 20, 'debug paxos': 20, 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'mon_down_mkfs_grace': '2m', 'mon_warn_on_filestore_osds': 'false', 'mon min osdmap epochs': 50, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'paxos service trim min': 10, 'osd pool default crimson': True}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok'}, 'mon.a': {}} 2026-02-26T03:43:42.453 DEBUG:teuthology.orchestra.run.trial132:> set -ex 2026-02-26T03:43:42.453 DEBUG:teuthology.orchestra.run.trial132:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2026-02-26T03:43:42.507 DEBUG:teuthology.orchestra.run.trial132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --add a 10.20.193.132 --print /home/ubuntu/cephtest/ceph.monmap 2026-02-26T03:43:42.588 INFO:teuthology.orchestra.run.trial132.stderr:2026-02-26T03:43:42.589+0000 7efd14c40f40 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-26T03:43:42.588 INFO:teuthology.orchestra.run.trial132.stderr:2026-02-26T03:43:42.589+0000 7efd14c40f40 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-26T03:43:42.588 INFO:teuthology.orchestra.run.trial132.stderr:2026-02-26T03:43:42.589+0000 7efd14c40f40 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-26T03:43:42.589 INFO:teuthology.orchestra.run.trial132.stdout:monmaptool: monmap file /home/ubuntu/cephtest/ceph.monmap 2026-02-26T03:43:42.589 INFO:teuthology.orchestra.run.trial132.stdout:monmaptool: generated fsid 109d3300-c629-4972-9fab-d8a7123a9741 2026-02-26T03:43:42.589 INFO:teuthology.orchestra.run.trial132.stdout:setting min_mon_release = tentacle 2026-02-26T03:43:42.589 INFO:teuthology.orchestra.run.trial132.stdout:epoch 0 2026-02-26T03:43:42.589 INFO:teuthology.orchestra.run.trial132.stdout:fsid 109d3300-c629-4972-9fab-d8a7123a9741 2026-02-26T03:43:42.589 INFO:teuthology.orchestra.run.trial132.stdout:last_changed 2026-02-26T03:43:42.590894+0000 2026-02-26T03:43:42.590 INFO:teuthology.orchestra.run.trial132.stdout:created 2026-02-26T03:43:42.590894+0000 2026-02-26T03:43:42.590 INFO:teuthology.orchestra.run.trial132.stdout:min_mon_release 20 (tentacle) 2026-02-26T03:43:42.590 INFO:teuthology.orchestra.run.trial132.stdout:election_strategy: 1 2026-02-26T03:43:42.590 INFO:teuthology.orchestra.run.trial132.stdout:0: [v2:10.20.193.132:3300/0,v1:10.20.193.132:6789/0] mon.a 2026-02-26T03:43:42.590 INFO:teuthology.orchestra.run.trial132.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/ceph.monmap (1 monitors) 2026-02-26T03:43:42.591 DEBUG:teuthology.orchestra.run.trial132:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2026-02-26T03:43:42.648 INFO:tasks.ceph:Writing /etc/ceph/ceph.conf for FSID 109d3300-c629-4972-9fab-d8a7123a9741... 2026-02-26T03:43:42.649 DEBUG:teuthology.orchestra.run.trial132:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2026-02-26T03:43:42.729 INFO:teuthology.orchestra.run.trial132.stdout:[global] 2026-02-26T03:43:42.730 INFO:teuthology.orchestra.run.trial132.stdout: chdir = "" 2026-02-26T03:43:42.730 INFO:teuthology.orchestra.run.trial132.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2026-02-26T03:43:42.730 INFO:teuthology.orchestra.run.trial132.stdout: auth supported = cephx 2026-02-26T03:43:42.730 INFO:teuthology.orchestra.run.trial132.stdout: 2026-02-26T03:43:42.730 INFO:teuthology.orchestra.run.trial132.stdout: filestore xattr use omap = true 2026-02-26T03:43:42.730 INFO:teuthology.orchestra.run.trial132.stdout: 2026-02-26T03:43:42.730 INFO:teuthology.orchestra.run.trial132.stdout: mon clock drift allowed = 1.000 2026-02-26T03:43:42.730 INFO:teuthology.orchestra.run.trial132.stdout: 2026-02-26T03:43:42.730 INFO:teuthology.orchestra.run.trial132.stdout: osd crush chooseleaf type = 0 2026-02-26T03:43:42.731 INFO:teuthology.orchestra.run.trial132.stdout: auth debug = true 2026-02-26T03:43:42.731 INFO:teuthology.orchestra.run.trial132.stdout: 2026-02-26T03:43:42.731 INFO:teuthology.orchestra.run.trial132.stdout: ms die on old message = true 2026-02-26T03:43:42.731 INFO:teuthology.orchestra.run.trial132.stdout: ms die on bug = true 2026-02-26T03:43:42.731 INFO:teuthology.orchestra.run.trial132.stdout: 2026-02-26T03:43:42.731 INFO:teuthology.orchestra.run.trial132.stdout: mon max pg per osd = 10000 # >= luminous 2026-02-26T03:43:42.731 INFO:teuthology.orchestra.run.trial132.stdout: mon pg warn max object skew = 0 2026-02-26T03:43:42.731 INFO:teuthology.orchestra.run.trial132.stdout: 2026-02-26T03:43:42.731 INFO:teuthology.orchestra.run.trial132.stdout: # disable pg_autoscaler by default for new pools 2026-02-26T03:43:42.731 INFO:teuthology.orchestra.run.trial132.stdout: osd_pool_default_pg_autoscale_mode = off 2026-02-26T03:43:42.731 INFO:teuthology.orchestra.run.trial132.stdout: 2026-02-26T03:43:42.731 INFO:teuthology.orchestra.run.trial132.stdout: osd pool default size = 2 2026-02-26T03:43:42.732 INFO:teuthology.orchestra.run.trial132.stdout: 2026-02-26T03:43:42.732 INFO:teuthology.orchestra.run.trial132.stdout: mon osd allow primary affinity = true 2026-02-26T03:43:42.732 INFO:teuthology.orchestra.run.trial132.stdout: mon osd allow pg remap = true 2026-02-26T03:43:42.732 INFO:teuthology.orchestra.run.trial132.stdout: mon warn on legacy crush tunables = false 2026-02-26T03:43:42.732 INFO:teuthology.orchestra.run.trial132.stdout: mon warn on crush straw calc version zero = false 2026-02-26T03:43:42.732 INFO:teuthology.orchestra.run.trial132.stdout: mon warn on no sortbitwise = false 2026-02-26T03:43:42.732 INFO:teuthology.orchestra.run.trial132.stdout: mon warn on osd down out interval zero = false 2026-02-26T03:43:42.732 INFO:teuthology.orchestra.run.trial132.stdout: mon warn on too few osds = false 2026-02-26T03:43:42.732 INFO:teuthology.orchestra.run.trial132.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2026-02-26T03:43:42.732 INFO:teuthology.orchestra.run.trial132.stdout: mon_warn_on_pool_no_redundancy = false 2026-02-26T03:43:42.732 INFO:teuthology.orchestra.run.trial132.stdout: mon_allow_pool_size_one = true 2026-02-26T03:43:42.732 INFO:teuthology.orchestra.run.trial132.stdout: 2026-02-26T03:43:42.733 INFO:teuthology.orchestra.run.trial132.stdout: osd pool default erasure code profile = plugin=isa technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2026-02-26T03:43:42.733 INFO:teuthology.orchestra.run.trial132.stdout: 2026-02-26T03:43:42.733 INFO:teuthology.orchestra.run.trial132.stdout: osd default data pool replay window = 5 2026-02-26T03:43:42.733 INFO:teuthology.orchestra.run.trial132.stdout: 2026-02-26T03:43:42.733 INFO:teuthology.orchestra.run.trial132.stdout: mon allow pool delete = true 2026-02-26T03:43:42.733 INFO:teuthology.orchestra.run.trial132.stdout: 2026-02-26T03:43:42.733 INFO:teuthology.orchestra.run.trial132.stdout: mon cluster log file level = debug 2026-02-26T03:43:42.733 INFO:teuthology.orchestra.run.trial132.stdout: debug asserts on shutdown = true 2026-02-26T03:43:42.733 INFO:teuthology.orchestra.run.trial132.stdout: mon health detail to clog = false 2026-02-26T03:43:42.733 INFO:teuthology.orchestra.run.trial132.stdout: mon host = 10.20.193.132 2026-02-26T03:43:42.733 INFO:teuthology.orchestra.run.trial132.stdout: enable experimental unrecoverable data corrupting features = crimson 2026-02-26T03:43:42.733 INFO:teuthology.orchestra.run.trial132.stdout: ms client mode = crc 2026-02-26T03:43:42.734 INFO:teuthology.orchestra.run.trial132.stdout: ms cluster mode = crc 2026-02-26T03:43:42.734 INFO:teuthology.orchestra.run.trial132.stdout: ms mon client mode = crc 2026-02-26T03:43:42.734 INFO:teuthology.orchestra.run.trial132.stdout: ms mon cluster mode = crc 2026-02-26T03:43:42.734 INFO:teuthology.orchestra.run.trial132.stdout: ms mon service mode = crc 2026-02-26T03:43:42.734 INFO:teuthology.orchestra.run.trial132.stdout: ms service mode = crc 2026-02-26T03:43:42.734 INFO:teuthology.orchestra.run.trial132.stdout: fsid = 109d3300-c629-4972-9fab-d8a7123a9741 2026-02-26T03:43:42.734 INFO:teuthology.orchestra.run.trial132.stdout: 2026-02-26T03:43:42.734 INFO:teuthology.orchestra.run.trial132.stdout:[osd] 2026-02-26T03:43:42.734 INFO:teuthology.orchestra.run.trial132.stdout: osd journal size = 100 2026-02-26T03:43:42.734 INFO:teuthology.orchestra.run.trial132.stdout: 2026-02-26T03:43:42.734 INFO:teuthology.orchestra.run.trial132.stdout: osd scrub load threshold = 5.0 2026-02-26T03:43:42.734 INFO:teuthology.orchestra.run.trial132.stdout: osd scrub max interval = 600 2026-02-26T03:43:42.734 INFO:teuthology.orchestra.run.trial132.stdout: osd mclock profile = high_recovery_ops 2026-02-26T03:43:42.735 INFO:teuthology.orchestra.run.trial132.stdout: osd mclock skip benchmark = true 2026-02-26T03:43:42.735 INFO:teuthology.orchestra.run.trial132.stdout: 2026-02-26T03:43:42.735 INFO:teuthology.orchestra.run.trial132.stdout: osd recover clone overlap = true 2026-02-26T03:43:42.735 INFO:teuthology.orchestra.run.trial132.stdout: osd recovery max chunk = 1048576 2026-02-26T03:43:42.735 INFO:teuthology.orchestra.run.trial132.stdout: 2026-02-26T03:43:42.735 INFO:teuthology.orchestra.run.trial132.stdout: osd debug shutdown = true 2026-02-26T03:43:42.735 INFO:teuthology.orchestra.run.trial132.stdout: osd debug op order = true 2026-02-26T03:43:42.735 INFO:teuthology.orchestra.run.trial132.stdout: osd debug verify stray on activate = true 2026-02-26T03:43:42.735 INFO:teuthology.orchestra.run.trial132.stdout: osd debug trim objects = true 2026-02-26T03:43:42.735 INFO:teuthology.orchestra.run.trial132.stdout: 2026-02-26T03:43:42.735 INFO:teuthology.orchestra.run.trial132.stdout: osd open classes on start = true 2026-02-26T03:43:42.735 INFO:teuthology.orchestra.run.trial132.stdout: osd debug pg log writeout = true 2026-02-26T03:43:42.735 INFO:teuthology.orchestra.run.trial132.stdout: 2026-02-26T03:43:42.736 INFO:teuthology.orchestra.run.trial132.stdout: osd deep scrub update digest min age = 30 2026-02-26T03:43:42.736 INFO:teuthology.orchestra.run.trial132.stdout: 2026-02-26T03:43:42.736 INFO:teuthology.orchestra.run.trial132.stdout: osd map max advance = 10 2026-02-26T03:43:42.736 INFO:teuthology.orchestra.run.trial132.stdout: 2026-02-26T03:43:42.736 INFO:teuthology.orchestra.run.trial132.stdout: journal zero on create = true 2026-02-26T03:43:42.736 INFO:teuthology.orchestra.run.trial132.stdout: 2026-02-26T03:43:42.736 INFO:teuthology.orchestra.run.trial132.stdout: filestore ondisk finisher threads = 3 2026-02-26T03:43:42.736 INFO:teuthology.orchestra.run.trial132.stdout: filestore apply finisher threads = 3 2026-02-26T03:43:42.736 INFO:teuthology.orchestra.run.trial132.stdout: 2026-02-26T03:43:42.736 INFO:teuthology.orchestra.run.trial132.stdout: bdev debug aio = true 2026-02-26T03:43:42.736 INFO:teuthology.orchestra.run.trial132.stdout: osd debug misdirected ops = true 2026-02-26T03:43:42.736 INFO:teuthology.orchestra.run.trial132.stdout: debug monc = 20 2026-02-26T03:43:42.736 INFO:teuthology.orchestra.run.trial132.stdout: crimson cpu num = 4 2026-02-26T03:43:42.737 INFO:teuthology.orchestra.run.trial132.stdout: crimson osd obc lru size = 10 2026-02-26T03:43:42.737 INFO:teuthology.orchestra.run.trial132.stdout: debug ms = 20 2026-02-26T03:43:42.737 INFO:teuthology.orchestra.run.trial132.stdout: debug osd = 20 2026-02-26T03:43:42.737 INFO:teuthology.orchestra.run.trial132.stdout: debug seastore = 20 2026-02-26T03:43:42.737 INFO:teuthology.orchestra.run.trial132.stdout: debug seastore backref = 20 2026-02-26T03:43:42.737 INFO:teuthology.orchestra.run.trial132.stdout: debug seastore cache = 20 2026-02-26T03:43:42.737 INFO:teuthology.orchestra.run.trial132.stdout: debug seastore cleaner = 20 2026-02-26T03:43:42.737 INFO:teuthology.orchestra.run.trial132.stdout: debug seastore device = 20 2026-02-26T03:43:42.737 INFO:teuthology.orchestra.run.trial132.stdout: debug seastore epm = 20 2026-02-26T03:43:42.737 INFO:teuthology.orchestra.run.trial132.stdout: debug seastore fixedkv tree = 20 2026-02-26T03:43:42.737 INFO:teuthology.orchestra.run.trial132.stdout: debug seastore journal = 20 2026-02-26T03:43:42.737 INFO:teuthology.orchestra.run.trial132.stdout: debug seastore lba = 20 2026-02-26T03:43:42.737 INFO:teuthology.orchestra.run.trial132.stdout: debug seastore odata = 20 2026-02-26T03:43:42.738 INFO:teuthology.orchestra.run.trial132.stdout: debug seastore omap = 20 2026-02-26T03:43:42.738 INFO:teuthology.orchestra.run.trial132.stdout: debug seastore onode = 20 2026-02-26T03:43:42.738 INFO:teuthology.orchestra.run.trial132.stdout: debug seastore t = 20 2026-02-26T03:43:42.738 INFO:teuthology.orchestra.run.trial132.stdout: debug seastore tm = 20 2026-02-26T03:43:42.738 INFO:teuthology.orchestra.run.trial132.stdout: osd objectstore = seastore 2026-02-26T03:43:42.738 INFO:teuthology.orchestra.run.trial132.stdout: osd shutdown pgref assert = True 2026-02-26T03:43:42.738 INFO:teuthology.orchestra.run.trial132.stdout: seastore cachepin size pershard = 64M 2026-02-26T03:43:42.738 INFO:teuthology.orchestra.run.trial132.stdout: seastore cachepin type = 2Q 2026-02-26T03:43:42.738 INFO:teuthology.orchestra.run.trial132.stdout: seastore max concurrent transactions = 8 2026-02-26T03:43:42.738 INFO:teuthology.orchestra.run.trial132.stdout: 2026-02-26T03:43:42.738 INFO:teuthology.orchestra.run.trial132.stdout:[mgr] 2026-02-26T03:43:42.738 INFO:teuthology.orchestra.run.trial132.stdout: debug ms = 1 2026-02-26T03:43:42.739 INFO:teuthology.orchestra.run.trial132.stdout: debug mgr = 20 2026-02-26T03:43:42.739 INFO:teuthology.orchestra.run.trial132.stdout: debug mon = 20 2026-02-26T03:43:42.739 INFO:teuthology.orchestra.run.trial132.stdout: debug auth = 20 2026-02-26T03:43:42.739 INFO:teuthology.orchestra.run.trial132.stdout: mon reweight min pgs per osd = 4 2026-02-26T03:43:42.739 INFO:teuthology.orchestra.run.trial132.stdout: mon reweight min bytes per osd = 10 2026-02-26T03:43:42.739 INFO:teuthology.orchestra.run.trial132.stdout: mgr/telemetry/nag = false 2026-02-26T03:43:42.739 INFO:teuthology.orchestra.run.trial132.stdout: 2026-02-26T03:43:42.739 INFO:teuthology.orchestra.run.trial132.stdout:[mon] 2026-02-26T03:43:42.739 INFO:teuthology.orchestra.run.trial132.stdout: debug ms = 1 2026-02-26T03:43:42.739 INFO:teuthology.orchestra.run.trial132.stdout: debug mon = 20 2026-02-26T03:43:42.739 INFO:teuthology.orchestra.run.trial132.stdout: debug paxos = 20 2026-02-26T03:43:42.739 INFO:teuthology.orchestra.run.trial132.stdout: debug auth = 20 2026-02-26T03:43:42.739 INFO:teuthology.orchestra.run.trial132.stdout: mon data avail warn = 5 2026-02-26T03:43:42.740 INFO:teuthology.orchestra.run.trial132.stdout: mon mgr mkfs grace = 240 2026-02-26T03:43:42.740 INFO:teuthology.orchestra.run.trial132.stdout: mon reweight min pgs per osd = 4 2026-02-26T03:43:42.740 INFO:teuthology.orchestra.run.trial132.stdout: mon osd reporter subtree level = osd 2026-02-26T03:43:42.740 INFO:teuthology.orchestra.run.trial132.stdout: mon osd prime pg temp = true 2026-02-26T03:43:42.740 INFO:teuthology.orchestra.run.trial132.stdout: mon reweight min bytes per osd = 10 2026-02-26T03:43:42.740 INFO:teuthology.orchestra.run.trial132.stdout: 2026-02-26T03:43:42.740 INFO:teuthology.orchestra.run.trial132.stdout: # rotate auth tickets quickly to exercise renewal paths 2026-02-26T03:43:42.740 INFO:teuthology.orchestra.run.trial132.stdout: auth mon ticket ttl = 660 # 11m 2026-02-26T03:43:42.740 INFO:teuthology.orchestra.run.trial132.stdout: auth service ticket ttl = 240 # 4m 2026-02-26T03:43:42.740 INFO:teuthology.orchestra.run.trial132.stdout: 2026-02-26T03:43:42.740 INFO:teuthology.orchestra.run.trial132.stdout: # don't complain about insecure global_id in the test suite 2026-02-26T03:43:42.740 INFO:teuthology.orchestra.run.trial132.stdout: mon_warn_on_insecure_global_id_reclaim = false 2026-02-26T03:43:42.740 INFO:teuthology.orchestra.run.trial132.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2026-02-26T03:43:42.741 INFO:teuthology.orchestra.run.trial132.stdout: 2026-02-26T03:43:42.741 INFO:teuthology.orchestra.run.trial132.stdout: # 1m isn't quite enough 2026-02-26T03:43:42.741 INFO:teuthology.orchestra.run.trial132.stdout: mon_down_mkfs_grace = 2m 2026-02-26T03:43:42.741 INFO:teuthology.orchestra.run.trial132.stdout: 2026-02-26T03:43:42.741 INFO:teuthology.orchestra.run.trial132.stdout: mon_warn_on_filestore_osds = false 2026-02-26T03:43:42.741 INFO:teuthology.orchestra.run.trial132.stdout: mon min osdmap epochs = 50 2026-02-26T03:43:42.741 INFO:teuthology.orchestra.run.trial132.stdout: mon osdmap full prune interval = 2 2026-02-26T03:43:42.741 INFO:teuthology.orchestra.run.trial132.stdout: mon osdmap full prune min = 15 2026-02-26T03:43:42.741 INFO:teuthology.orchestra.run.trial132.stdout: mon osdmap full prune txsize = 2 2026-02-26T03:43:42.741 INFO:teuthology.orchestra.run.trial132.stdout: paxos service trim min = 10 2026-02-26T03:43:42.741 INFO:teuthology.orchestra.run.trial132.stdout: osd pool default crimson = True 2026-02-26T03:43:42.741 INFO:teuthology.orchestra.run.trial132.stdout: 2026-02-26T03:43:42.741 INFO:teuthology.orchestra.run.trial132.stdout:[client] 2026-02-26T03:43:42.742 INFO:teuthology.orchestra.run.trial132.stdout: rgw cache enabled = true 2026-02-26T03:43:42.742 INFO:teuthology.orchestra.run.trial132.stdout: rgw enable ops log = true 2026-02-26T03:43:42.742 INFO:teuthology.orchestra.run.trial132.stdout: rgw enable usage log = true 2026-02-26T03:43:42.742 INFO:teuthology.orchestra.run.trial132.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2026-02-26T03:43:42.742 INFO:teuthology.orchestra.run.trial132.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2026-02-26T03:43:42.742 INFO:teuthology.orchestra.run.trial132.stdout:[mon.a] 2026-02-26T03:43:42.743 INFO:tasks.ceph:Creating admin key on mon.a... 2026-02-26T03:43:42.743 DEBUG:teuthology.orchestra.run.trial132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/ceph.keyring 2026-02-26T03:43:42.833 INFO:tasks.ceph:Copying monmap to all nodes... 2026-02-26T03:43:42.833 DEBUG:teuthology.orchestra.run.trial132:> set -ex 2026-02-26T03:43:42.833 DEBUG:teuthology.orchestra.run.trial132:> dd if=/etc/ceph/ceph.keyring of=/dev/stdout 2026-02-26T03:43:42.850 DEBUG:teuthology.orchestra.run.trial132:> set -ex 2026-02-26T03:43:42.850 DEBUG:teuthology.orchestra.run.trial132:> dd if=/home/ubuntu/cephtest/ceph.monmap of=/dev/stdout 2026-02-26T03:43:42.906 INFO:tasks.ceph:Sending monmap to node ubuntu@trial132.front.sepia.ceph.com 2026-02-26T03:43:42.907 DEBUG:teuthology.orchestra.run.trial132:> set -ex 2026-02-26T03:43:42.907 DEBUG:teuthology.orchestra.run.trial132:> sudo dd of=/etc/ceph/ceph.keyring 2026-02-26T03:43:42.907 DEBUG:teuthology.orchestra.run.trial132:> sudo chmod 0644 /etc/ceph/ceph.keyring 2026-02-26T03:43:42.979 DEBUG:teuthology.orchestra.run.trial132:> set -ex 2026-02-26T03:43:42.980 DEBUG:teuthology.orchestra.run.trial132:> dd of=/home/ubuntu/cephtest/ceph.monmap 2026-02-26T03:43:43.036 INFO:tasks.ceph:Setting up mon nodes... 2026-02-26T03:43:43.036 INFO:tasks.ceph:Setting up mgr nodes... 2026-02-26T03:43:43.036 DEBUG:teuthology.orchestra.run.trial132:> sudo mkdir -p /var/lib/ceph/mgr/ceph-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/ceph-x/keyring 2026-02-26T03:43:43.126 INFO:teuthology.orchestra.run.trial132.stdout:creating /var/lib/ceph/mgr/ceph-x/keyring 2026-02-26T03:43:43.129 INFO:tasks.ceph:Setting up mds nodes... 2026-02-26T03:43:43.129 INFO:tasks.ceph_client:Setting up client nodes... 2026-02-26T03:43:43.130 DEBUG:teuthology.orchestra.run.trial132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/ceph.client.0.keyring && sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2026-02-26T03:43:43.172 INFO:teuthology.orchestra.run.trial132.stdout:creating /etc/ceph/ceph.client.0.keyring 2026-02-26T03:43:43.183 INFO:tasks.ceph:Running mkfs on osd nodes... 2026-02-26T03:43:43.183 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@trial132.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'}} 2026-02-26T03:43:43.183 DEBUG:teuthology.orchestra.run.trial132:> sudo mkdir -p /var/lib/ceph/osd/ceph-0 2026-02-26T03:43:43.247 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2026-02-26T03:43:43.247 INFO:tasks.ceph:role: osd.0 2026-02-26T03:43:43.247 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@trial132.front.sepia.ceph.com 2026-02-26T03:43:43.248 DEBUG:teuthology.orchestra.run.trial132:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2026-02-26T03:43:43.312 INFO:teuthology.orchestra.run.trial132.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=45875200 blks 2026-02-26T03:43:43.312 INFO:teuthology.orchestra.run.trial132.stdout: = sectsz=4096 attr=2, projid32bit=1 2026-02-26T03:43:43.312 INFO:teuthology.orchestra.run.trial132.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2026-02-26T03:43:43.312 INFO:teuthology.orchestra.run.trial132.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2026-02-26T03:43:43.313 INFO:teuthology.orchestra.run.trial132.stdout:data = bsize=4096 blocks=183500800, imaxpct=25 2026-02-26T03:43:43.313 INFO:teuthology.orchestra.run.trial132.stdout: = sunit=0 swidth=0 blks 2026-02-26T03:43:43.313 INFO:teuthology.orchestra.run.trial132.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2026-02-26T03:43:43.313 INFO:teuthology.orchestra.run.trial132.stdout:log =internal log bsize=4096 blocks=89600, version=2 2026-02-26T03:43:43.313 INFO:teuthology.orchestra.run.trial132.stdout: = sectsz=4096 sunit=1 blks, lazy-count=1 2026-02-26T03:43:43.313 INFO:teuthology.orchestra.run.trial132.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2026-02-26T03:43:43.885 INFO:teuthology.orchestra.run.trial132.stdout:Discarding blocks...Done. 2026-02-26T03:43:43.887 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@trial132.front.sepia.ceph.com -o noatime 2026-02-26T03:43:43.887 DEBUG:teuthology.orchestra.run.trial132:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-0 2026-02-26T03:43:43.956 DEBUG:teuthology.orchestra.run.trial132:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-0 2026-02-26T03:43:44.022 DEBUG:teuthology.orchestra.run.trial132:> sudo mkdir -p /var/lib/ceph/osd/ceph-1 2026-02-26T03:43:44.086 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2026-02-26T03:43:44.086 INFO:tasks.ceph:role: osd.1 2026-02-26T03:43:44.087 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@trial132.front.sepia.ceph.com 2026-02-26T03:43:44.087 DEBUG:teuthology.orchestra.run.trial132:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2026-02-26T03:43:44.152 INFO:teuthology.orchestra.run.trial132.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=45875200 blks 2026-02-26T03:43:44.152 INFO:teuthology.orchestra.run.trial132.stdout: = sectsz=4096 attr=2, projid32bit=1 2026-02-26T03:43:44.152 INFO:teuthology.orchestra.run.trial132.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2026-02-26T03:43:44.152 INFO:teuthology.orchestra.run.trial132.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2026-02-26T03:43:44.152 INFO:teuthology.orchestra.run.trial132.stdout:data = bsize=4096 blocks=183500800, imaxpct=25 2026-02-26T03:43:44.153 INFO:teuthology.orchestra.run.trial132.stdout: = sunit=0 swidth=0 blks 2026-02-26T03:43:44.153 INFO:teuthology.orchestra.run.trial132.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2026-02-26T03:43:44.153 INFO:teuthology.orchestra.run.trial132.stdout:log =internal log bsize=4096 blocks=89600, version=2 2026-02-26T03:43:44.153 INFO:teuthology.orchestra.run.trial132.stdout: = sectsz=4096 sunit=1 blks, lazy-count=1 2026-02-26T03:43:44.153 INFO:teuthology.orchestra.run.trial132.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2026-02-26T03:43:44.712 INFO:teuthology.orchestra.run.trial132.stdout:Discarding blocks...Done. 2026-02-26T03:43:44.714 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@trial132.front.sepia.ceph.com -o noatime 2026-02-26T03:43:44.714 DEBUG:teuthology.orchestra.run.trial132:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-1 2026-02-26T03:43:44.783 DEBUG:teuthology.orchestra.run.trial132:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-1 2026-02-26T03:43:44.850 DEBUG:teuthology.orchestra.run.trial132:> sudo mkdir -p /var/lib/ceph/osd/ceph-2 2026-02-26T03:43:44.914 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2026-02-26T03:43:44.914 INFO:tasks.ceph:role: osd.2 2026-02-26T03:43:44.914 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@trial132.front.sepia.ceph.com 2026-02-26T03:43:44.914 DEBUG:teuthology.orchestra.run.trial132:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2026-02-26T03:43:44.979 INFO:teuthology.orchestra.run.trial132.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=45875200 blks 2026-02-26T03:43:44.979 INFO:teuthology.orchestra.run.trial132.stdout: = sectsz=4096 attr=2, projid32bit=1 2026-02-26T03:43:44.979 INFO:teuthology.orchestra.run.trial132.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2026-02-26T03:43:44.979 INFO:teuthology.orchestra.run.trial132.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2026-02-26T03:43:44.979 INFO:teuthology.orchestra.run.trial132.stdout:data = bsize=4096 blocks=183500800, imaxpct=25 2026-02-26T03:43:44.979 INFO:teuthology.orchestra.run.trial132.stdout: = sunit=0 swidth=0 blks 2026-02-26T03:43:44.980 INFO:teuthology.orchestra.run.trial132.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2026-02-26T03:43:44.980 INFO:teuthology.orchestra.run.trial132.stdout:log =internal log bsize=4096 blocks=89600, version=2 2026-02-26T03:43:44.980 INFO:teuthology.orchestra.run.trial132.stdout: = sectsz=4096 sunit=1 blks, lazy-count=1 2026-02-26T03:43:44.980 INFO:teuthology.orchestra.run.trial132.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2026-02-26T03:43:45.536 INFO:teuthology.orchestra.run.trial132.stdout:Discarding blocks...Done. 2026-02-26T03:43:45.538 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@trial132.front.sepia.ceph.com -o noatime 2026-02-26T03:43:45.538 DEBUG:teuthology.orchestra.run.trial132:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-2 2026-02-26T03:43:45.618 DEBUG:teuthology.orchestra.run.trial132:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-2 2026-02-26T03:43:45.686 DEBUG:teuthology.orchestra.run.trial132:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/ceph.monmap 2026-02-26T03:43:45.837 INFO:teuthology.orchestra.run.trial132.stderr:INFO  osd - OSD::main: parsing early config 2026-02-26T03:43:45.893 INFO:teuthology.orchestra.run.trial132.stderr:INFO 2026-02-26 03:43:45,896 seastar - Reactor backend: linux-aio 2026-02-26T03:43:45.908 INFO:teuthology.orchestra.run.trial132.stderr:INFO 2026-02-26 03:43:45,910 [shard 0:main] io - IO queue was unable to find a suitable maximum request length, the search was cut-off early at: 16MB 2026-02-26T03:43:45.908 INFO:teuthology.orchestra.run.trial132.stderr:INFO 2026-02-26 03:43:45,910 [shard 0:main] io - IO queue was unable to find a suitable maximum request length, the search was cut-off early at: 16MB 2026-02-26T03:43:45.910 INFO:teuthology.orchestra.run.trial132.stderr:==39439==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2026-02-26T03:43:45.966 INFO:teuthology.orchestra.run.trial132.stderr:INFO 2026-02-26 03:43:45,968 [shard 0:main] osd - bypassing the config fetch due to --no-mon-config 2026-02-26T03:43:45.966 INFO:teuthology.orchestra.run.trial132.stderr:INFO 2026-02-26 03:43:45,968 [shard 0:main] osd - Configure option_name crimson_reactor_task_quota_ms with value : 0.500000 2026-02-26T03:43:45.966 INFO:teuthology.orchestra.run.trial132.stderr:INFO 2026-02-26 03:43:45,968 [shard 0:main] osd - Configure option_name crimson_reactor_idle_poll_time_us with value : 200 2026-02-26T03:43:45.966 INFO:teuthology.orchestra.run.trial132.stderr:INFO 2026-02-26 03:43:45,968 [shard 0:main] osd - get_early_config: set --thread-affinity 0 --smp 4 2026-02-26T03:43:45.966 INFO:teuthology.orchestra.run.trial132.stderr:DEBUG 2026-02-26 03:43:45,968 [shard 0:main] seastar - reactor::drain 2026-02-26T03:43:48.015 INFO:teuthology.orchestra.run.trial132.stderr:----------------------------------------------------- 2026-02-26T03:43:48.015 INFO:teuthology.orchestra.run.trial132.stderr:Suppressions used: 2026-02-26T03:43:48.015 INFO:teuthology.orchestra.run.trial132.stderr: count bytes template 2026-02-26T03:43:48.015 INFO:teuthology.orchestra.run.trial132.stderr: 1 8 MallocExtension::Register 2026-02-26T03:43:48.015 INFO:teuthology.orchestra.run.trial132.stderr:----------------------------------------------------- 2026-02-26T03:43:48.015 INFO:teuthology.orchestra.run.trial132.stderr: 2026-02-26T03:43:48.034 INFO:teuthology.orchestra.run.trial132.stderr:INFO  osd - OSD::main: early config parsed successfully 2026-02-26T03:43:48.034 INFO:teuthology.orchestra.run.trial132.stderr:INFO  osd - OSD::main: initializing seastar app_template 2026-02-26T03:43:48.082 INFO:teuthology.orchestra.run.trial132.stderr:INFO  osd - OSD::main: registering CLI options 2026-02-26T03:43:48.082 INFO:teuthology.orchestra.run.trial132.stderr:INFO  osd - OSD::main: entering seastar runtime 2026-02-26T03:43:48.088 INFO:teuthology.orchestra.run.trial132.stderr:INFO 2026-02-26 03:43:48,091 seastar - Reactor backend: linux-aio 2026-02-26T03:43:48.100 INFO:teuthology.orchestra.run.trial132.stderr:INFO 2026-02-26 03:43:48,102 [shard 2:main] io - IO queue was unable to find a suitable maximum request length, the search was cut-off early at: 16MB 2026-02-26T03:43:48.100 INFO:teuthology.orchestra.run.trial132.stderr:INFO 2026-02-26 03:43:48,102 [shard 2:main] io - IO queue was unable to find a suitable maximum request length, the search was cut-off early at: 16MB 2026-02-26T03:43:48.104 INFO:teuthology.orchestra.run.trial132.stderr:==39437==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2026-02-26T03:43:48.104 INFO:teuthology.orchestra.run.trial132.stderr:INFO 2026-02-26 03:43:48,106 [shard 0:main] osd - OSD::main: seastar runtime started 2026-02-26T03:43:48.161 INFO:teuthology.orchestra.run.trial132.stderr:DEBUG 2026-02-26 03:43:48,164 [shard 0:main] osd - OSD::main: initializing logger output 2026-02-26T03:43:48.243 INFO:teuthology.orchestra.run.trial132.stderr:Reactor stalled for 32 ms on shard 3, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0x9e28178 0x9e28461 0x9e88dff 0x9e895f2 0x9e89860 0x9e899a8 0x9e89e67 /lib64/libc.so.6+0x3fc2f /lib64/libasan.so.8+0xeab96 0xe1d2ce 0xe1d689 0x9d258d6 0x9d25a92 0x9d25bfb 0x9d25d5c 0x9d26250 0x9d42adb 0x9d477c3 0x9d07d91 0x9d06d91 0x7bea2f1 0x7bf3e60 0x7a431c9 0x7a807bd 0x711f05d 0x6fa06ea 0x6fa44eb 0x6faa095 0x7419054 0x743ba5c 0x728f369 0x728f461 0x736153a 0x73615f3 0x73617aa 0x743badd 0x728f4de 0x728f563 0x72bb301 0x7423ee4 0x7423f69 0x7424047 0x72a6c86 0x73b49a4 0x743027a 0x74302f5 0x7430370 0x74f3b82 0x75ebd86 0x9ee2cc7 0x9e36f9c 0x9f10737 0x9f78f1a 0x9f939aa 0x9f94026 0x26ea5d3 0xa5a1f30 /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-26T03:43:48.243 INFO:teuthology.orchestra.run.trial132.stderr:Reactor stalled for 32 ms on shard 2, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0x9e28178 0x9e28461 0x9e88dff 0x9e895f2 0x9e89860 0x9e899a8 0x9e89e67 /lib64/libc.so.6+0x3fc2f 0x9c513a8 0x9c51a44 0x9c59eeb 0x9c5a5cb 0x9d43118 0x9d477c3 0x9d07d91 0x9d06d91 0x7bea2f1 0x7bf3e60 0x7a431c9 0x7a807bd 0x711f05d 0x6fa06ea 0x6fa44eb 0x6faa095 0x7419054 0x743ba5c 0x728f369 0x728f461 0x736153a 0x73615f3 0x73617aa 0x743badd 0x728f4de 0x728f563 0x72bb301 0x7423ee4 0x7423f69 0x7424047 0x72a6c86 0x73b49a4 0x743027a 0x74302f5 0x7430370 0x74f3b82 0x75ebd86 0x9ee2cc7 0x9e36f9c 0x9f10737 0x9f78f1a 0x9f939aa 0x9f94026 0x26ea5d3 0xa5a1f30 /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-26T03:43:48.243 INFO:teuthology.orchestra.run.trial132.stderr:Reactor stalled for 32 ms on shard 1, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0x9e28178 0x9e28461 0x9e88dff 0x9e895f2 0x9e89860 0x9e899a8 0x9e89e67 /lib64/libc.so.6+0x3fc2f 0x9c50b04 0x9c50f59 0x9c511cc 0x9c51482 0x9c51f31 0x9c5242f 0x9c52a5f 0x9c52bc9 0x9d32f39 0x9d45d3c 0x9d07d91 0x9d06d91 0x7bea2f1 0x7bf3e60 0x7a431c9 0x7a807bd 0x711f05d 0x6fa06ea 0x6fa44eb 0x6faa095 0x7419054 0x743ba5c 0x728f369 0x728f461 0x736153a 0x73615f3 0x73617aa 0x743badd 0x728f4de 0x728f563 0x72bb301 0x7423ee4 0x7423f69 0x7424047 0x72a6c86 0x73b49a4 0x743027a 0x74302f5 0x7430370 0x74f3b82 0x75ebd86 0x9ee2cc7 0x9e36f9c 0x9f10737 0x9f78f1a 0x9f939aa 0x9f94026 0x26ea5d3 0xa5a1f30 /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-26T03:43:48.276 INFO:teuthology.orchestra.run.trial132.stderr:Reactor stalled for 65 ms on shard 0, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0x9e28178 0x9e28461 0x9e88dff 0x9e895f2 0x9e89860 0x9e899a8 0x9e89e67 /lib64/libc.so.6+0x3fc2f 0x9d49252 0x78947dd 0x7beb047 0x7bf3e60 0x7a431c9 0x7a807bd 0x711f05d 0x6fa14d6 0x755a4ec 0x755a640 0x9ee2cc7 0x9e36f9c 0x9f10737 0x9f78f1a 0x9f7a738 0x9bf83c1 0x9bf8bfc 0xecf14d /lib64/libc.so.6+0x2a60f /lib64/libc.so.6+0x2a6bf 0xb46f44 2026-02-26T03:43:48.317 INFO:teuthology.orchestra.run.trial132.stderr:Reactor stalled for 65 ms on shard 3, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0x9e28178 0x9e28461 0x9e88dff 0x9e895f2 0x9e89860 0x9e899a8 0x9e89e67 /lib64/libc.so.6+0x3fc2f /lib64/libc.so.6+0xff18e /lib64/libasan.so.8+0x470c3 /lib64/libstdc++.so.6+0xd0f6d /lib64/libstdc++.so.6+0x1138d1 /lib64/libstdc++.so.6+0x113d17 /lib64/libstdc++.so.6+0x111a9c /lib64/libstdc++.so.6+0x139f62 0xa62c4dc 0xe73a5e 0x84a0ac6 0x7e20390 0x7e2059e 0x7a8082a 0x711f05d 0x6fa14d6 0x755a4ec 0x755a640 0x9ee2cc7 0x9e36f9c 0x9f10737 0x9f78f1a 0x9f939aa 0x9f94026 0x26ea5d3 0xa5a1f30 /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-26T03:43:48.317 INFO:teuthology.orchestra.run.trial132.stderr:kernel callstack: 0xffffffffffffff80 0xffffffffbb79d006 0xffffffffbc35d26a 0xffffffffbbab90d6 0xffffffffbba8f8da 0xffffffffbc34bf8c 0xffffffffbc40012b 2026-02-26T03:43:48.345 INFO:teuthology.orchestra.run.trial132.stderr:Reactor stalled for 65 ms on shard 0, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0x9e28178 0x9e28461 0x9e88dff 0x9e895f2 0x9e89860 0x9e899a8 0x9e89e67 /lib64/libc.so.6+0x3fc2f 0x2a55954 0x7d837b9 0x7d9fd5e 0x7e419e3 0x7dc12dd 0x7877bf3 0x78788e8 0x79adc2a 0x7836cfc 0x78b1d9e 0x78fa531 0x78fa7aa 0x799b982 0x799bae1 0x79add8d 0x7868874 0x798a5f1 0x798a680 0x798a724 0x73e442a 0x7404ce9 0x740998a 0x754d7e2 0x9ee2cc7 0x9e36f9c 0x9f10737 0x9f78f1a 0x9f7a738 0x9bf83c1 0x9bf8bfc 0xecf14d /lib64/libc.so.6+0x2a60f /lib64/libc.so.6+0x2a6bf 0xb46f44 2026-02-26T03:43:48.384 INFO:teuthology.orchestra.run.trial132.stderr:Reactor stalled for 65 ms on shard 1, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0x9e28178 0x9e28461 0x9e88dff 0x9e895f2 0x9e89860 0x9e899a8 0x9e89e67 /lib64/libc.so.6+0x3fc2f /lib64/ld-linux-x86-64.so.2+0x16ebc /lib64/libasan.so.8+0xc65b1 /lib64/libasan.so.8+0x15197 0x2a55a3f 0x7d837b9 0x7d9fd5e 0x7e419e3 0x7dc12dd 0x7877bf3 0x78788e8 0x79adc2a 0x7836cfc 0x78b1d9e 0x78fa531 0x78fa7aa 0x799b982 0x799bae1 0x79add8d 0x7868874 0x798a5f1 0x798a680 0x798a724 0x73e442a 0x7404ce9 0x740998a 0x754d7e2 0x9ee2cc7 0x9e36f9c 0x9f10737 0x9f78f1a 0x9f939aa 0x9f94026 0x26ea5d3 0xa5a1f30 /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-26T03:43:48.385 INFO:teuthology.orchestra.run.trial132.stderr:Reactor stalled for 65 ms on shard 3, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0x9e28178 0x9e28461 0x9e88dff 0x9e895f2 0x9e89860 0x9e899a8 0x9e89e67 /lib64/libc.so.6+0x3fc2f 0x2a00bea 0x2a559e0 0x7d837b9 0x7d9fd5e 0x7e419e3 0x7dc12dd 0x7877bf3 0x78788e8 0x79adc2a 0x7836cfc 0x78b1d9e 0x78fa531 0x78fa7aa 0x799b982 0x799bae1 0x79add8d 0x7868874 0x798a5f1 0x798a680 0x798a724 0x73e442a 0x7404ce9 0x740998a 0x754d7e2 0x9ee2cc7 0x9e36f9c 0x9f10737 0x9f78f1a 0x9f939aa 0x9f94026 0x26ea5d3 0xa5a1f30 /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-26T03:43:48.385 INFO:teuthology.orchestra.run.trial132.stderr:Reactor stalled for 65 ms on shard 2, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0x9e28178 0x9e28461 0x9e88dff 0x9e895f2 0x9e89860 0x9e899a8 0x9e89e67 /lib64/libc.so.6+0x3fc2f /lib64/libc.so.6+0x8cc0c /lib64/libasan.so.8+0x10069c /lib64/libasan.so.8+0xc65b9 /lib64/libasan.so.8+0x15197 0x2a55a3f 0x7d837b9 0x7d9fd5e 0x7e419e3 0x7dc12dd 0x7877bf3 0x78788e8 0x79adc2a 0x7836cfc 0x78b1d9e 0x78fa531 0x78fa7aa 0x799b982 0x799bae1 0x79add8d 0x7868874 0x798a5f1 0x798a680 0x798a724 0x73e442a 0x7404ce9 0x740998a 0x754d7e2 0x9ee2cc7 0x9e36f9c 0x9f10737 0x9f78f1a 0x9f939aa 0x9f94026 0x26ea5d3 0xa5a1f30 /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-26T03:43:48.508 INFO:teuthology.orchestra.run.trial132.stderr:Reactor stalled for 32 ms on shard 1, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0x9e28178 0x9e28461 0x9e88dff 0x9e895f2 0x9e89860 0x9e899a8 0x9e89e67 /lib64/libc.so.6+0x3fc2f 0x7b099fe 0x7bea776 0x7bf3e60 0x7a431c9 0x7a807bd 0x711f05d 0x6f43702 0x6f45a6a 0x7483f52 0x72a690f 0x72a6a11 0x72a6b09 0x7360706 0x73607bf 0x7360972 0x7360aaa 0x7360b25 0x7360baa 0x7360d05 0x7423961 0x7483fdb 0x728dda4 0x72a6c86 0x73b49a4 0x743027a 0x74302f5 0x7430370 0x74f3b82 0x75ebd86 0x9ee2cc7 0x9e36f9c 0x9f10737 0x9f78f1a 0x9f939aa 0x9f94026 0x26ea5d3 0xa5a1f30 /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-26T03:43:48.508 INFO:teuthology.orchestra.run.trial132.stderr:Reactor stalled for 32 ms on shard 3, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0x9e28178 0x9e28461 0x9e88dff 0x9e895f2 0x9e89860 0x9e899a8 0x9e89e67 /lib64/libc.so.6+0x3fc2f 0x9d340e6 0x9d34209 0x9d44360 0x9d477c3 0x9d07d91 0x9d06d91 0x7bea2f1 0x7bf3e60 0x7a431c9 0x7a807bd 0x711f05d 0x6f43702 0x6f45a6a 0x7483f52 0x72a690f 0x72a6a11 0x72a6b09 0x7360706 0x73607bf 0x7360972 0x7360aaa 0x7360b25 0x7360baa 0x7360d05 0x7423961 0x7483fdb 0x728dda4 0x72a6c86 0x73b49a4 0x743027a 0x74302f5 0x7430370 0x74f3b82 0x75ebd86 0x9ee2cc7 0x9e36f9c 0x9f10737 0x9f78f1a 0x9f939aa 0x9f94026 0x26ea5d3 0xa5a1f30 /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-26T03:43:48.508 INFO:teuthology.orchestra.run.trial132.stderr:Reactor stalled for 32 ms on shard 2, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0x9e28178 0x9e28461 0x9e88dff 0x9e895f2 0x9e89860 0x9e899a8 0x9e89e67 /lib64/libc.so.6+0x3fc2f 0x9c0739a 0x9d1b543 0x9d47a30 0x9d07d91 0x9d06d91 0x7bea2f1 0x7bf3e60 0x7a431c9 0x7a807bd 0x711f05d 0x6f43702 0x6f45a6a 0x7483f52 0x72a690f 0x72a6a11 0x72a6b09 0x7360706 0x73607bf 0x7360972 0x7360aaa 0x7360b25 0x7360baa 0x7360d05 0x7423961 0x7483fdb 0x728dda4 0x72a6c86 0x73b49a4 0x743027a 0x74302f5 0x7430370 0x74f3b82 0x75ebd86 0x9ee2cc7 0x9e36f9c 0x9f10737 0x9f78f1a 0x9f939aa 0x9f94026 0x26ea5d3 0xa5a1f30 /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-26T03:43:48.509 INFO:teuthology.orchestra.run.trial132.stderr:kernel callstack: 0xffffffffffffff80 0xffffffffbb9cc8c9 0xffffffffbb9d4bab 0xffffffffbb9d500b 0xffffffffbb6910d2 0xffffffffbc350451 0xffffffffbc400ba2 2026-02-26T03:43:48.591 INFO:teuthology.orchestra.run.trial132.stderr:Reactor stalled for 65 ms on shard 0, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0x9e28178 0x9e28461 0x9e88dff 0x9e895f2 0x9e89860 0x9e899a8 0x9e89e67 /lib64/libc.so.6+0x3fc2f 0x2a55388 0x7d837f4 0x7d9fd5e 0x7e419e3 0x7dc12dd 0x7877bf3 0x78788e8 0x79adc2a 0x7836cfc 0x78b1d9e 0x78fa531 0x78fa7aa 0x799b982 0x799bae1 0x79add8d 0x7868874 0x798a5f1 0x798a680 0x798a724 0x73e442a 0x7404ce9 0x740998a 0x754d7e2 0x9ee2cc7 0x9e36f9c 0x9f10737 0x9f78f1a 0x9f7a738 0x9bf83c1 0x9bf8bfc 0xecf14d /lib64/libc.so.6+0x2a60f /lib64/libc.so.6+0x2a6bf 0xb46f44 2026-02-26T03:43:48.605 INFO:teuthology.orchestra.run.trial132.stderr:Reactor stalled for 65 ms on shard 3, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0x9e28178 0x9e28461 0x9e88dff 0x9e895f2 0x9e89860 0x9e899a8 0x9e89e67 /lib64/libc.so.6+0x3fc2f /lib64/libasan.so.8+0x151b5 0x2a55a3f 0x7d837b9 0x7d9fd5e 0x7e419e3 0x7dc12dd 0x7877bf3 0x78788e8 0x79adc2a 0x7836cfc 0x78b1d9e 0x78fa531 0x78fa7aa 0x799b982 0x799bae1 0x79add8d 0x7868874 0x798a5f1 0x798a680 0x798a724 0x73e442a 0x7404ce9 0x740998a 0x754d7e2 0x9ee2cc7 0x9e36f9c 0x9f10737 0x9f78f1a 0x9f939aa 0x9f94026 0x26ea5d3 0xa5a1f30 /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-26T03:43:48.606 INFO:teuthology.orchestra.run.trial132.stderr:Reactor stalled for 65 ms on shard 1, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0x9e28178 0x9e28461 0x9e88dff 0x9e895f2 0x9e89860 0x9e899a8 0x9e89e67 /lib64/libc.so.6+0x3fc2f /lib64/ld-linux-x86-64.so.2+0x16ed7 /lib64/libasan.so.8+0xc65b1 /lib64/libasan.so.8+0x15197 0x2a55a3f 0x7d837b9 0x7d9fd5e 0x7e419e3 0x7dc12dd 0x7877bf3 0x78788e8 0x79adc2a 0x7836cfc 0x78b1d9e 0x78fa531 0x78fa7aa 0x799b982 0x799bae1 0x79add8d 0x7868874 0x798a5f1 0x798a680 0x798a724 0x73e442a 0x7404ce9 0x740998a 0x754d7e2 0x9ee2cc7 0x9e36f9c 0x9f10737 0x9f78f1a 0x9f939aa 0x9f94026 0x26ea5d3 0xa5a1f30 /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-26T03:43:48.606 INFO:teuthology.orchestra.run.trial132.stderr:Reactor stalled for 65 ms on shard 2, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0x9e28178 0x9e28461 0x9e88dff 0x9e895f2 0x9e89860 0x9e899a8 0x9e89e67 /lib64/libc.so.6+0x3fc2f 0x2a55901 0x7d837b9 0x7d9fd5e 0x7e419e3 0x7dc12dd 0x7877bf3 0x78788e8 0x79adc2a 0x7836cfc 0x78b1d9e 0x78fa531 0x78fa7aa 0x799b982 0x799bae1 0x79add8d 0x7868874 0x798a5f1 0x798a680 0x798a724 0x73e442a 0x7404ce9 0x740998a 0x754d7e2 0x9ee2cc7 0x9e36f9c 0x9f10737 0x9f78f1a 0x9f939aa 0x9f94026 0x26ea5d3 0xa5a1f30 /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-26T03:43:50.756 INFO:teuthology.orchestra.run.trial132.stderr:----------------------------------------------------- 2026-02-26T03:43:50.756 INFO:teuthology.orchestra.run.trial132.stderr:Suppressions used: 2026-02-26T03:43:50.756 INFO:teuthology.orchestra.run.trial132.stderr: count bytes template 2026-02-26T03:43:50.757 INFO:teuthology.orchestra.run.trial132.stderr: 1 8 MallocExtension::Register 2026-02-26T03:43:50.757 INFO:teuthology.orchestra.run.trial132.stderr:----------------------------------------------------- 2026-02-26T03:43:50.757 INFO:teuthology.orchestra.run.trial132.stderr: 2026-02-26T03:43:50.791 DEBUG:teuthology.orchestra.run.trial132:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2026-02-26T03:43:50.816 DEBUG:teuthology.orchestra.run.trial132:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/ceph.monmap 2026-02-26T03:43:50.968 INFO:teuthology.orchestra.run.trial132.stderr:INFO  osd - OSD::main: parsing early config 2026-02-26T03:43:51.021 INFO:teuthology.orchestra.run.trial132.stderr:INFO 2026-02-26 03:43:51,023 seastar - Reactor backend: linux-aio 2026-02-26T03:43:51.032 INFO:teuthology.orchestra.run.trial132.stderr:INFO 2026-02-26 03:43:51,034 [shard 0:main] io - IO queue was unable to find a suitable maximum request length, the search was cut-off early at: 16MB 2026-02-26T03:43:51.032 INFO:teuthology.orchestra.run.trial132.stderr:INFO 2026-02-26 03:43:51,034 [shard 0:main] io - IO queue was unable to find a suitable maximum request length, the search was cut-off early at: 16MB 2026-02-26T03:43:51.034 INFO:teuthology.orchestra.run.trial132.stderr:==39514==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2026-02-26T03:43:51.090 INFO:teuthology.orchestra.run.trial132.stderr:INFO 2026-02-26 03:43:51,092 [shard 0:main] osd - bypassing the config fetch due to --no-mon-config 2026-02-26T03:43:51.090 INFO:teuthology.orchestra.run.trial132.stderr:INFO 2026-02-26 03:43:51,092 [shard 0:main] osd - Configure option_name crimson_reactor_task_quota_ms with value : 0.500000 2026-02-26T03:43:51.090 INFO:teuthology.orchestra.run.trial132.stderr:INFO 2026-02-26 03:43:51,092 [shard 0:main] osd - Configure option_name crimson_reactor_idle_poll_time_us with value : 200 2026-02-26T03:43:51.090 INFO:teuthology.orchestra.run.trial132.stderr:INFO 2026-02-26 03:43:51,092 [shard 0:main] osd - get_early_config: set --thread-affinity 0 --smp 4 2026-02-26T03:43:51.090 INFO:teuthology.orchestra.run.trial132.stderr:DEBUG 2026-02-26 03:43:51,092 [shard 0:main] seastar - reactor::drain 2026-02-26T03:43:53.131 INFO:teuthology.orchestra.run.trial132.stderr:----------------------------------------------------- 2026-02-26T03:43:53.131 INFO:teuthology.orchestra.run.trial132.stderr:Suppressions used: 2026-02-26T03:43:53.132 INFO:teuthology.orchestra.run.trial132.stderr: count bytes template 2026-02-26T03:43:53.132 INFO:teuthology.orchestra.run.trial132.stderr: 1 8 MallocExtension::Register 2026-02-26T03:43:53.132 INFO:teuthology.orchestra.run.trial132.stderr:----------------------------------------------------- 2026-02-26T03:43:53.132 INFO:teuthology.orchestra.run.trial132.stderr: 2026-02-26T03:43:53.151 INFO:teuthology.orchestra.run.trial132.stderr:INFO  osd - OSD::main: early config parsed successfully 2026-02-26T03:43:53.151 INFO:teuthology.orchestra.run.trial132.stderr:INFO  osd - OSD::main: initializing seastar app_template 2026-02-26T03:43:53.197 INFO:teuthology.orchestra.run.trial132.stderr:INFO  osd - OSD::main: registering CLI options 2026-02-26T03:43:53.197 INFO:teuthology.orchestra.run.trial132.stderr:INFO  osd - OSD::main: entering seastar runtime 2026-02-26T03:43:53.203 INFO:teuthology.orchestra.run.trial132.stderr:INFO 2026-02-26 03:43:53,206 seastar - Reactor backend: linux-aio 2026-02-26T03:43:53.217 INFO:teuthology.orchestra.run.trial132.stderr:INFO 2026-02-26 03:43:53,219 [shard 3:main] io - IO queue was unable to find a suitable maximum request length, the search was cut-off early at: 16MB 2026-02-26T03:43:53.217 INFO:teuthology.orchestra.run.trial132.stderr:INFO 2026-02-26 03:43:53,219 [shard 3:main] io - IO queue was unable to find a suitable maximum request length, the search was cut-off early at: 16MB 2026-02-26T03:43:53.221 INFO:teuthology.orchestra.run.trial132.stderr:==39512==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2026-02-26T03:43:53.221 INFO:teuthology.orchestra.run.trial132.stderr:INFO 2026-02-26 03:43:53,223 [shard 0:main] osd - OSD::main: seastar runtime started 2026-02-26T03:43:53.278 INFO:teuthology.orchestra.run.trial132.stderr:DEBUG 2026-02-26 03:43:53,281 [shard 0:main] osd - OSD::main: initializing logger output 2026-02-26T03:43:53.359 INFO:teuthology.orchestra.run.trial132.stderr:Reactor stalled for 32 ms on shard 3, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0x9e28178 0x9e28461 0x9e88dff 0x9e895f2 0x9e89860 0x9e899a8 0x9e89e67 /lib64/libc.so.6+0x3fc2f /lib64/libasan.so.8+0x157f2 0x10908ef 0xf2e0dc 0x9c50a8f 0x9c50f59 0x9c511cc 0x9c51482 0x9c51f31 0x9c5242f 0x9c52a5f 0x9c52bc9 0x9d32f39 0x9d45d3c 0x9d07d91 0x9d06d91 0x7bea2f1 0x7bf3e60 0x7a431c9 0x7a807bd 0x711f05d 0x6fa06ea 0x6fa44eb 0x6faa095 0x7419054 0x743ba5c 0x728f369 0x728f461 0x736153a 0x73615f3 0x73617aa 0x743badd 0x728f4de 0x728f563 0x72bb301 0x7423ee4 0x7423f69 0x7424047 0x72a6c86 0x73b49a4 0x743027a 0x74302f5 0x7430370 0x74f3b82 0x75ebd86 0x9ee2cc7 0x9e36f9c 0x9f10737 0x9f78f1a 0x9f939aa 0x9f94026 0x26ea5d3 0xa5a1f30 /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-26T03:43:53.359 INFO:teuthology.orchestra.run.trial132.stderr:Reactor stalled for 32 ms on shard 2, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0x9e28178 0x9e28461 0x9e88dff 0x9e895f2 0x9e89860 0x9e899a8 0x9e89e67 /lib64/libc.so.6+0x3fc2f 0x9d0c572 0x9d0c66c 0x9d3ea77 0x9d3f21a 0x9d49b10 0x78947dd 0x7be9c23 0x7bf3e60 0x7a431c9 0x7a807bd 0x711f05d 0x6fa06ea 0x6fa44eb 0x6faa095 0x7419054 0x743ba5c 0x728f369 0x728f461 0x736153a 0x73615f3 0x73617aa 0x743badd 0x728f4de 0x728f563 0x72bb301 0x7423ee4 0x7423f69 0x7424047 0x72a6c86 0x73b49a4 0x743027a 0x74302f5 0x7430370 0x74f3b82 0x75ebd86 0x9ee2cc7 0x9e36f9c 0x9f10737 0x9f78f1a 0x9f939aa 0x9f94026 0x26ea5d3 0xa5a1f30 /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-26T03:43:53.359 INFO:teuthology.orchestra.run.trial132.stderr:Reactor stalled for 32 ms on shard 1, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0x9e28178 0x9e28461 0x9e88dff 0x9e895f2 0x9e89860 0x9e899a8 0x9e89e67 /lib64/libc.so.6+0x3fc2f /lib64/libasan.so.8+0x1586f 0x10908ef 0xf2e0dc 0x9d3994e 0x9d3ebdc 0x9d3f21a 0x9d496e7 0x78947dd 0x7bea228 0x7bf3e60 0x7a431c9 0x7a807bd 0x711f05d 0x6fa06ea 0x6fa44eb 0x6faa095 0x7419054 0x743ba5c 0x728f369 0x728f461 0x736153a 0x73615f3 0x73617aa 0x743badd 0x728f4de 0x728f563 0x72bb301 0x7423ee4 0x7423f69 0x7424047 0x72a6c86 0x73b49a4 0x743027a 0x74302f5 0x7430370 0x74f3b82 0x75ebd86 0x9ee2cc7 0x9e36f9c 0x9f10737 0x9f78f1a 0x9f939aa 0x9f94026 0x26ea5d3 0xa5a1f30 /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-26T03:43:53.391 INFO:teuthology.orchestra.run.trial132.stderr:Reactor stalled for 65 ms on shard 0, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0x9e28178 0x9e28461 0x9e88dff 0x9e895f2 0x9e89860 0x9e899a8 0x9e89e67 /lib64/libc.so.6+0x3fc2f 0x7d0e7e7 0x7b4a81c 0x7b4efff 0x7b66f2c 0x7b88e26 0x78b3526 0x6fa18b5 0x755a4ec 0x755a640 0x9ee2cc7 0x9e36f9c 0x9f10737 0x9f78f1a 0x9f7a738 0x9bf83c1 0x9bf8bfc 0xecf14d /lib64/libc.so.6+0x2a60f /lib64/libc.so.6+0x2a6bf 0xb46f44 2026-02-26T03:43:53.432 INFO:teuthology.orchestra.run.trial132.stderr:Reactor stalled for 65 ms on shard 1, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0x9e28178 0x9e28461 0x9e88dff 0x9e895f2 0x9e89860 0x9e899a8 0x9e89e67 /lib64/libc.so.6+0x3fc2f /lib64/libasan.so.8+0x361f /lib64/libasan.so.8+0x157c7 0x10908ef 0xf2e0dc 0x9c554b6 0x9c55df5 0x9d41dc5 0x9d44a61 0x9d477c3 0x9d07d91 0x9d06d91 0x8f5bc05 0x8f5ce1a 0x8f5ff8b 0x787cd7f 0x78b3782 0x6fa18b5 0x755a4ec 0x755a640 0x9ee2cc7 0x9e36f9c 0x9f10737 0x9f78f1a 0x9f939aa 0x9f94026 0x26ea5d3 0xa5a1f30 /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-26T03:43:53.432 INFO:teuthology.orchestra.run.trial132.stderr:Reactor stalled for 65 ms on shard 3, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0x9e28178 0x9e28461 0x9e88dff 0x9e895f2 0x9e89860 0x9e899a8 0x9e89e67 /lib64/libc.so.6+0x3fc2f /lib64/libasan.so.8+0xc65c2 /lib64/libasan.so.8+0x15197 0x316f517 0x9c511ba 0x9c51482 0x9c51f31 0x9c5242f 0x9c52a5f 0x9c52bc9 0x9d32f39 0x9d45d3c 0x9d07d91 0x9d06d91 0x7e61695 0x7e64490 0x8f5d61d 0x8f5ff8b 0x787cd7f 0x78b3782 0x6fa18b5 0x755a4ec 0x755a640 0x9ee2cc7 0x9e36f9c 0x9f10737 0x9f78f1a 0x9f939aa 0x9f94026 0x26ea5d3 0xa5a1f30 /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-26T03:43:53.458 INFO:teuthology.orchestra.run.trial132.stderr:Reactor stalled for 65 ms on shard 0, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0x9e28178 0x9e28461 0x9e88dff 0x9e895f2 0x9e89860 0x9e899a8 0x9e89e67 /lib64/libc.so.6+0x3fc2f 0x2a54d60 0x7d837e7 0x7d9fd5e 0x7e419e3 0x7dc12dd 0x7877bf3 0x78788e8 0x79adc2a 0x7836cfc 0x78b1d9e 0x78fa531 0x78fa7aa 0x799b982 0x799bae1 0x79add8d 0x7868874 0x798a5f1 0x798a680 0x798a724 0x73e442a 0x7404ce9 0x740998a 0x754d7e2 0x9ee2cc7 0x9e36f9c 0x9f10737 0x9f78f1a 0x9f7a738 0x9bf83c1 0x9bf8bfc 0xecf14d /lib64/libc.so.6+0x2a60f /lib64/libc.so.6+0x2a6bf 0xb46f44 2026-02-26T03:43:53.496 INFO:teuthology.orchestra.run.trial132.stderr:Reactor stalled for 65 ms on shard 2, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0x9e28178 0x9e28461 0x9e88dff 0x9e895f2 0x9e89860 0x9e899a8 0x9e89e67 /lib64/libc.so.6+0x3fc2f /lib64/libasan.so.8+0x129764 /lib64/libasan.so.8+0xc65d1 /lib64/libasan.so.8+0x15197 0x2a55a3f 0x7d837b9 0x7d9fd5e 0x7e419e3 0x7dc12dd 0x7877bf3 0x78788e8 0x79adc2a 0x7836cfc 0x78b1d9e 0x78fa531 0x78fa7aa 0x799b982 0x799bae1 0x79add8d 0x7868874 0x798a5f1 0x798a680 0x798a724 0x73e442a 0x7404ce9 0x740998a 0x754d7e2 0x9ee2cc7 0x9e36f9c 0x9f10737 0x9f78f1a 0x9f939aa 0x9f94026 0x26ea5d3 0xa5a1f30 /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-26T03:43:53.498 INFO:teuthology.orchestra.run.trial132.stderr:Reactor stalled for 65 ms on shard 3, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0x9e28178 0x9e28461 0x9e88dff 0x9e895f2 0x9e89860 0x9e899a8 0x9e89e67 /lib64/libc.so.6+0x3fc2f /lib64/libasan.so.8+0xc65b1 /lib64/libasan.so.8+0x15197 0x2a55a3f 0x7d837b9 0x7d9fd5e 0x7e419e3 0x7dc12dd 0x7877bf3 0x78788e8 0x79adc2a 0x7836cfc 0x78b1d9e 0x78fa531 0x78fa7aa 0x799b982 0x799bae1 0x79add8d 0x7868874 0x798a5f1 0x798a680 0x798a724 0x73e442a 0x7404ce9 0x740998a 0x754d7e2 0x9ee2cc7 0x9e36f9c 0x9f10737 0x9f78f1a 0x9f939aa 0x9f94026 0x26ea5d3 0xa5a1f30 /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-26T03:43:53.498 INFO:teuthology.orchestra.run.trial132.stderr:Reactor stalled for 65 ms on shard 1, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0x9e28178 0x9e28461 0x9e88dff 0x9e895f2 0x9e89860 0x9e899a8 0x9e89e67 /lib64/libc.so.6+0x3fc2f 0x2a55875 0x7d837b9 0x7d9fd5e 0x7e419e3 0x7dc12dd 0x7877bf3 0x78788e8 0x79adc2a 0x7836cfc 0x78b1d9e 0x78fa531 0x78fa7aa 0x799b982 0x799bae1 0x79add8d 0x7868874 0x798a5f1 0x798a680 0x798a724 0x73e442a 0x7404ce9 0x740998a 0x754d7e2 0x9ee2cc7 0x9e36f9c 0x9f10737 0x9f78f1a 0x9f939aa 0x9f94026 0x26ea5d3 0xa5a1f30 /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-26T03:43:53.620 INFO:teuthology.orchestra.run.trial132.stderr:Reactor stalled for 32 ms on shard 2, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0x9e28178 0x9e28461 0x9e88dff 0x9e895f2 0x9e89860 0x9e899a8 0x9e89e67 /lib64/libc.so.6+0x3fc2f /lib64/libasan.so.8+0x1657f 0xa5d7167 0x9d4877c 0x9d48963 0x9d48b2c 0x9d48bbf 0x9d49a04 0x78947dd 0x7be9c23 0x7bf3e60 0x7a431c9 0x7a807bd 0x711f05d 0x6f43702 0x6f45a6a 0x7483f52 0x72a690f 0x72a6a11 0x72a6b09 0x7360706 0x73607bf 0x7360972 0x7360aaa 0x7360b25 0x7360baa 0x7360d05 0x7423961 0x7483fdb 0x728dda4 0x72a6c86 0x73b49a4 0x743027a 0x74302f5 0x7430370 0x74f3b82 0x75ebd86 0x9ee2cc7 0x9e36f9c 0x9f10737 0x9f78f1a 0x9f939aa 0x9f94026 0x26ea5d3 0xa5a1f30 /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-26T03:43:53.620 INFO:teuthology.orchestra.run.trial132.stderr:Reactor stalled for 32 ms on shard 1, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0x9e28178 0x9e28461 0x9e88dff 0x9e895f2 0x9e89860 0x9e899a8 0x9e89e67 /lib64/libc.so.6+0x3fc2f 0x9b250da 0x9d46c67 0x9d07d91 0x9d06d91 0x7bea2f1 0x7bf3e60 0x7a431c9 0x7a807bd 0x711f05d 0x6f43702 0x6f45a6a 0x7483f52 0x72a690f 0x72a6a11 0x72a6b09 0x7360706 0x73607bf 0x7360972 0x7360aaa 0x7360b25 0x7360baa 0x7360d05 0x7423961 0x7483fdb 0x728dda4 0x72a6c86 0x73b49a4 0x743027a 0x74302f5 0x7430370 0x74f3b82 0x75ebd86 0x9ee2cc7 0x9e36f9c 0x9f10737 0x9f78f1a 0x9f939aa 0x9f94026 0x26ea5d3 0xa5a1f30 /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-26T03:43:53.620 INFO:teuthology.orchestra.run.trial132.stderr:Reactor stalled for 32 ms on shard 3, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0x9e28178 0x9e28461 0x9e88dff 0x9e895f2 0x9e89860 0x9e899a8 0x9e89e67 /lib64/libc.so.6+0x3fc2f /lib64/libc.so.6+0x8cc0c /lib64/libasan.so.8+0x10069c /lib64/libasan.so.8+0xc65b9 /lib64/libasan.so.8+0x15197 0x316f517 0x9c511ba 0x9c51482 0x9c51f31 0x9c5242f 0x9c52a5f 0x9c52bc9 0x9d32f39 0x9d45d3c 0x9d07d91 0x9d06d91 0x7bea2f1 0x7bf3e60 0x7a431c9 0x7a807bd 0x711f05d 0x6f43702 0x6f45a6a 0x7483f52 0x72a690f 0x72a6a11 0x72a6b09 0x7360706 0x73607bf 0x7360972 0x7360aaa 0x7360b25 0x7360baa 0x7360d05 0x7423961 0x7483fdb 0x728dda4 0x72a6c86 0x73b49a4 0x743027a 0x74302f5 0x7430370 0x74f3b82 0x75ebd86 0x9ee2cc7 0x9e36f9c 0x9f10737 0x9f78f1a 0x9f939aa 0x9f94026 0x26ea5d3 0xa5a1f30 /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-26T03:43:53.702 INFO:teuthology.orchestra.run.trial132.stderr:Reactor stalled for 65 ms on shard 0, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0x9e28178 0x9e28461 0x9e88dff 0x9e895f2 0x9e89860 0x9e899a8 0x9e89e67 /lib64/libc.so.6+0x3fc2f /lib64/libasan.so.8+0x1297e4 /lib64/libasan.so.8+0xc65d1 /lib64/libasan.so.8+0x15197 0x2a55a3f 0x7d837b9 0x7d9fd5e 0x7e419e3 0x7dc12dd 0x7877bf3 0x78788e8 0x79adc2a 0x7836cfc 0x78b1d9e 0x78fa531 0x78fa7aa 0x799b982 0x799bae1 0x79add8d 0x7868874 0x798a5f1 0x798a680 0x798a724 0x73e442a 0x7404ce9 0x740998a 0x754d7e2 0x9ee2cc7 0x9e36f9c 0x9f10737 0x9f78f1a 0x9f7a738 0x9bf83c1 0x9bf8bfc 0xecf14d /lib64/libc.so.6+0x2a60f /lib64/libc.so.6+0x2a6bf 0xb46f44 2026-02-26T03:43:53.716 INFO:teuthology.orchestra.run.trial132.stderr:Reactor stalled for 65 ms on shard 2, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0x9e28178 0x9e28461 0x9e88dff 0x9e895f2 0x9e89860 0x9e899a8 0x9e89e67 /lib64/libc.so.6+0x3fc2f /lib64/libc.so.6+0x8cc0c /lib64/libasan.so.8+0x10069c /lib64/libasan.so.8+0xc65b9 /lib64/libasan.so.8+0x15197 0x2a55a3f 0x7d837b9 0x7d9fd5e 0x7e419e3 0x7dc12dd 0x7877bf3 0x78788e8 0x79adc2a 0x7836cfc 0x78b1d9e 0x78fa531 0x78fa7aa 0x799b982 0x799bae1 0x79add8d 0x7868874 0x798a5f1 0x798a680 0x798a724 0x73e442a 0x7404ce9 0x740998a 0x754d7e2 0x9ee2cc7 0x9e36f9c 0x9f10737 0x9f78f1a 0x9f939aa 0x9f94026 0x26ea5d3 0xa5a1f30 /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-26T03:43:53.717 INFO:teuthology.orchestra.run.trial132.stderr:Reactor stalled for 65 ms on shard 3, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0x9e28178 0x9e28461 0x9e88dff 0x9e895f2 0x9e89860 0x9e899a8 0x9e89e67 /lib64/libc.so.6+0x3fc2f 0x2a00b9c 0x2a559e0 0x7d837b9 0x7d9fd5e 0x7e419e3 0x7dc12dd 0x7877bf3 0x78788e8 0x79adc2a 0x7836cfc 0x78b1d9e 0x78fa531 0x78fa7aa 0x799b982 0x799bae1 0x79add8d 0x7868874 0x798a5f1 0x798a680 0x798a724 0x73e442a 0x7404ce9 0x740998a 0x754d7e2 0x9ee2cc7 0x9e36f9c 0x9f10737 0x9f78f1a 0x9f939aa 0x9f94026 0x26ea5d3 0xa5a1f30 /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-26T03:43:53.717 INFO:teuthology.orchestra.run.trial132.stderr:Reactor stalled for 65 ms on shard 1, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0x9e28178 0x9e28461 0x9e88dff 0x9e895f2 0x9e89860 0x9e899a8 0x9e89e67 /lib64/libc.so.6+0x3fc2f 0x2a5530d 0x7d837f4 0x7d9fd5e 0x7e419e3 0x7dc12dd 0x7877bf3 0x78788e8 0x79adc2a 0x7836cfc 0x78b1d9e 0x78fa531 0x78fa7aa 0x799b982 0x799bae1 0x79add8d 0x7868874 0x798a5f1 0x798a680 0x798a724 0x73e442a 0x7404ce9 0x740998a 0x754d7e2 0x9ee2cc7 0x9e36f9c 0x9f10737 0x9f78f1a 0x9f939aa 0x9f94026 0x26ea5d3 0xa5a1f30 /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-26T03:43:55.866 INFO:teuthology.orchestra.run.trial132.stderr:----------------------------------------------------- 2026-02-26T03:43:55.867 INFO:teuthology.orchestra.run.trial132.stderr:Suppressions used: 2026-02-26T03:43:55.867 INFO:teuthology.orchestra.run.trial132.stderr: count bytes template 2026-02-26T03:43:55.867 INFO:teuthology.orchestra.run.trial132.stderr: 1 8 MallocExtension::Register 2026-02-26T03:43:55.867 INFO:teuthology.orchestra.run.trial132.stderr:----------------------------------------------------- 2026-02-26T03:43:55.867 INFO:teuthology.orchestra.run.trial132.stderr: 2026-02-26T03:43:55.902 DEBUG:teuthology.orchestra.run.trial132:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2026-02-26T03:43:55.927 DEBUG:teuthology.orchestra.run.trial132:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/ceph.monmap 2026-02-26T03:43:56.078 INFO:teuthology.orchestra.run.trial132.stderr:INFO  osd - OSD::main: parsing early config 2026-02-26T03:43:56.131 INFO:teuthology.orchestra.run.trial132.stderr:INFO 2026-02-26 03:43:56,134 seastar - Reactor backend: linux-aio 2026-02-26T03:43:56.141 INFO:teuthology.orchestra.run.trial132.stderr:INFO 2026-02-26 03:43:56,143 [shard 0:main] io - IO queue was unable to find a suitable maximum request length, the search was cut-off early at: 16MB 2026-02-26T03:43:56.141 INFO:teuthology.orchestra.run.trial132.stderr:INFO 2026-02-26 03:43:56,143 [shard 0:main] io - IO queue was unable to find a suitable maximum request length, the search was cut-off early at: 16MB 2026-02-26T03:43:56.143 INFO:teuthology.orchestra.run.trial132.stderr:==39579==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2026-02-26T03:43:56.198 INFO:teuthology.orchestra.run.trial132.stderr:INFO 2026-02-26 03:43:56,200 [shard 0:main] osd - bypassing the config fetch due to --no-mon-config 2026-02-26T03:43:56.199 INFO:teuthology.orchestra.run.trial132.stderr:INFO 2026-02-26 03:43:56,200 [shard 0:main] osd - Configure option_name crimson_reactor_task_quota_ms with value : 0.500000 2026-02-26T03:43:56.199 INFO:teuthology.orchestra.run.trial132.stderr:INFO 2026-02-26 03:43:56,200 [shard 0:main] osd - Configure option_name crimson_reactor_idle_poll_time_us with value : 200 2026-02-26T03:43:56.199 INFO:teuthology.orchestra.run.trial132.stderr:INFO 2026-02-26 03:43:56,200 [shard 0:main] osd - get_early_config: set --thread-affinity 0 --smp 4 2026-02-26T03:43:56.199 INFO:teuthology.orchestra.run.trial132.stderr:DEBUG 2026-02-26 03:43:56,201 [shard 0:main] seastar - reactor::drain 2026-02-26T03:43:58.276 INFO:teuthology.orchestra.run.trial132.stderr:----------------------------------------------------- 2026-02-26T03:43:58.277 INFO:teuthology.orchestra.run.trial132.stderr:Suppressions used: 2026-02-26T03:43:58.277 INFO:teuthology.orchestra.run.trial132.stderr: count bytes template 2026-02-26T03:43:58.277 INFO:teuthology.orchestra.run.trial132.stderr: 1 8 MallocExtension::Register 2026-02-26T03:43:58.277 INFO:teuthology.orchestra.run.trial132.stderr:----------------------------------------------------- 2026-02-26T03:43:58.277 INFO:teuthology.orchestra.run.trial132.stderr: 2026-02-26T03:43:58.296 INFO:teuthology.orchestra.run.trial132.stderr:INFO  osd - OSD::main: early config parsed successfully 2026-02-26T03:43:58.296 INFO:teuthology.orchestra.run.trial132.stderr:INFO  osd - OSD::main: initializing seastar app_template 2026-02-26T03:43:58.336 INFO:teuthology.orchestra.run.trial132.stderr:INFO  osd - OSD::main: registering CLI options 2026-02-26T03:43:58.336 INFO:teuthology.orchestra.run.trial132.stderr:INFO  osd - OSD::main: entering seastar runtime 2026-02-26T03:43:58.342 INFO:teuthology.orchestra.run.trial132.stderr:INFO 2026-02-26 03:43:58,345 seastar - Reactor backend: linux-aio 2026-02-26T03:43:58.350 INFO:teuthology.orchestra.run.trial132.stderr:INFO 2026-02-26 03:43:58,352 [shard 1:main] io - IO queue was unable to find a suitable maximum request length, the search was cut-off early at: 16MB 2026-02-26T03:43:58.350 INFO:teuthology.orchestra.run.trial132.stderr:INFO 2026-02-26 03:43:58,352 [shard 1:main] io - IO queue was unable to find a suitable maximum request length, the search was cut-off early at: 16MB 2026-02-26T03:43:58.354 INFO:teuthology.orchestra.run.trial132.stderr:==39577==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2026-02-26T03:43:58.354 INFO:teuthology.orchestra.run.trial132.stderr:INFO 2026-02-26 03:43:58,356 [shard 0:main] osd - OSD::main: seastar runtime started 2026-02-26T03:43:58.412 INFO:teuthology.orchestra.run.trial132.stderr:DEBUG 2026-02-26 03:43:58,414 [shard 0:main] osd - OSD::main: initializing logger output 2026-02-26T03:43:58.492 INFO:teuthology.orchestra.run.trial132.stderr:Reactor stalled for 32 ms on shard 2, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0x9e28178 0x9e28461 0x9e88dff 0x9e895f2 0x9e89860 0x9e899a8 0x9e89e67 /lib64/libc.so.6+0x3fc2f /lib64/libasan.so.8+0x1297e4 /lib64/libasan.so.8+0xc65d1 /lib64/libasan.so.8+0x15197 0x9c0d0ed 0x9d394ec 0x9d3ebdc 0x9d3f21a 0x9d496e7 0x78947dd 0x7be9c23 0x7bf3e60 0x7a431c9 0x7a807bd 0x711f05d 0x6fa06ea 0x6fa44eb 0x6faa095 0x7419054 0x743ba5c 0x728f369 0x728f461 0x736153a 0x73615f3 0x73617aa 0x743badd 0x728f4de 0x728f563 0x72bb301 0x7423ee4 0x7423f69 0x7424047 0x72a6c86 0x73b49a4 0x743027a 0x74302f5 0x7430370 0x74f3b82 0x75ebd86 0x9ee2cc7 0x9e36f9c 0x9f10737 0x9f78f1a 0x9f939aa 0x9f94026 0x26ea5d3 0xa5a1f30 /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-26T03:43:58.493 INFO:teuthology.orchestra.run.trial132.stderr:Reactor stalled for 32 ms on shard 3, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0x9e28178 0x9e28461 0x9e88dff 0x9e895f2 0x9e89860 0x9e899a8 0x9e89e67 /lib64/libc.so.6+0x3fc2f /lib64/libasan.so.8+0x12978a /lib64/libasan.so.8+0xc65d1 /lib64/libasan.so.8+0x15197 0x1125428 0xf2e0d1 0x9c50a8f 0x9c50f59 0x9c511cc 0x9c52ef3 0x9c61c89 0x9c6209b 0x9d328ee 0x9d32d22 0x9d330b5 0x9d45d3c 0x9d07d91 0x9d06d91 0x7bea2f1 0x7bf3e60 0x7a431c9 0x7a807bd 0x711f05d 0x6fa06ea 0x6fa44eb 0x6faa095 0x7419054 0x743ba5c 0x728f369 0x728f461 0x736153a 0x73615f3 0x73617aa 0x743badd 0x728f4de 0x728f563 0x72bb301 0x7423ee4 0x7423f69 0x7424047 0x72a6c86 0x73b49a4 0x743027a 0x74302f5 0x7430370 0x74f3b82 0x75ebd86 0x9ee2cc7 0x9e36f9c 0x9f10737 0x9f78f1a 0x9f939aa 0x9f94026 0x26ea5d3 0xa5a1f30 /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-26T03:43:58.493 INFO:teuthology.orchestra.run.trial132.stderr:Reactor stalled for 32 ms on shard 1, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0x9e28178 0x9e28461 0x9e88dff 0x9e895f2 0x9e89860 0x9e899a8 0x9e89e67 /lib64/libc.so.6+0x3fc2f /lib64/libasan.so.8+0xc597 /lib64/libasan.so.8+0x91c1 /lib64/libasan.so.8+0xee7dc 0x748c80d 0x7289368 0x72b2d61 0x73d6fea 0x7894741 0x7bea228 0x7bf3e60 0x7a431c9 0x7a807bd 0x711f05d 0x6fa06ea 0x6fa44eb 0x6faa095 0x7419054 0x743ba5c 0x728f369 0x728f461 0x736153a 0x73615f3 0x73617aa 0x743badd 0x728f4de 0x728f563 0x72bb301 0x7423ee4 0x7423f69 0x7424047 0x72a6c86 0x73b49a4 0x743027a 0x74302f5 0x7430370 0x74f3b82 0x75ebd86 0x9ee2cc7 0x9e36f9c 0x9f10737 0x9f78f1a 0x9f939aa 0x9f94026 0x26ea5d3 0xa5a1f30 /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-26T03:43:58.493 INFO:teuthology.orchestra.run.trial132.stderr:kernel callstack: 0xffffffffffffff80 0xffffffffbbd2a157 0xffffffffbb9f9006 0xffffffffbb9fc9c0 0xffffffffbb9fdd21 0xffffffffbba25a53 0xffffffffbba25c15 0xffffffffbb9ca888 0xffffffffbb9d4bab 0xffffffffbb9d500b 0xffffffffbb6910d2 0xffffffffbc350451 0xffffffffbc400ba2 2026-02-26T03:43:58.525 INFO:teuthology.orchestra.run.trial132.stderr:Reactor stalled for 65 ms on shard 0, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0x9e28178 0x9e28461 0x9e88dff 0x9e895f2 0x9e89860 0x9e899a8 0x9e89e67 /lib64/libc.so.6+0x3fc2f /lib64/libasan.so.8+0x119155 /lib64/libasan.so.8+0xc5f4 /lib64/libasan.so.8+0x91c1 /lib64/libasan.so.8+0xee7dc 0xce18db 0xce1ee3 0xce1fe0 0x9b8799b 0x9b87c9b 0x9baf5ad 0x9bb0747 0x9bc9fc9 0x9bcaca0 0x9bc7c4b 0x772d8ea 0x77e0616 0x77e2a38 0x943658e 0x947b43c 0x7dd81cb 0x7ee070c 0x7f55cd5 0x7d8b2fd 0x7e6482d 0x8f5d61d 0x8f5ff8b 0x787cd7f 0x78b3782 0x6fa18b5 0x755a4ec 0x755a640 0x9ee2cc7 0x9e36f9c 0x9f10737 0x9f78f1a 0x9f7a738 0x9bf83c1 0x9bf8bfc 0xecf14d /lib64/libc.so.6+0x2a60f /lib64/libc.so.6+0x2a6bf 0xb46f44 2026-02-26T03:43:58.590 INFO:teuthology.orchestra.run.trial132.stderr:Reactor stalled for 65 ms on shard 0, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0x9e28178 0x9e28461 0x9e88dff 0x9e895f2 0x9e89860 0x9e899a8 0x9e89e67 /lib64/libc.so.6+0x3fc2f 0x2a55a3a 0x7d837b9 0x7d9fd5e 0x7e419e3 0x7dc12dd 0x7877bf3 0x78788e8 0x79adc2a 0x7836cfc 0x78b1d9e 0x78fa531 0x78fa7aa 0x799b982 0x799bae1 0x79add8d 0x7868874 0x798a5f1 0x798a680 0x798a724 0x73e442a 0x7404ce9 0x740998a 0x754d7e2 0x9ee2cc7 0x9e36f9c 0x9f10737 0x9f78f1a 0x9f7a738 0x9bf83c1 0x9bf8bfc 0xecf14d /lib64/libc.so.6+0x2a60f /lib64/libc.so.6+0x2a6bf 0xb46f44 2026-02-26T03:43:58.596 INFO:teuthology.orchestra.run.trial132.stderr:Reactor stalled for 32 ms on shard 3, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0x9e28178 0x9e28461 0x9e88dff 0x9e895f2 0x9e89860 0x9e899a8 0x9e89e67 /lib64/libc.so.6+0x3fc2f /lib64/libasan.so.8+0x10069c /lib64/libasan.so.8+0xc65b9 /lib64/libasan.so.8+0x15197 0x2a55a3f 0x7d837b9 0x7d9fd5e 0x7e419e3 0x7dc12dd 0x7877bf3 0x78788e8 0x79adc2a 0x7836cfc 0x78b1d9e 0x78fa531 0x78fa7aa 0x799b982 0x799bae1 0x79add8d 0x7868874 0x798a5f1 0x798a680 0x798a724 0x73e442a 0x7404ce9 0x740998a 0x754d7e2 0x9ee2cc7 0x9e36f9c 0x9f10737 0x9f78f1a 0x9f939aa 0x9f94026 0x26ea5d3 0xa5a1f30 /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-26T03:43:58.627 INFO:teuthology.orchestra.run.trial132.stderr:Reactor stalled for 65 ms on shard 1, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0x9e28178 0x9e28461 0x9e88dff 0x9e895f2 0x9e89860 0x9e899a8 0x9e89e67 /lib64/libc.so.6+0x3fc2f 0x2a553df 0x7d837f4 0x7d9fd5e 0x7e419e3 0x7dc12dd 0x7877bf3 0x78788e8 0x79adc2a 0x7836cfc 0x78b1d9e 0x78fa531 0x78fa7aa 0x799b982 0x799bae1 0x79add8d 0x7868874 0x798a5f1 0x798a680 0x798a724 0x73e442a 0x7404ce9 0x740998a 0x754d7e2 0x9ee2cc7 0x9e36f9c 0x9f10737 0x9f78f1a 0x9f939aa 0x9f94026 0x26ea5d3 0xa5a1f30 /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-26T03:43:58.628 INFO:teuthology.orchestra.run.trial132.stderr:Reactor stalled for 65 ms on shard 2, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0x9e28178 0x9e28461 0x9e88dff 0x9e895f2 0x9e89860 0x9e899a8 0x9e89e67 /lib64/libc.so.6+0x3fc2f /lib64/libasan.so.8+0x1297e4 /lib64/libasan.so.8+0xc65d1 /lib64/libasan.so.8+0x15197 0x2a55a3f 0x7d837b9 0x7d9fd5e 0x7e419e3 0x7dc12dd 0x7877bf3 0x78788e8 0x79adc2a 0x7836cfc 0x78b1d9e 0x78fa531 0x78fa7aa 0x799b982 0x799bae1 0x79add8d 0x7868874 0x798a5f1 0x798a680 0x798a724 0x73e442a 0x7404ce9 0x740998a 0x754d7e2 0x9ee2cc7 0x9e36f9c 0x9f10737 0x9f78f1a 0x9f939aa 0x9f94026 0x26ea5d3 0xa5a1f30 /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-26T03:43:58.749 INFO:teuthology.orchestra.run.trial132.stderr:Reactor stalled for 32 ms on shard 1, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0x9e28178 0x9e28461 0x9e88dff 0x9e895f2 0x9e89860 0x9e899a8 0x9e89e67 /lib64/libc.so.6+0x3fc2f 0xe06a20 0x11253d1 0xf2e0d1 0x9c50a8f 0x9c50f59 0x9c511cc 0x9c51482 0x9c51f31 0x9c5242f 0x9c52a5f 0x9c52bc9 0x9d32f39 0x9d45d3c 0x9d07d91 0x9d06d91 0x7bea2f1 0x7bf3e60 0x7a431c9 0x7a807bd 0x711f05d 0x6f43702 0x6f45a6a 0x7483f52 0x72a690f 0x72a6a11 0x72a6b09 0x7360706 0x73607bf 0x7360972 0x7360aaa 0x7360b25 0x7360baa 0x7360d05 0x7423961 0x7483fdb 0x728dda4 0x72a6c86 0x73b49a4 0x743027a 0x74302f5 0x7430370 0x74f3b82 0x75ebd86 0x9ee2cc7 0x9e36f9c 0x9f10737 0x9f78f1a 0x9f939aa 0x9f94026 0x26ea5d3 0xa5a1f30 /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-26T03:43:58.749 INFO:teuthology.orchestra.run.trial132.stderr:Reactor stalled for 32 ms on shard 3, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0x9e28178 0x9e28461 0x9e88dff 0x9e895f2 0x9e89860 0x9e899a8 0x9e89e67 /lib64/libc.so.6+0x3fc2f 0x109078b 0xf2e0dc 0x9d33899 0x9d33e86 0x9d34209 0x9d44360 0x9d477c3 0x9d07d91 0x9d06d91 0x7bea2f1 0x7bf3e60 0x7a431c9 0x7a807bd 0x711f05d 0x6f43702 0x6f45a6a 0x7483f52 0x72a690f 0x72a6a11 0x72a6b09 0x7360706 0x73607bf 0x7360972 0x7360aaa 0x7360b25 0x7360baa 0x7360d05 0x7423961 0x7483fdb 0x728dda4 0x72a6c86 0x73b49a4 0x743027a 0x74302f5 0x7430370 0x74f3b82 0x75ebd86 0x9ee2cc7 0x9e36f9c 0x9f10737 0x9f78f1a 0x9f939aa 0x9f94026 0x26ea5d3 0xa5a1f30 /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-26T03:43:58.749 INFO:teuthology.orchestra.run.trial132.stderr:Reactor stalled for 32 ms on shard 2, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0x9e28178 0x9e28461 0x9e88dff 0x9e895f2 0x9e89860 0x9e899a8 0x9e89e67 /lib64/libc.so.6+0x3fc2f /lib64/libasan.so.8+0xc250 /lib64/libasan.so.8+0xc553 /lib64/libasan.so.8+0x91c1 /lib64/libasan.so.8+0xee7dc 0x9b2559b 0x9ba2d36 0x9ba2e4c 0x9ba2f28 0x9ba31a4 0x9ba374b 0x9ba3b4f 0x9ba3e0c 0x9ba4c54 0x9d1b824 0x9d1b91d 0x7be9c60 0x7bf3e60 0x7a431c9 0x7a807bd 0x711f05d 0x6f43702 0x6f45a6a 0x7483f52 0x72a690f 0x72a6a11 0x72a6b09 0x7360706 0x73607bf 0x7360972 0x7360aaa 0x7360b25 0x7360baa 0x7360d05 0x7423961 0x7483fdb 0x728dda4 0x72a6c86 0x73b49a4 0x743027a 0x74302f5 0x7430370 0x74f3b82 0x75ebd86 0x9ee2cc7 0x9e36f9c 0x9f10737 0x9f78f1a 0x9f939aa 0x9f94026 0x26ea5d3 0xa5a1f30 /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-26T03:43:58.832 INFO:teuthology.orchestra.run.trial132.stderr:Reactor stalled for 65 ms on shard 0, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0x9e28178 0x9e28461 0x9e88dff 0x9e895f2 0x9e89860 0x9e899a8 0x9e89e67 /lib64/libc.so.6+0x3fc2f 0x2a00bea 0x2a559e0 0x7d837b9 0x7d9fd5e 0x7e419e3 0x7dc12dd 0x7877bf3 0x78788e8 0x79adc2a 0x7836cfc 0x78b1d9e 0x78fa531 0x78fa7aa 0x799b982 0x799bae1 0x79add8d 0x7868874 0x798a5f1 0x798a680 0x798a724 0x73e442a 0x7404ce9 0x740998a 0x754d7e2 0x9ee2cc7 0x9e36f9c 0x9f10737 0x9f78f1a 0x9f7a738 0x9bf83c1 0x9bf8bfc 0xecf14d /lib64/libc.so.6+0x2a60f /lib64/libc.so.6+0x2a6bf 0xb46f44 2026-02-26T03:43:58.845 INFO:teuthology.orchestra.run.trial132.stderr:Reactor stalled for 65 ms on shard 1, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0x9e28178 0x9e28461 0x9e88dff 0x9e895f2 0x9e89860 0x9e899a8 0x9e89e67 /lib64/libc.so.6+0x3fc2f 0x2a559ce 0x7d837b9 0x7d9fd5e 0x7e419e3 0x7dc12dd 0x7877bf3 0x78788e8 0x79adc2a 0x7836cfc 0x78b1d9e 0x78fa531 0x78fa7aa 0x799b982 0x799bae1 0x79add8d 0x7868874 0x798a5f1 0x798a680 0x798a724 0x73e442a 0x7404ce9 0x740998a 0x754d7e2 0x9ee2cc7 0x9e36f9c 0x9f10737 0x9f78f1a 0x9f939aa 0x9f94026 0x26ea5d3 0xa5a1f30 /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-26T03:43:58.846 INFO:teuthology.orchestra.run.trial132.stderr:Reactor stalled for 65 ms on shard 2, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0x9e28178 0x9e28461 0x9e88dff 0x9e895f2 0x9e89860 0x9e899a8 0x9e89e67 /lib64/libc.so.6+0x3fc2f /lib64/libasan.so.8+0x1297e4 /lib64/libasan.so.8+0xc65d1 /lib64/libasan.so.8+0x15197 0x2a55a3f 0x7d837b9 0x7d9fd5e 0x7e419e3 0x7dc12dd 0x7877bf3 0x78788e8 0x79adc2a 0x7836cfc 0x78b1d9e 0x78fa531 0x78fa7aa 0x799b982 0x799bae1 0x79add8d 0x7868874 0x798a5f1 0x798a680 0x798a724 0x73e442a 0x7404ce9 0x740998a 0x754d7e2 0x9ee2cc7 0x9e36f9c 0x9f10737 0x9f78f1a 0x9f939aa 0x9f94026 0x26ea5d3 0xa5a1f30 /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-26T03:43:58.846 INFO:teuthology.orchestra.run.trial132.stderr:Reactor stalled for 65 ms on shard 3, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0x9e28178 0x9e28461 0x9e88dff 0x9e895f2 0x9e89860 0x9e899a8 0x9e89e67 /lib64/libc.so.6+0x3fc2f 0x2a55a3a 0x7d837b9 0x7d9fd5e 0x7e419e3 0x7dc12dd 0x7877bf3 0x78788e8 0x79adc2a 0x7836cfc 0x78b1d9e 0x78fa531 0x78fa7aa 0x799b982 0x799bae1 0x79add8d 0x7868874 0x798a5f1 0x798a680 0x798a724 0x73e442a 0x7404ce9 0x740998a 0x754d7e2 0x9ee2cc7 0x9e36f9c 0x9f10737 0x9f78f1a 0x9f939aa 0x9f94026 0x26ea5d3 0xa5a1f30 /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-26T03:44:00.976 INFO:teuthology.orchestra.run.trial132.stderr:----------------------------------------------------- 2026-02-26T03:44:00.976 INFO:teuthology.orchestra.run.trial132.stderr:Suppressions used: 2026-02-26T03:44:00.976 INFO:teuthology.orchestra.run.trial132.stderr: count bytes template 2026-02-26T03:44:00.976 INFO:teuthology.orchestra.run.trial132.stderr: 1 8 MallocExtension::Register 2026-02-26T03:44:00.977 INFO:teuthology.orchestra.run.trial132.stderr:----------------------------------------------------- 2026-02-26T03:44:00.977 INFO:teuthology.orchestra.run.trial132.stderr: 2026-02-26T03:44:01.009 DEBUG:teuthology.orchestra.run.trial132:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2026-02-26T03:44:01.034 INFO:tasks.ceph:Reading keys from all nodes... 2026-02-26T03:44:01.035 DEBUG:teuthology.orchestra.run.trial132:> set -ex 2026-02-26T03:44:01.035 DEBUG:teuthology.orchestra.run.trial132:> sudo dd if=/var/lib/ceph/mgr/ceph-x/keyring of=/dev/stdout 2026-02-26T03:44:01.098 DEBUG:teuthology.orchestra.run.trial132:> set -ex 2026-02-26T03:44:01.099 DEBUG:teuthology.orchestra.run.trial132:> sudo dd if=/var/lib/ceph/osd/ceph-0/keyring of=/dev/stdout 2026-02-26T03:44:01.162 DEBUG:teuthology.orchestra.run.trial132:> set -ex 2026-02-26T03:44:01.162 DEBUG:teuthology.orchestra.run.trial132:> sudo dd if=/var/lib/ceph/osd/ceph-1/keyring of=/dev/stdout 2026-02-26T03:44:01.226 DEBUG:teuthology.orchestra.run.trial132:> set -ex 2026-02-26T03:44:01.226 DEBUG:teuthology.orchestra.run.trial132:> sudo dd if=/var/lib/ceph/osd/ceph-2/keyring of=/dev/stdout 2026-02-26T03:44:01.289 DEBUG:teuthology.orchestra.run.trial132:> set -ex 2026-02-26T03:44:01.290 DEBUG:teuthology.orchestra.run.trial132:> dd if=/etc/ceph/ceph.client.0.keyring of=/dev/stdout 2026-02-26T03:44:01.345 INFO:tasks.ceph:Adding keys to all mons... 2026-02-26T03:44:01.346 DEBUG:teuthology.orchestra.run.trial132:> sudo tee -a /etc/ceph/ceph.keyring 2026-02-26T03:44:01.408 INFO:teuthology.orchestra.run.trial132.stdout:[mgr.x] 2026-02-26T03:44:01.408 INFO:teuthology.orchestra.run.trial132.stdout: key = AQBvwZ9pMNq1BxAAKTZqL4LGzePPVqc6kQN6xA== 2026-02-26T03:44:01.408 INFO:teuthology.orchestra.run.trial132.stdout:[osd.0] 2026-02-26T03:44:01.408 INFO:teuthology.orchestra.run.trial132.stdout: key = AQB0wZ9pg0YKChAAc0w+Mv/RyelAHvItyUcUmQ== 2026-02-26T03:44:01.409 INFO:teuthology.orchestra.run.trial132.stdout:[osd.1] 2026-02-26T03:44:01.409 INFO:teuthology.orchestra.run.trial132.stdout: key = AQB5wZ9pk9D3EBAA4MaLLBpTL0qakSkuVBE2DA== 2026-02-26T03:44:01.409 INFO:teuthology.orchestra.run.trial132.stdout:[osd.2] 2026-02-26T03:44:01.409 INFO:teuthology.orchestra.run.trial132.stdout: key = AQB+wZ9piK7zGBAA/c47fxCZqAaMdY8B3NH+Iw== 2026-02-26T03:44:01.409 INFO:teuthology.orchestra.run.trial132.stdout:[client.0] 2026-02-26T03:44:01.409 INFO:teuthology.orchestra.run.trial132.stdout: key = AQBvwZ9pGt5pChAAD2WPuivx5fmLQwGMSrIbFg== 2026-02-26T03:44:01.410 DEBUG:teuthology.orchestra.run.trial132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2026-02-26T03:44:01.498 DEBUG:teuthology.orchestra.run.trial132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2026-02-26T03:44:01.547 DEBUG:teuthology.orchestra.run.trial132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2026-02-26T03:44:01.596 DEBUG:teuthology.orchestra.run.trial132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2026-02-26T03:44:01.643 DEBUG:teuthology.orchestra.run.trial132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2026-02-26T03:44:01.693 INFO:tasks.ceph:Running mkfs on mon nodes... 2026-02-26T03:44:01.693 DEBUG:teuthology.orchestra.run.trial132:> sudo mkdir -p /var/lib/ceph/mon/ceph-a 2026-02-26T03:44:01.717 DEBUG:teuthology.orchestra.run.trial132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i a --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2026-02-26T03:44:01.802 INFO:teuthology.orchestra.run.trial132.stderr:2026-02-26T03:44:01.804+0000 7f52f12b2c00 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-26T03:44:01.803 INFO:teuthology.orchestra.run.trial132.stderr:2026-02-26T03:44:01.804+0000 7f52f12b2c00 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-26T03:44:01.814 DEBUG:teuthology.orchestra.run.trial132:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-a 2026-02-26T03:44:01.839 DEBUG:teuthology.orchestra.run.trial132:> rm -- /home/ubuntu/cephtest/ceph.monmap 2026-02-26T03:44:01.895 INFO:tasks.ceph:Starting mon daemons in cluster ceph... 2026-02-26T03:44:01.896 INFO:tasks.ceph.mon.a:Restarting daemon 2026-02-26T03:44:01.896 DEBUG:teuthology.orchestra.run.trial132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2026-02-26T03:44:01.937 INFO:tasks.ceph.mon.a:Started 2026-02-26T03:44:01.938 INFO:tasks.ceph:Starting mgr daemons in cluster ceph... 2026-02-26T03:44:01.938 INFO:tasks.ceph.mgr.x:Restarting daemon 2026-02-26T03:44:01.938 DEBUG:teuthology.orchestra.run.trial132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i x 2026-02-26T03:44:01.941 INFO:tasks.ceph.mgr.x:Started 2026-02-26T03:44:01.941 DEBUG:tasks.ceph:set 0 configs 2026-02-26T03:44:01.941 DEBUG:teuthology.orchestra.run.trial132:> sudo ceph --cluster ceph config dump 2026-02-26T03:44:02.000 INFO:tasks.ceph.mon.a.trial132.stderr:2026-02-26T03:44:02.002+0000 7f4f72095c00 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-26T03:44:02.001 INFO:tasks.ceph.mon.a.trial132.stderr:2026-02-26T03:44:02.002+0000 7f4f72095c00 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-26T03:44:02.001 INFO:tasks.ceph.mon.a.trial132.stderr:2026-02-26T03:44:02.002+0000 7f4f72095c00 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-26T03:44:02.005 INFO:tasks.ceph.mgr.x.trial132.stderr:2026-02-26T03:44:02.007+0000 7f4d439c4fc0 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-26T03:44:02.018 INFO:teuthology.orchestra.run.trial132.stderr:2026-02-26T03:44:02.017+0000 7fb698acb640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-26T03:44:02.021 INFO:teuthology.orchestra.run.trial132.stderr:2026-02-26T03:44:02.023+0000 7fb698acb640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-26T03:44:02.055 INFO:teuthology.orchestra.run.trial132.stdout:WHO MASK LEVEL OPTION VALUE RO 2026-02-26T03:44:02.064 INFO:tasks.ceph:Setting crush tunables to default 2026-02-26T03:44:02.065 DEBUG:teuthology.orchestra.run.trial132:> sudo ceph --cluster ceph osd crush tunables default 2026-02-26T03:44:02.139 INFO:teuthology.orchestra.run.trial132.stderr:2026-02-26T03:44:02.137+0000 7f9c23fa4640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-26T03:44:02.141 INFO:teuthology.orchestra.run.trial132.stderr:2026-02-26T03:44:02.143+0000 7f9c23fa4640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-26T03:44:02.175 INFO:teuthology.orchestra.run.trial132.stderr:adjusted tunables profile to default 2026-02-26T03:44:02.183 INFO:tasks.ceph:check_enable_crimson: True 2026-02-26T03:44:02.183 INFO:tasks.ceph:check_enable_crimson: setting set-allow-crimson 2026-02-26T03:44:02.184 DEBUG:teuthology.orchestra.run.trial132:> sudo ceph --cluster ceph osd set-allow-crimson --yes-i-really-mean-it 2026-02-26T03:44:02.210 INFO:tasks.ceph.mgr.x.trial132.stderr:2026-02-26T03:44:02.211+0000 7f4d439c4fc0 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-26T03:44:02.210 INFO:tasks.ceph.mgr.x.trial132.stderr:2026-02-26T03:44:02.212+0000 7f4d439c4fc0 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-26T03:44:02.258 INFO:teuthology.orchestra.run.trial132.stderr:2026-02-26T03:44:02.256+0000 7f149395b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-26T03:44:02.260 INFO:teuthology.orchestra.run.trial132.stderr:2026-02-26T03:44:02.262+0000 7f149395b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-26T03:44:02.299 INFO:tasks.ceph.mon.a.trial132.stderr:2026-02-26T03:44:02.300+0000 7f4f697f1640 -1 WARNING: experimental feature 'crimson' is enabled 2026-02-26T03:44:02.299 INFO:tasks.ceph.mon.a.trial132.stderr:Please be aware that this feature is experimental, untested, 2026-02-26T03:44:02.299 INFO:tasks.ceph.mon.a.trial132.stderr:unsupported, and may result in data corruption, data loss, 2026-02-26T03:44:02.299 INFO:tasks.ceph.mon.a.trial132.stderr:and/or irreparable damage to your cluster. Do not use 2026-02-26T03:44:02.299 INFO:tasks.ceph.mon.a.trial132.stderr:feature with important data. 2026-02-26T03:44:02.299 INFO:tasks.ceph.mon.a.trial132.stderr: 2026-02-26T03:44:03.058 INFO:tasks.ceph.mgr.x.trial132.stderr:/usr/lib64/python3.9/site-packages/scipy/__init__.py:73: UserWarning: NumPy was imported from a Python sub-interpreter but NumPy does not properly support sub-interpreters. This will likely work for most users but might cause hard to track down issues or subtle bugs. A common user of the rare sub-interpreter feature is wsgi which also allows single-interpreter mode. 2026-02-26T03:44:03.058 INFO:tasks.ceph.mgr.x.trial132.stderr:Improvements in the case of bugs are welcome, but is not on the NumPy roadmap, and full support may require significant effort to achieve. 2026-02-26T03:44:03.059 INFO:tasks.ceph.mgr.x.trial132.stderr: from numpy import show_config as show_numpy_config 2026-02-26T03:44:03.187 INFO:tasks.ceph:Starting osd daemons in cluster ceph... 2026-02-26T03:44:03.187 DEBUG:teuthology.orchestra.run.trial132:> set -ex 2026-02-26T03:44:03.187 DEBUG:teuthology.orchestra.run.trial132:> sudo dd if=/var/lib/ceph/osd/ceph-0/fsid of=/dev/stdout 2026-02-26T03:44:03.251 DEBUG:teuthology.orchestra.run.trial132:> set -ex 2026-02-26T03:44:03.251 DEBUG:teuthology.orchestra.run.trial132:> sudo dd if=/var/lib/ceph/osd/ceph-1/fsid of=/dev/stdout 2026-02-26T03:44:03.315 DEBUG:teuthology.orchestra.run.trial132:> set -ex 2026-02-26T03:44:03.316 DEBUG:teuthology.orchestra.run.trial132:> sudo dd if=/var/lib/ceph/osd/ceph-2/fsid of=/dev/stdout 2026-02-26T03:44:03.381 DEBUG:teuthology.orchestra.run.trial132:> sudo ceph --cluster ceph osd new de6b87e2-9687-4f88-b0cb-1e4799f76b1c 0 2026-02-26T03:44:03.495 INFO:teuthology.orchestra.run.trial132.stderr:2026-02-26T03:44:03.493+0000 7f9a157f5640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-26T03:44:03.497 INFO:teuthology.orchestra.run.trial132.stderr:2026-02-26T03:44:03.499+0000 7f9a157f5640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-26T03:44:03.534 INFO:teuthology.orchestra.run.trial132.stdout:0 2026-02-26T03:44:03.542 DEBUG:teuthology.orchestra.run.trial132:> sudo ceph --cluster ceph osd new 7d7e4e45-9f58-40d0-b37a-0973e1bc7504 1 2026-02-26T03:44:03.614 INFO:teuthology.orchestra.run.trial132.stderr:2026-02-26T03:44:03.613+0000 7fc7a19c1640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-26T03:44:03.616 INFO:teuthology.orchestra.run.trial132.stderr:2026-02-26T03:44:03.618+0000 7fc7a19c1640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-26T03:44:03.653 INFO:teuthology.orchestra.run.trial132.stdout:1 2026-02-26T03:44:03.661 DEBUG:teuthology.orchestra.run.trial132:> sudo ceph --cluster ceph osd new f83e5653-ffd0-41dd-b067-fa02dc7934e4 2 2026-02-26T03:44:03.737 INFO:teuthology.orchestra.run.trial132.stderr:2026-02-26T03:44:03.735+0000 7f56e761a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-26T03:44:03.739 INFO:teuthology.orchestra.run.trial132.stderr:2026-02-26T03:44:03.741+0000 7f56e761a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-26T03:44:03.777 INFO:teuthology.orchestra.run.trial132.stdout:2 2026-02-26T03:44:03.785 DEBUG:teuthology.orchestra.run.trial132:> sudo install -b -o ubuntu -g ubuntu /dev/null /var/log/ceph/ceph-osd.0.log 2026-02-26T03:44:03.813 INFO:tasks.ceph.osd.0:Restarting daemon 2026-02-26T03:44:03.813 DEBUG:teuthology.orchestra.run.trial132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 0 2>> /var/log/ceph/ceph-osd.0.log 2026-02-26T03:44:03.855 INFO:tasks.ceph.osd.0:Started 2026-02-26T03:44:03.856 DEBUG:teuthology.orchestra.run.trial132:> sudo install -b -o ubuntu -g ubuntu /dev/null /var/log/ceph/ceph-osd.1.log 2026-02-26T03:44:03.885 INFO:tasks.ceph.osd.1:Restarting daemon 2026-02-26T03:44:03.885 DEBUG:teuthology.orchestra.run.trial132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 1 2>> /var/log/ceph/ceph-osd.1.log 2026-02-26T03:44:03.927 INFO:tasks.ceph.osd.1:Started 2026-02-26T03:44:03.928 DEBUG:teuthology.orchestra.run.trial132:> sudo install -b -o ubuntu -g ubuntu /dev/null /var/log/ceph/ceph-osd.2.log 2026-02-26T03:44:03.954 INFO:tasks.ceph.osd.2:Restarting daemon 2026-02-26T03:44:03.955 DEBUG:teuthology.orchestra.run.trial132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 2 2>> /var/log/ceph/ceph-osd.2.log 2026-02-26T03:44:03.997 INFO:tasks.ceph.osd.2:Started 2026-02-26T03:44:03.998 DEBUG:teuthology.orchestra.run.trial132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2026-02-26T03:44:04.076 INFO:teuthology.orchestra.run.trial132.stderr:2026-02-26T03:44:04.076+0000 7f2e6d411640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-26T03:44:04.079 INFO:teuthology.orchestra.run.trial132.stderr:2026-02-26T03:44:04.080+0000 7f2e6d411640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-26T03:44:04.113 INFO:teuthology.orchestra.run.trial132.stdout: 2026-02-26T03:44:04.113 INFO:teuthology.orchestra.run.trial132.stdout:{"epoch":6,"fsid":"109d3300-c629-4972-9fab-d8a7123a9741","created":"2026-02-26T03:44:02.016900+0000","modified":"2026-02-26T03:44:03.779206+0000","last_up_change":"0.000000","last_in_change":"2026-02-26T03:44:03.779206+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[],"osds":[{"osd":0,"uuid":"de6b87e2-9687-4f88-b0cb-1e4799f76b1c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"7d7e4e45-9f58-40d0-b37a-0973e1bc7504","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"f83e5653-ffd0-41dd-b067-fa02dc7934e4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-26T03:44:04.122 INFO:tasks.ceph.ceph_manager.ceph:[] 2026-02-26T03:44:04.122 INFO:tasks.ceph:Waiting for OSDs to come up 2026-02-26T03:44:04.424 DEBUG:teuthology.orchestra.run.trial132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2026-02-26T03:44:04.492 INFO:teuthology.misc.health.trial132.stderr:2026-02-26T03:44:04.491+0000 7f68b83ad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-26T03:44:04.495 INFO:teuthology.misc.health.trial132.stderr:2026-02-26T03:44:04.496+0000 7f68b83ad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-26T03:44:04.528 INFO:teuthology.misc.health.trial132.stdout: 2026-02-26T03:44:04.528 INFO:teuthology.misc.health.trial132.stdout:{"epoch":6,"fsid":"109d3300-c629-4972-9fab-d8a7123a9741","created":"2026-02-26T03:44:02.016900+0000","modified":"2026-02-26T03:44:03.779206+0000","last_up_change":"0.000000","last_in_change":"2026-02-26T03:44:03.779206+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[],"osds":[{"osd":0,"uuid":"de6b87e2-9687-4f88-b0cb-1e4799f76b1c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"7d7e4e45-9f58-40d0-b37a-0973e1bc7504","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"f83e5653-ffd0-41dd-b067-fa02dc7934e4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-26T03:44:04.536 DEBUG:teuthology.misc:0 of 3 OSDs are up 2026-02-26T03:44:05.469 INFO:tasks.ceph.mgr.x.trial132.stderr:2026-02-26T03:44:05.470+0000 7f4c5e887640 -1 mgr.server handle_report got status from non-daemon mon.a 2026-02-26T03:44:10.838 DEBUG:teuthology.orchestra.run.trial132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2026-02-26T03:44:10.927 INFO:teuthology.misc.health.trial132.stderr:2026-02-26T03:44:10.925+0000 7fccd99b8640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-26T03:44:10.931 INFO:teuthology.misc.health.trial132.stderr:2026-02-26T03:44:10.932+0000 7fccd99b8640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-26T03:44:11.064 INFO:teuthology.misc.health.trial132.stdout: 2026-02-26T03:44:11.064 INFO:teuthology.misc.health.trial132.stdout:{"epoch":7,"fsid":"109d3300-c629-4972-9fab-d8a7123a9741","created":"2026-02-26T03:44:02.016900+0000","modified":"2026-02-26T03:44:10.182635+0000","last_up_change":"0.000000","last_in_change":"2026-02-26T03:44:03.779206+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":3,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[],"osds":[{"osd":0,"uuid":"de6b87e2-9687-4f88-b0cb-1e4799f76b1c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"7d7e4e45-9f58-40d0-b37a-0973e1bc7504","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"f83e5653-ffd0-41dd-b067-fa02dc7934e4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-26T03:44:11.082 DEBUG:teuthology.misc:0 of 3 OSDs are up 2026-02-26T03:44:17.385 DEBUG:teuthology.orchestra.run.trial132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2026-02-26T03:44:17.456 INFO:teuthology.misc.health.trial132.stderr:2026-02-26T03:44:17.455+0000 7f79cb406640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-26T03:44:17.459 INFO:teuthology.misc.health.trial132.stderr:2026-02-26T03:44:17.460+0000 7f79cb406640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-26T03:44:17.569 INFO:teuthology.misc.health.trial132.stdout: 2026-02-26T03:44:17.569 INFO:teuthology.misc.health.trial132.stdout:{"epoch":13,"fsid":"109d3300-c629-4972-9fab-d8a7123a9741","created":"2026-02-26T03:44:02.016900+0000","modified":"2026-02-26T03:44:17.214941+0000","last_up_change":"2026-02-26T03:44:13.195912+0000","last_in_change":"2026-02-26T03:44:03.779206+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-26T03:44:14.486080+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"13","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.9900000095367432,"score_stable":2.9900000095367432,"optimal_score":0.67000001668930054,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"de6b87e2-9687-4f88-b0cb-1e4799f76b1c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":10,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.132:6805","nonce":2385043468}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.132:6804","nonce":2385043468}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.132:6809","nonce":2385043468}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.132:6808","nonce":2385043468}]},"public_addr":"10.20.193.132:6805/2385043468","cluster_addr":"10.20.193.132:6804/2385043468","heartbeat_back_addr":"10.20.193.132:6809/2385043468","heartbeat_front_addr":"10.20.193.132:6808/2385043468","state":["exists","up"]},{"osd":1,"uuid":"7d7e4e45-9f58-40d0-b37a-0973e1bc7504","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":11,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.132:6803","nonce":546026106}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.132:6802","nonce":546026106}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.132:6815","nonce":546026106}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.132:6816","nonce":546026106}]},"public_addr":"10.20.193.132:6803/546026106","cluster_addr":"10.20.193.132:6802/546026106","heartbeat_back_addr":"10.20.193.132:6815/546026106","heartbeat_front_addr":"10.20.193.132:6816/546026106","state":["exists","up"]},{"osd":2,"uuid":"f83e5653-ffd0-41dd-b067-fa02dc7934e4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":10,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.132:6810","nonce":798308673}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.132:6811","nonce":798308673}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.132:6807","nonce":798308673}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.132:6806","nonce":798308673}]},"public_addr":"10.20.193.132:6810/798308673","cluster_addr":"10.20.193.132:6811/798308673","heartbeat_back_addr":"10.20.193.132:6807/798308673","heartbeat_front_addr":"10.20.193.132:6806/798308673","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-26T03:44:17.578 DEBUG:teuthology.misc:3 of 3 OSDs are up 2026-02-26T03:44:17.578 INFO:tasks.ceph:Creating RBD pool 2026-02-26T03:44:17.578 DEBUG:teuthology.orchestra.run.trial132:> sudo ceph --cluster ceph osd pool create rbd 8 2026-02-26T03:44:17.652 INFO:teuthology.orchestra.run.trial132.stderr:2026-02-26T03:44:17.650+0000 7f6f324b7640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-26T03:44:17.654 INFO:teuthology.orchestra.run.trial132.stderr:2026-02-26T03:44:17.655+0000 7f6f324b7640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-26T03:44:18.219 INFO:teuthology.orchestra.run.trial132.stderr:pool 'rbd' created 2026-02-26T03:44:18.231 DEBUG:teuthology.orchestra.run.trial132:> rbd --cluster ceph pool init rbd 2026-02-26T03:44:18.274 INFO:teuthology.orchestra.run.trial132.stderr:2026-02-26T03:44:18.275+0000 7ff57a8ec240 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-26T03:44:18.274 INFO:teuthology.orchestra.run.trial132.stderr:2026-02-26T03:44:18.275+0000 7ff57a8ec240 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-26T03:44:18.277 INFO:teuthology.orchestra.run.trial132.stderr:2026-02-26T03:44:18.279+0000 7ff57a8ec240 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-26T03:44:21.254 INFO:tasks.ceph:Starting mds daemons in cluster ceph... 2026-02-26T03:44:21.255 DEBUG:teuthology.orchestra.run.trial132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config log 1 --format=json 2026-02-26T03:44:21.256 INFO:tasks.daemonwatchdog.daemon_watchdog:watchdog starting 2026-02-26T03:44:21.370 INFO:teuthology.orchestra.run.trial132.stderr:2026-02-26T03:44:21.369+0000 7f04269a4640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-26T03:44:21.372 INFO:teuthology.orchestra.run.trial132.stderr:2026-02-26T03:44:21.374+0000 7f04269a4640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-26T03:44:21.481 INFO:teuthology.orchestra.run.trial132.stdout: 2026-02-26T03:44:21.491 INFO:teuthology.orchestra.run.trial132.stdout:[{"version":1,"timestamp":"0.000000","name":"","changes":[]}] 2026-02-26T03:44:21.491 INFO:tasks.ceph_manager:config epoch is 1 2026-02-26T03:44:21.491 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2026-02-26T03:44:21.492 INFO:tasks.ceph.ceph_manager.ceph:waiting for mgr available 2026-02-26T03:44:21.492 DEBUG:teuthology.orchestra.run.trial132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr dump --format=json 2026-02-26T03:44:21.567 INFO:teuthology.orchestra.run.trial132.stderr:2026-02-26T03:44:21.567+0000 7f7c37f0b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-26T03:44:21.569 INFO:teuthology.orchestra.run.trial132.stderr:2026-02-26T03:44:21.571+0000 7f7c37f0b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-26T03:44:21.714 INFO:teuthology.orchestra.run.trial132.stdout: 2026-02-26T03:44:21.724 INFO:teuthology.orchestra.run.trial132.stdout:{"epoch":5,"flags":0,"active_gid":4105,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.132:6800","nonce":130993697},{"type":"v1","addr":"10.20.193.132:6801","nonce":130993697}]},"active_addr":"10.20.193.132:6801/130993697","active_change":"2026-02-26T03:44:04.465888+0000","active_mgr_features":4544132024016699391,"available":true,"standbys":[],"modules":["iostat","nfs"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to, use commas to separate multiple","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"update_pg_upmap_activity":{"name":"update_pg_upmap_activity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Updates pg_upmap activity stats to be used in `balancer status detail`","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often the agent on each host will gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"The first TCP port the agent will try to bind to. Up to 1000 subsequent ports will be attempted if this port cannot be bound","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this option can facilitate debugging daemons that encounter runtime problems.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"How frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"Fraction of total system memory to divide among autotuned daemons. Converged systems hosting both Ceph and compute should set a lower value; systems with more than 10 GiB of memory per OSD may benefit from a slightly larger value so that the abundance of RAM can be fully utilized","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command persistent logging","long_desc":"","tags":[],"see_also":[]},"certificate_automated_rotation_enabled":{"name":"certificate_automated_rotation_enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"This flag controls whether cephadm automatically rotates certificates upon expiration.","long_desc":"","tags":[],"see_also":[]},"certificate_check_debug_mode":{"name":"certificate_check_debug_mode","type":"bool","level":"dev","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"FOR TESTING ONLY: This flag forces the certificate check instead of waiting for certificate_check_period.","long_desc":"","tags":[],"see_also":[]},"certificate_check_period":{"name":"certificate_check_period","type":"int","level":"advanced","flags":0,"default_value":"1","min":"0","max":"30","enum_allowed":[],"desc":"Specifies how often (in days) the certificate should be checked for validity.","long_desc":"","tags":[],"see_also":[]},"certificate_duration_days":{"name":"certificate_duration_days","type":"int","level":"advanced","flags":0,"default_value":"1095","min":"90","max":"3650","enum_allowed":[],"desc":"Specifies the duration of self-signed certificates generated and signed by the cephadm root CA","long_desc":"","tags":[],"see_also":[]},"certificate_renewal_threshold_days":{"name":"certificate_renewal_threshold_days","type":"int","level":"advanced","flags":0,"default_value":"30","min":"10","max":"90","enum_allowed":[],"desc":"Specifies the lead time in days to initiate certificate renewal before expiration.","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently Podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.28.1","min":"","max":"","enum_allowed":[],"desc":"Alertmanager container image","long_desc":"","tags":[],"see_also":[]},"container_image_alloy":{"name":"container_image_alloy","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/alloy:latest","min":"","max":"","enum_allowed":[],"desc":"Alloy container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"Elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:12.3.1","min":"","max":"","enum_allowed":[],"desc":"Grafana container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"Haproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_nginx":{"name":"container_image_nginx","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nginx:sclorg-nginx-126","min":"","max":"","enum_allowed":[],"desc":"Nginx container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.9.1","min":"","max":"","enum_allowed":[],"desc":"Node exporter container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.6","min":"","max":"","enum_allowed":[],"desc":"Nvmeof container image","long_desc":"","tags":[],"see_also":[]},"container_image_oauth2_proxy":{"name":"container_image_oauth2_proxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/oauth2-proxy/oauth2-proxy:v7.6.0","min":"","max":"","enum_allowed":[],"desc":"Oauth2 proxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v3.6.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba":{"name":"container_image_samba","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-server:devbuilds-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba_metrics":{"name":"container_image_samba_metrics","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-metrics:devbuilds-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba metrics container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"Snmp gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"Seconds for which to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"quay.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"Seconds for which to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"Seconds for which to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"Location of dashboards to include in Grafana","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"How frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether the hardware monitoring daemon be deployed on every host?","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at the debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"Max number of daemons per service per host: guard against mistakes and runaways","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"Max number of OSDs that will be drained simultaneously when OSDs are removed","long_desc":"","tags":[],"see_also":[]},"max_parallel_osd_upgrades":{"name":"max_parallel_osd_upgrades","type":"int","level":"advanced","flags":0,"default_value":"16","min":"","max":"","enum_allowed":[],"desc":"Maximum number of OSD daemons upgraded in parallel.","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"Mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default IP address for RedFish API (OOB management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"Location of alerts to include in Prometheus Alertmanager","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry URL for login purposes. This is not the default registry.","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"Cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Customized SSH config file to distribute to managed hosts","long_desc":"","tags":[],"see_also":[]},"ssh_keepalive_count_max":{"name":"ssh_keepalive_count_max","type":"int","level":"advanced","flags":0,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"How many sequential SSH connection liveness check failures before the host is marked offline","long_desc":"","tags":[],"see_also":[]},"ssh_keepalive_interval":{"name":"ssh_keepalive_interval","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"How often SSH connections are checked for liveness","long_desc":"","tags":[],"see_also":[]},"stray_daemon_check_interval":{"name":"stray_daemon_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"How frequently cephadm should check for the presence of stray daemons","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use the cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest to ensure that all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MANAGED_BY_CLUSTERS":{"name":"MANAGED_BY_CLUSTERS","type":"str","level":"advanced","flags":0,"default_value":"[]","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MULTICLUSTER_CONFIG":{"name":"MULTICLUSTER_CONFIG","type":"str","level":"advanced","flags":0,"default_value":"{}","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROM_ALERT_CREDENTIAL_CACHE_TTL":{"name":"PROM_ALERT_CREDENTIAL_CACHE_TTL","type":"int","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_HOSTNAME_PER_DAEMON":{"name":"RGW_HOSTNAME_PER_DAEMON","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crypto_caller":{"name":"crypto_caller","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sso_oauth2":{"name":"sso_oauth2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":true,"error_string":"","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"healthcheck_history_max_entries":{"name":"healthcheck_history_max_entries","type":"int","level":"advanced","flags":1,"default_value":"1000","min":"","max":"","enum_allowed":[],"desc":"Maximum number of health check history entries to keep","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"secondary_zone_period_retry_limit":{"name":"secondary_zone_period_retry_limit","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"RGW module period update retry limit for secondary site","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"prometheus_tls_secret_name":{"name":"prometheus_tls_secret_name","type":"str","level":"advanced","flags":0,"default_value":"rook-ceph-prometheus-server-tls","min":"","max":"","enum_allowed":[],"desc":"name of tls secret in k8s for prometheus","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"smb","can_run":true,"error_string":"","module_options":{"internal_store_backend":{"name":"internal_store_backend","type":"str","level":"dev","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"set internal store backend. for develoment and testing only","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"update_orchestration":{"name":"update_orchestration","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically update orchestration when smb resources are changed","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"pause_cloning":{"name":"pause_cloning","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Pause asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"pause_purging":{"name":"pause_purging","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Pause asynchronous subvolume purge threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"squid":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"tentacle":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"force_disabled_modules":{},"last_failure_osd_epoch":0,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"10.20.193.132:0","nonce":810771794}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"10.20.193.132:0","nonce":727128813}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"10.20.193.132:0","nonce":2544256071}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"10.20.193.132:0","nonce":1723804355}]}]} 2026-02-26T03:44:21.728 INFO:tasks.ceph.ceph_manager.ceph:mgr available! 2026-02-26T03:44:21.728 INFO:tasks.ceph.ceph_manager.ceph:waiting for all up 2026-02-26T03:44:21.729 DEBUG:teuthology.orchestra.run.trial132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2026-02-26T03:44:21.806 INFO:teuthology.orchestra.run.trial132.stderr:2026-02-26T03:44:21.805+0000 7efca810b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-26T03:44:21.808 INFO:teuthology.orchestra.run.trial132.stderr:2026-02-26T03:44:21.810+0000 7efca810b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-26T03:44:21.920 INFO:teuthology.orchestra.run.trial132.stdout: 2026-02-26T03:44:21.920 INFO:teuthology.orchestra.run.trial132.stdout:{"epoch":17,"fsid":"109d3300-c629-4972-9fab-d8a7123a9741","created":"2026-02-26T03:44:02.016900+0000","modified":"2026-02-26T03:44:21.227851+0000","last_up_change":"2026-02-26T03:44:13.195912+0000","last_in_change":"2026-02-26T03:44:03.779206+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-26T03:44:14.486080+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"13","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.9900000095367432,"score_stable":2.9900000095367432,"optimal_score":0.67000001668930054,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2026-02-26T03:44:17.770703+0000","flags":270337,"flags_names":"hashpspool,selfmanaged_snaps,crimson","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":2,"snap_epoch":17,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"de6b87e2-9687-4f88-b0cb-1e4799f76b1c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":10,"up_thru":14,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.132:6805","nonce":2385043468}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.132:6804","nonce":2385043468}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.132:6809","nonce":2385043468}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.132:6808","nonce":2385043468}]},"public_addr":"10.20.193.132:6805/2385043468","cluster_addr":"10.20.193.132:6804/2385043468","heartbeat_back_addr":"10.20.193.132:6809/2385043468","heartbeat_front_addr":"10.20.193.132:6808/2385043468","state":["exists","up"]},{"osd":1,"uuid":"7d7e4e45-9f58-40d0-b37a-0973e1bc7504","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":14,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.132:6803","nonce":546026106}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.132:6802","nonce":546026106}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.132:6815","nonce":546026106}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.132:6816","nonce":546026106}]},"public_addr":"10.20.193.132:6803/546026106","cluster_addr":"10.20.193.132:6802/546026106","heartbeat_back_addr":"10.20.193.132:6815/546026106","heartbeat_front_addr":"10.20.193.132:6816/546026106","state":["exists","up"]},{"osd":2,"uuid":"f83e5653-ffd0-41dd-b067-fa02dc7934e4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":10,"up_thru":14,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.132:6810","nonce":798308673}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.132:6811","nonce":798308673}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.132:6807","nonce":798308673}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.132:6806","nonce":798308673}]},"public_addr":"10.20.193.132:6810/798308673","cluster_addr":"10.20.193.132:6811/798308673","heartbeat_back_addr":"10.20.193.132:6807/798308673","heartbeat_front_addr":"10.20.193.132:6806/798308673","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":2,"snaps":[{"begin":2,"length":1}]}],"new_removed_snaps":[{"pool":2,"snaps":[{"begin":2,"length":1}]}],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-26T03:44:21.930 INFO:tasks.ceph.ceph_manager.ceph:all up! 2026-02-26T03:44:21.930 DEBUG:teuthology.orchestra.run.trial132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2026-02-26T03:44:22.007 INFO:teuthology.orchestra.run.trial132.stderr:2026-02-26T03:44:22.007+0000 7f379c4cb640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-26T03:44:22.010 INFO:teuthology.orchestra.run.trial132.stderr:2026-02-26T03:44:22.011+0000 7f379c4cb640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-26T03:44:22.120 INFO:teuthology.orchestra.run.trial132.stdout: 2026-02-26T03:44:22.120 INFO:teuthology.orchestra.run.trial132.stdout:{"epoch":17,"fsid":"109d3300-c629-4972-9fab-d8a7123a9741","created":"2026-02-26T03:44:02.016900+0000","modified":"2026-02-26T03:44:21.227851+0000","last_up_change":"2026-02-26T03:44:13.195912+0000","last_in_change":"2026-02-26T03:44:03.779206+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-26T03:44:14.486080+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"13","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.9900000095367432,"score_stable":2.9900000095367432,"optimal_score":0.67000001668930054,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2026-02-26T03:44:17.770703+0000","flags":270337,"flags_names":"hashpspool,selfmanaged_snaps,crimson","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":2,"snap_epoch":17,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"de6b87e2-9687-4f88-b0cb-1e4799f76b1c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":10,"up_thru":14,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.132:6805","nonce":2385043468}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.132:6804","nonce":2385043468}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.132:6809","nonce":2385043468}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.132:6808","nonce":2385043468}]},"public_addr":"10.20.193.132:6805/2385043468","cluster_addr":"10.20.193.132:6804/2385043468","heartbeat_back_addr":"10.20.193.132:6809/2385043468","heartbeat_front_addr":"10.20.193.132:6808/2385043468","state":["exists","up"]},{"osd":1,"uuid":"7d7e4e45-9f58-40d0-b37a-0973e1bc7504","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":14,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.132:6803","nonce":546026106}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.132:6802","nonce":546026106}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.132:6815","nonce":546026106}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.132:6816","nonce":546026106}]},"public_addr":"10.20.193.132:6803/546026106","cluster_addr":"10.20.193.132:6802/546026106","heartbeat_back_addr":"10.20.193.132:6815/546026106","heartbeat_front_addr":"10.20.193.132:6816/546026106","state":["exists","up"]},{"osd":2,"uuid":"f83e5653-ffd0-41dd-b067-fa02dc7934e4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":10,"up_thru":14,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.132:6810","nonce":798308673}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.132:6811","nonce":798308673}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.132:6807","nonce":798308673}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.132:6806","nonce":798308673}]},"public_addr":"10.20.193.132:6810/798308673","cluster_addr":"10.20.193.132:6811/798308673","heartbeat_back_addr":"10.20.193.132:6807/798308673","heartbeat_front_addr":"10.20.193.132:6806/798308673","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":2,"snaps":[{"begin":2,"length":1}]}],"new_removed_snaps":[{"pool":2,"snaps":[{"begin":2,"length":1}]}],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-26T03:44:22.130 DEBUG:teuthology.orchestra.run.trial132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2026-02-26T03:44:22.131 DEBUG:teuthology.orchestra.run.trial132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2026-02-26T03:44:22.131 DEBUG:teuthology.orchestra.run.trial132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2026-02-26T03:44:22.212 INFO:teuthology.orchestra.run.trial132.stderr:2026-02-26T03:44:22.211+0000 7f119125a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-26T03:44:22.213 INFO:teuthology.orchestra.run.trial132.stderr:2026-02-26T03:44:22.212+0000 7efd247e4640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-26T03:44:22.213 INFO:teuthology.orchestra.run.trial132.stderr:2026-02-26T03:44:22.212+0000 7fd11b3a4640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-26T03:44:22.215 INFO:teuthology.orchestra.run.trial132.stderr:2026-02-26T03:44:22.216+0000 7efd247e4640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-26T03:44:22.215 INFO:teuthology.orchestra.run.trial132.stderr:2026-02-26T03:44:22.216+0000 7f119125a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-26T03:44:22.215 INFO:teuthology.orchestra.run.trial132.stderr:2026-02-26T03:44:22.217+0000 7fd11b3a4640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-26T03:44:22.232 INFO:teuthology.orchestra.run.trial132.stdout:42949672969 2026-02-26T03:44:22.233 DEBUG:teuthology.orchestra.run.trial132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2026-02-26T03:44:22.234 INFO:teuthology.orchestra.run.trial132.stdout:38654705674 2026-02-26T03:44:22.235 DEBUG:teuthology.orchestra.run.trial132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2026-02-26T03:44:22.236 INFO:teuthology.orchestra.run.trial132.stdout:42949672969 2026-02-26T03:44:22.236 DEBUG:teuthology.orchestra.run.trial132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2026-02-26T03:44:22.355 INFO:teuthology.orchestra.run.trial132.stderr:2026-02-26T03:44:22.354+0000 7f50c765a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-26T03:44:22.355 INFO:teuthology.orchestra.run.trial132.stderr:2026-02-26T03:44:22.354+0000 7fa5af41a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-26T03:44:22.356 INFO:teuthology.orchestra.run.trial132.stderr:2026-02-26T03:44:22.354+0000 7f1d26606640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-26T03:44:22.357 INFO:teuthology.orchestra.run.trial132.stderr:2026-02-26T03:44:22.359+0000 7fa5af41a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-26T03:44:22.358 INFO:teuthology.orchestra.run.trial132.stderr:2026-02-26T03:44:22.359+0000 7f50c765a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-26T03:44:22.358 INFO:teuthology.orchestra.run.trial132.stderr:2026-02-26T03:44:22.360+0000 7f1d26606640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-26T03:44:22.469 INFO:teuthology.orchestra.run.trial132.stdout:42949672969 2026-02-26T03:44:22.474 INFO:teuthology.orchestra.run.trial132.stdout:38654705674 2026-02-26T03:44:22.478 INFO:teuthology.orchestra.run.trial132.stdout:42949672969 2026-02-26T03:44:22.479 INFO:tasks.ceph.ceph_manager.ceph:need seq 42949672969 got 42949672969 for osd.0 2026-02-26T03:44:22.480 DEBUG:teuthology.parallel:result is None 2026-02-26T03:44:22.485 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654705674 got 38654705674 for osd.1 2026-02-26T03:44:22.486 DEBUG:teuthology.parallel:result is None 2026-02-26T03:44:22.489 INFO:tasks.ceph.ceph_manager.ceph:need seq 42949672969 got 42949672969 for osd.2 2026-02-26T03:44:22.489 DEBUG:teuthology.parallel:result is None 2026-02-26T03:44:22.489 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2026-02-26T03:44:22.490 DEBUG:teuthology.orchestra.run.trial132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-26T03:44:22.604 INFO:teuthology.orchestra.run.trial132.stderr:2026-02-26T03:44:22.603+0000 7fa3fb45a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-26T03:44:22.606 INFO:teuthology.orchestra.run.trial132.stderr:2026-02-26T03:44:22.608+0000 7fa3fb45a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-26T03:44:22.714 INFO:teuthology.orchestra.run.trial132.stdout: 2026-02-26T03:44:22.715 INFO:teuthology.orchestra.run.trial132.stderr:dumped all 2026-02-26T03:44:22.723 INFO:teuthology.orchestra.run.trial132.stdout:{"pg_ready":true,"pg_map":{"version":15,"stamp":"2026-02-26T03:44:22.470110+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":53,"num_read_kb":46,"num_write":81,"num_write_kb":1108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":55,"ondisk_log_size":55,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":3,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":156499968,"kb_used":2540,"kb_used_data":2540,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":156497428,"statfs":{"total":160255967232,"available":160253366272,"internally_reserved":0,"allocated":2600960,"data_stored":2600960,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.253443"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":15,"reported_epoch":17,"state":"active+clean","last_fresh":"2026-02-26T03:44:21.240834+0000","last_change":"2026-02-26T03:44:21.244227+0000","last_active":"2026-02-26T03:44:21.240834+0000","last_peered":"2026-02-26T03:44:21.240834+0000","last_clean":"2026-02-26T03:44:21.240834+0000","last_became_active":"2026-02-26T03:44:19.263201+0000","last_became_peered":"2026-02-26T03:44:19.263201+0000","last_unstale":"2026-02-26T03:44:21.240834+0000","last_undegraded":"2026-02-26T03:44:21.240834+0000","last_fullsized":"2026-02-26T03:44:21.240834+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-26T03:44:18.219221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-26T03:44:18.219221+0000","last_clean_scrub_stamp":"2026-02-26T03:44:18.219221+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":15,"reported_epoch":17,"state":"active+clean","last_fresh":"2026-02-26T03:44:21.243071+0000","last_change":"2026-02-26T03:44:21.247517+0000","last_active":"2026-02-26T03:44:21.243071+0000","last_peered":"2026-02-26T03:44:21.243071+0000","last_clean":"2026-02-26T03:44:21.243071+0000","last_became_active":"2026-02-26T03:44:19.261313+0000","last_became_peered":"2026-02-26T03:44:19.261313+0000","last_unstale":"2026-02-26T03:44:21.243071+0000","last_undegraded":"2026-02-26T03:44:21.243071+0000","last_fullsized":"2026-02-26T03:44:21.243071+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-26T03:44:18.219221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-26T03:44:18.219221+0000","last_clean_scrub_stamp":"2026-02-26T03:44:18.219221+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":15,"reported_epoch":17,"state":"active+clean","last_fresh":"2026-02-26T03:44:21.243595+0000","last_change":"2026-02-26T03:44:21.246959+0000","last_active":"2026-02-26T03:44:21.243595+0000","last_peered":"2026-02-26T03:44:21.243595+0000","last_clean":"2026-02-26T03:44:21.243595+0000","last_became_active":"2026-02-26T03:44:19.268173+0000","last_became_peered":"2026-02-26T03:44:19.268173+0000","last_unstale":"2026-02-26T03:44:21.243595+0000","last_undegraded":"2026-02-26T03:44:21.243595+0000","last_fullsized":"2026-02-26T03:44:21.243595+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-26T03:44:18.219221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-26T03:44:18.219221+0000","last_clean_scrub_stamp":"2026-02-26T03:44:18.219221+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":15,"reported_epoch":17,"state":"active+clean","last_fresh":"2026-02-26T03:44:21.242480+0000","last_change":"2026-02-26T03:44:21.247143+0000","last_active":"2026-02-26T03:44:21.242480+0000","last_peered":"2026-02-26T03:44:21.242480+0000","last_clean":"2026-02-26T03:44:21.242480+0000","last_became_active":"2026-02-26T03:44:19.271270+0000","last_became_peered":"2026-02-26T03:44:19.271270+0000","last_unstale":"2026-02-26T03:44:21.242480+0000","last_undegraded":"2026-02-26T03:44:21.242480+0000","last_fullsized":"2026-02-26T03:44:21.242480+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-26T03:44:18.219221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-26T03:44:18.219221+0000","last_clean_scrub_stamp":"2026-02-26T03:44:18.219221+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"17'2","reported_seq":17,"reported_epoch":17,"state":"active+clean","last_fresh":"2026-02-26T03:44:21.243842+0000","last_change":"2026-02-26T03:44:21.245047+0000","last_active":"2026-02-26T03:44:21.243842+0000","last_peered":"2026-02-26T03:44:21.243842+0000","last_clean":"2026-02-26T03:44:21.243842+0000","last_became_active":"2026-02-26T03:44:19.276841+0000","last_became_peered":"2026-02-26T03:44:19.276841+0000","last_unstale":"2026-02-26T03:44:21.243842+0000","last_undegraded":"2026-02-26T03:44:21.243842+0000","last_fullsized":"2026-02-26T03:44:21.243842+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-26T03:44:18.219221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-26T03:44:18.219221+0000","last_clean_scrub_stamp":"2026-02-26T03:44:18.219221+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":15,"reported_epoch":17,"state":"active+clean","last_fresh":"2026-02-26T03:44:21.240126+0000","last_change":"2026-02-26T03:44:21.243284+0000","last_active":"2026-02-26T03:44:21.240126+0000","last_peered":"2026-02-26T03:44:21.240126+0000","last_clean":"2026-02-26T03:44:21.240126+0000","last_became_active":"2026-02-26T03:44:19.283254+0000","last_became_peered":"2026-02-26T03:44:19.283254+0000","last_unstale":"2026-02-26T03:44:21.240126+0000","last_undegraded":"2026-02-26T03:44:21.240126+0000","last_fullsized":"2026-02-26T03:44:21.240126+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-26T03:44:18.219221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-26T03:44:18.219221+0000","last_clean_scrub_stamp":"2026-02-26T03:44:18.219221+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":15,"reported_epoch":17,"state":"active+clean","last_fresh":"2026-02-26T03:44:21.237390+0000","last_change":"2026-02-26T03:44:21.240663+0000","last_active":"2026-02-26T03:44:21.237390+0000","last_peered":"2026-02-26T03:44:21.237390+0000","last_clean":"2026-02-26T03:44:21.237390+0000","last_became_active":"2026-02-26T03:44:19.283733+0000","last_became_peered":"2026-02-26T03:44:19.283733+0000","last_unstale":"2026-02-26T03:44:21.237390+0000","last_undegraded":"2026-02-26T03:44:21.237390+0000","last_fullsized":"2026-02-26T03:44:21.237390+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-26T03:44:18.219221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-26T03:44:18.219221+0000","last_clean_scrub_stamp":"2026-02-26T03:44:18.219221+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"15'1","reported_seq":16,"reported_epoch":17,"state":"active+clean","last_fresh":"2026-02-26T03:44:21.242435+0000","last_change":"2026-02-26T03:44:21.246671+0000","last_active":"2026-02-26T03:44:21.242435+0000","last_peered":"2026-02-26T03:44:21.242435+0000","last_clean":"2026-02-26T03:44:21.242435+0000","last_became_active":"2026-02-26T03:44:19.259527+0000","last_became_peered":"2026-02-26T03:44:19.259527+0000","last_unstale":"2026-02-26T03:44:21.242435+0000","last_undegraded":"2026-02-26T03:44:21.242435+0000","last_fullsized":"2026-02-26T03:44:21.242435+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-26T03:44:18.219221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-26T03:44:18.219221+0000","last_clean_scrub_stamp":"2026-02-26T03:44:18.219221+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"12'52","reported_seq":90,"reported_epoch":17,"state":"active+clean","last_fresh":"2026-02-26T03:44:21.240559+0000","last_change":"2026-02-26T03:44:16.250074+0000","last_active":"2026-02-26T03:44:21.240559+0000","last_peered":"2026-02-26T03:44:21.240559+0000","last_clean":"2026-02-26T03:44:21.240559+0000","last_became_active":"2026-02-26T03:44:16.240654+0000","last_became_peered":"2026-02-26T03:44:16.240654+0000","last_unstale":"2026-02-26T03:44:21.240559+0000","last_undegraded":"2026-02-26T03:44:21.240559+0000","last_fullsized":"2026-02-26T03:44:21.240559+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-26T03:44:15.206229+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-26T03:44:15.206229+0000","last_clean_scrub_stamp":"2026-02-26T03:44:15.206229+0000","objects_scrubbed":0,"log_size":52,"log_dups_size":0,"ondisk_log_size":52,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":53,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":160255967232,"available":160253366272,"internally_reserved":0,"allocated":2600960,"data_stored":2600960,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":53,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":53418655744,"available":53417480192,"internally_reserved":0,"allocated":1175552,"data_stored":1175552,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":52,"ondisk_log_size":52,"up":2,"acting":2,"num_store_stats":1}],"osd_stats":[{"osd":2,"up_from":10,"seq":42949672969,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52166656,"kb_used":316,"kb_used_data":316,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":52166340,"statfs":{"total":53418655744,"available":53418332160,"internally_reserved":0,"allocated":323584,"data_stored":323584,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":10,"seq":42949672969,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52166656,"kb_used":1076,"kb_used_data":1076,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":52165580,"statfs":{"total":53418655744,"available":53417553920,"internally_reserved":0,"allocated":1101824,"data_stored":1101824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":9,"seq":38654705674,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52166656,"kb_used":1148,"kb_used_data":1148,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":52165508,"statfs":{"total":53418655744,"available":53417480192,"internally_reserved":0,"allocated":1175552,"data_stored":1175552,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":1,"total":53418655744,"available":53417480192,"internally_reserved":0,"allocated":1175552,"data_stored":1175552,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":53418655744,"available":53417553920,"internally_reserved":0,"allocated":1101824,"data_stored":1101824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":53418655744,"available":53417480192,"internally_reserved":0,"allocated":1175552,"data_stored":1175552,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":53418655744,"available":53418332160,"internally_reserved":0,"allocated":323584,"data_stored":323584,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-26T03:44:22.724 DEBUG:teuthology.orchestra.run.trial132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-26T03:44:22.800 INFO:teuthology.orchestra.run.trial132.stderr:2026-02-26T03:44:22.799+0000 7f36bd70b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-26T03:44:22.802 INFO:teuthology.orchestra.run.trial132.stderr:2026-02-26T03:44:22.804+0000 7f36bd70b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-26T03:44:22.914 INFO:teuthology.orchestra.run.trial132.stdout: 2026-02-26T03:44:22.915 INFO:teuthology.orchestra.run.trial132.stderr:dumped all 2026-02-26T03:44:22.924 INFO:teuthology.orchestra.run.trial132.stdout:{"pg_ready":true,"pg_map":{"version":15,"stamp":"2026-02-26T03:44:22.470110+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":53,"num_read_kb":46,"num_write":81,"num_write_kb":1108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":55,"ondisk_log_size":55,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":3,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":156499968,"kb_used":2540,"kb_used_data":2540,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":156497428,"statfs":{"total":160255967232,"available":160253366272,"internally_reserved":0,"allocated":2600960,"data_stored":2600960,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.253443"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":15,"reported_epoch":17,"state":"active+clean","last_fresh":"2026-02-26T03:44:21.240834+0000","last_change":"2026-02-26T03:44:21.244227+0000","last_active":"2026-02-26T03:44:21.240834+0000","last_peered":"2026-02-26T03:44:21.240834+0000","last_clean":"2026-02-26T03:44:21.240834+0000","last_became_active":"2026-02-26T03:44:19.263201+0000","last_became_peered":"2026-02-26T03:44:19.263201+0000","last_unstale":"2026-02-26T03:44:21.240834+0000","last_undegraded":"2026-02-26T03:44:21.240834+0000","last_fullsized":"2026-02-26T03:44:21.240834+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-26T03:44:18.219221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-26T03:44:18.219221+0000","last_clean_scrub_stamp":"2026-02-26T03:44:18.219221+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":15,"reported_epoch":17,"state":"active+clean","last_fresh":"2026-02-26T03:44:21.243071+0000","last_change":"2026-02-26T03:44:21.247517+0000","last_active":"2026-02-26T03:44:21.243071+0000","last_peered":"2026-02-26T03:44:21.243071+0000","last_clean":"2026-02-26T03:44:21.243071+0000","last_became_active":"2026-02-26T03:44:19.261313+0000","last_became_peered":"2026-02-26T03:44:19.261313+0000","last_unstale":"2026-02-26T03:44:21.243071+0000","last_undegraded":"2026-02-26T03:44:21.243071+0000","last_fullsized":"2026-02-26T03:44:21.243071+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-26T03:44:18.219221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-26T03:44:18.219221+0000","last_clean_scrub_stamp":"2026-02-26T03:44:18.219221+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":15,"reported_epoch":17,"state":"active+clean","last_fresh":"2026-02-26T03:44:21.243595+0000","last_change":"2026-02-26T03:44:21.246959+0000","last_active":"2026-02-26T03:44:21.243595+0000","last_peered":"2026-02-26T03:44:21.243595+0000","last_clean":"2026-02-26T03:44:21.243595+0000","last_became_active":"2026-02-26T03:44:19.268173+0000","last_became_peered":"2026-02-26T03:44:19.268173+0000","last_unstale":"2026-02-26T03:44:21.243595+0000","last_undegraded":"2026-02-26T03:44:21.243595+0000","last_fullsized":"2026-02-26T03:44:21.243595+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-26T03:44:18.219221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-26T03:44:18.219221+0000","last_clean_scrub_stamp":"2026-02-26T03:44:18.219221+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":15,"reported_epoch":17,"state":"active+clean","last_fresh":"2026-02-26T03:44:21.242480+0000","last_change":"2026-02-26T03:44:21.247143+0000","last_active":"2026-02-26T03:44:21.242480+0000","last_peered":"2026-02-26T03:44:21.242480+0000","last_clean":"2026-02-26T03:44:21.242480+0000","last_became_active":"2026-02-26T03:44:19.271270+0000","last_became_peered":"2026-02-26T03:44:19.271270+0000","last_unstale":"2026-02-26T03:44:21.242480+0000","last_undegraded":"2026-02-26T03:44:21.242480+0000","last_fullsized":"2026-02-26T03:44:21.242480+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-26T03:44:18.219221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-26T03:44:18.219221+0000","last_clean_scrub_stamp":"2026-02-26T03:44:18.219221+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"17'2","reported_seq":17,"reported_epoch":17,"state":"active+clean","last_fresh":"2026-02-26T03:44:21.243842+0000","last_change":"2026-02-26T03:44:21.245047+0000","last_active":"2026-02-26T03:44:21.243842+0000","last_peered":"2026-02-26T03:44:21.243842+0000","last_clean":"2026-02-26T03:44:21.243842+0000","last_became_active":"2026-02-26T03:44:19.276841+0000","last_became_peered":"2026-02-26T03:44:19.276841+0000","last_unstale":"2026-02-26T03:44:21.243842+0000","last_undegraded":"2026-02-26T03:44:21.243842+0000","last_fullsized":"2026-02-26T03:44:21.243842+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-26T03:44:18.219221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-26T03:44:18.219221+0000","last_clean_scrub_stamp":"2026-02-26T03:44:18.219221+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":15,"reported_epoch":17,"state":"active+clean","last_fresh":"2026-02-26T03:44:21.240126+0000","last_change":"2026-02-26T03:44:21.243284+0000","last_active":"2026-02-26T03:44:21.240126+0000","last_peered":"2026-02-26T03:44:21.240126+0000","last_clean":"2026-02-26T03:44:21.240126+0000","last_became_active":"2026-02-26T03:44:19.283254+0000","last_became_peered":"2026-02-26T03:44:19.283254+0000","last_unstale":"2026-02-26T03:44:21.240126+0000","last_undegraded":"2026-02-26T03:44:21.240126+0000","last_fullsized":"2026-02-26T03:44:21.240126+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-26T03:44:18.219221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-26T03:44:18.219221+0000","last_clean_scrub_stamp":"2026-02-26T03:44:18.219221+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":15,"reported_epoch":17,"state":"active+clean","last_fresh":"2026-02-26T03:44:21.237390+0000","last_change":"2026-02-26T03:44:21.240663+0000","last_active":"2026-02-26T03:44:21.237390+0000","last_peered":"2026-02-26T03:44:21.237390+0000","last_clean":"2026-02-26T03:44:21.237390+0000","last_became_active":"2026-02-26T03:44:19.283733+0000","last_became_peered":"2026-02-26T03:44:19.283733+0000","last_unstale":"2026-02-26T03:44:21.237390+0000","last_undegraded":"2026-02-26T03:44:21.237390+0000","last_fullsized":"2026-02-26T03:44:21.237390+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-26T03:44:18.219221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-26T03:44:18.219221+0000","last_clean_scrub_stamp":"2026-02-26T03:44:18.219221+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"15'1","reported_seq":16,"reported_epoch":17,"state":"active+clean","last_fresh":"2026-02-26T03:44:21.242435+0000","last_change":"2026-02-26T03:44:21.246671+0000","last_active":"2026-02-26T03:44:21.242435+0000","last_peered":"2026-02-26T03:44:21.242435+0000","last_clean":"2026-02-26T03:44:21.242435+0000","last_became_active":"2026-02-26T03:44:19.259527+0000","last_became_peered":"2026-02-26T03:44:19.259527+0000","last_unstale":"2026-02-26T03:44:21.242435+0000","last_undegraded":"2026-02-26T03:44:21.242435+0000","last_fullsized":"2026-02-26T03:44:21.242435+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-26T03:44:18.219221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-26T03:44:18.219221+0000","last_clean_scrub_stamp":"2026-02-26T03:44:18.219221+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"12'52","reported_seq":90,"reported_epoch":17,"state":"active+clean","last_fresh":"2026-02-26T03:44:21.240559+0000","last_change":"2026-02-26T03:44:16.250074+0000","last_active":"2026-02-26T03:44:21.240559+0000","last_peered":"2026-02-26T03:44:21.240559+0000","last_clean":"2026-02-26T03:44:21.240559+0000","last_became_active":"2026-02-26T03:44:16.240654+0000","last_became_peered":"2026-02-26T03:44:16.240654+0000","last_unstale":"2026-02-26T03:44:21.240559+0000","last_undegraded":"2026-02-26T03:44:21.240559+0000","last_fullsized":"2026-02-26T03:44:21.240559+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-26T03:44:15.206229+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-26T03:44:15.206229+0000","last_clean_scrub_stamp":"2026-02-26T03:44:15.206229+0000","objects_scrubbed":0,"log_size":52,"log_dups_size":0,"ondisk_log_size":52,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":53,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":160255967232,"available":160253366272,"internally_reserved":0,"allocated":2600960,"data_stored":2600960,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":53,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":53418655744,"available":53417480192,"internally_reserved":0,"allocated":1175552,"data_stored":1175552,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":52,"ondisk_log_size":52,"up":2,"acting":2,"num_store_stats":1}],"osd_stats":[{"osd":2,"up_from":10,"seq":42949672969,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52166656,"kb_used":316,"kb_used_data":316,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":52166340,"statfs":{"total":53418655744,"available":53418332160,"internally_reserved":0,"allocated":323584,"data_stored":323584,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":10,"seq":42949672969,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52166656,"kb_used":1076,"kb_used_data":1076,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":52165580,"statfs":{"total":53418655744,"available":53417553920,"internally_reserved":0,"allocated":1101824,"data_stored":1101824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":9,"seq":38654705674,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52166656,"kb_used":1148,"kb_used_data":1148,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":52165508,"statfs":{"total":53418655744,"available":53417480192,"internally_reserved":0,"allocated":1175552,"data_stored":1175552,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":1,"total":53418655744,"available":53417480192,"internally_reserved":0,"allocated":1175552,"data_stored":1175552,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":53418655744,"available":53417553920,"internally_reserved":0,"allocated":1101824,"data_stored":1101824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":53418655744,"available":53417480192,"internally_reserved":0,"allocated":1175552,"data_stored":1175552,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":53418655744,"available":53418332160,"internally_reserved":0,"allocated":323584,"data_stored":323584,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-26T03:44:22.925 INFO:tasks.ceph.ceph_manager.ceph:clean! 2026-02-26T03:44:22.925 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2026-02-26T03:44:22.925 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy 2026-02-26T03:44:22.925 DEBUG:teuthology.orchestra.run.trial132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2026-02-26T03:44:23.000 INFO:teuthology.orchestra.run.trial132.stderr:2026-02-26T03:44:23.001+0000 7fc66decb640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-26T03:44:23.003 INFO:teuthology.orchestra.run.trial132.stderr:2026-02-26T03:44:23.004+0000 7fc66decb640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-26T03:44:23.142 INFO:teuthology.orchestra.run.trial132.stdout: 2026-02-26T03:44:23.142 INFO:teuthology.orchestra.run.trial132.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2026-02-26T03:44:23.151 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy done 2026-02-26T03:44:23.152 INFO:teuthology.run_tasks:Running task workunit... 2026-02-26T03:44:23.177 INFO:tasks.workunit:Pulling workunits from ref 1f87b69b1a8041112e26ae0ea3c0a2a0fd14d996 2026-02-26T03:44:23.177 INFO:tasks.workunit:Making a separate scratch dir for every client... 2026-02-26T03:44:23.178 DEBUG:teuthology.orchestra.run.trial132:> stat -- /home/ubuntu/cephtest/mnt.0 2026-02-26T03:44:23.194 DEBUG:teuthology.orchestra.run:got remote process result: 1 2026-02-26T03:44:23.195 INFO:teuthology.orchestra.run.trial132.stderr:stat: cannot statx '/home/ubuntu/cephtest/mnt.0': No such file or directory 2026-02-26T03:44:23.195 DEBUG:teuthology.orchestra.run.trial132:> mkdir -- /home/ubuntu/cephtest/mnt.0 2026-02-26T03:44:23.251 INFO:tasks.workunit:Created dir /home/ubuntu/cephtest/mnt.0 2026-02-26T03:44:23.252 DEBUG:teuthology.orchestra.run.trial132:> cd -- /home/ubuntu/cephtest/mnt.0 && mkdir -- client.0 2026-02-26T03:44:23.307 INFO:tasks.workunit:timeout=3h 2026-02-26T03:44:23.308 INFO:tasks.workunit:cleanup=True 2026-02-26T03:44:23.308 DEBUG:teuthology.orchestra.run.trial132:> rm -rf /home/ubuntu/cephtest/clone.client.0 && git clone https://github.com/ceph/ceph.git /home/ubuntu/cephtest/clone.client.0 && cd /home/ubuntu/cephtest/clone.client.0 && git checkout 1f87b69b1a8041112e26ae0ea3c0a2a0fd14d996 2026-02-26T03:44:23.371 INFO:tasks.workunit.client.0.trial132.stderr:Cloning into '/home/ubuntu/cephtest/clone.client.0'... 2026-02-26T03:44:46.810 INFO:tasks.workunit.client.0.trial132.stderr:Note: switching to '1f87b69b1a8041112e26ae0ea3c0a2a0fd14d996'. 2026-02-26T03:44:46.810 INFO:tasks.workunit.client.0.trial132.stderr: 2026-02-26T03:44:46.810 INFO:tasks.workunit.client.0.trial132.stderr:You are in 'detached HEAD' state. You can look around, make experimental 2026-02-26T03:44:46.810 INFO:tasks.workunit.client.0.trial132.stderr:changes and commit them, and you can discard any commits you make in this 2026-02-26T03:44:46.811 INFO:tasks.workunit.client.0.trial132.stderr:state without impacting any branches by switching back to a branch. 2026-02-26T03:44:46.811 INFO:tasks.workunit.client.0.trial132.stderr: 2026-02-26T03:44:46.811 INFO:tasks.workunit.client.0.trial132.stderr:If you want to create a new branch to retain commits you create, you may 2026-02-26T03:44:46.811 INFO:tasks.workunit.client.0.trial132.stderr:do so (now or later) by using -c with the switch command. Example: 2026-02-26T03:44:46.811 INFO:tasks.workunit.client.0.trial132.stderr: 2026-02-26T03:44:46.811 INFO:tasks.workunit.client.0.trial132.stderr: git switch -c 2026-02-26T03:44:46.811 INFO:tasks.workunit.client.0.trial132.stderr: 2026-02-26T03:44:46.811 INFO:tasks.workunit.client.0.trial132.stderr:Or undo this operation with: 2026-02-26T03:44:46.811 INFO:tasks.workunit.client.0.trial132.stderr: 2026-02-26T03:44:46.811 INFO:tasks.workunit.client.0.trial132.stderr: git switch - 2026-02-26T03:44:46.811 INFO:tasks.workunit.client.0.trial132.stderr: 2026-02-26T03:44:46.812 INFO:tasks.workunit.client.0.trial132.stderr:Turn off this advice by setting config variable advice.detachedHead to false 2026-02-26T03:44:46.812 INFO:tasks.workunit.client.0.trial132.stderr: 2026-02-26T03:44:46.812 INFO:tasks.workunit.client.0.trial132.stderr:HEAD is now at 1f87b69b1a8 Merge pull request #67522 from rhcs-dashboard/notification-banner-simple-mode 2026-02-26T03:44:46.813 DEBUG:teuthology.orchestra.run.trial132:> cd -- /home/ubuntu/cephtest/clone.client.0/qa/workunits && if test -e Makefile ; then make ; fi && find -executable -type f -printf '%P\0' >/home/ubuntu/cephtest/workunits.list.client.0 2026-02-26T03:44:46.871 INFO:tasks.workunit.client.0.trial132.stdout:for d in direct_io fs ; do ( cd $d ; make all ) ; done 2026-02-26T03:44:46.873 INFO:tasks.workunit.client.0.trial132.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2026-02-26T03:44:46.873 INFO:tasks.workunit.client.0.trial132.stdout:cc -Wall -Wextra -D_GNU_SOURCE direct_io_test.c -o direct_io_test 2026-02-26T03:44:46.964 INFO:tasks.workunit.client.0.trial132.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_sync_io.c -o test_sync_io 2026-02-26T03:44:46.994 INFO:tasks.workunit.client.0.trial132.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_short_dio_read.c -o test_short_dio_read 2026-02-26T03:44:47.023 INFO:tasks.workunit.client.0.trial132.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2026-02-26T03:44:47.025 INFO:tasks.workunit.client.0.trial132.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2026-02-26T03:44:47.025 INFO:tasks.workunit.client.0.trial132.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_o_trunc.c -o test_o_trunc 2026-02-26T03:44:47.050 INFO:tasks.workunit.client.0.trial132.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2026-02-26T03:44:47.054 DEBUG:teuthology.orchestra.run.trial132:> set -ex 2026-02-26T03:44:47.054 DEBUG:teuthology.orchestra.run.trial132:> dd if=/home/ubuntu/cephtest/workunits.list.client.0 of=/dev/stdout 2026-02-26T03:44:47.111 INFO:tasks.workunit:Running workunits matching rbd/test_lock_fence.sh on client.0... 2026-02-26T03:44:47.113 INFO:tasks.workunit:Running workunit rbd/test_lock_fence.sh... 2026-02-26T03:44:47.114 DEBUG:teuthology.orchestra.run.trial132:workunit test rbd/test_lock_fence.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=1f87b69b1a8041112e26ae0ea3c0a2a0fd14d996 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 CRIMSON_COMPAT=1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/rbd/test_lock_fence.sh 2026-02-26T03:44:47.171 INFO:tasks.workunit.client.0.trial132.stderr:+ IMAGE=rbdrw-image 2026-02-26T03:44:47.171 INFO:tasks.workunit.client.0.trial132.stderr:+ LOCKID=rbdrw 2026-02-26T03:44:47.171 INFO:tasks.workunit.client.0.trial132.stderr:++ dirname /home/ubuntu/cephtest/clone.client.0/qa/workunits/rbd/test_lock_fence.sh 2026-02-26T03:44:47.172 INFO:tasks.workunit.client.0.trial132.stderr:+ RELPATH=/home/ubuntu/cephtest/clone.client.0/qa/workunits/rbd/../../../src/test/librbd 2026-02-26T03:44:47.172 INFO:tasks.workunit.client.0.trial132.stderr:+ RBDRW=/home/ubuntu/cephtest/clone.client.0/qa/workunits/rbd/../../../src/test/librbd/rbdrw.py 2026-02-26T03:44:47.172 INFO:tasks.workunit.client.0.trial132.stderr:+ rbd create rbdrw-image --size 10 --image-format 2 --image-shared 2026-02-26T03:44:47.199 INFO:tasks.workunit.client.0.trial132.stderr:2026-02-26T03:44:47.200+0000 7f158dce4240 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-26T03:44:47.199 INFO:tasks.workunit.client.0.trial132.stderr:2026-02-26T03:44:47.200+0000 7f158dce4240 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-26T03:44:47.202 INFO:tasks.workunit.client.0.trial132.stderr:2026-02-26T03:44:47.203+0000 7f158dce4240 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-26T03:44:47.263 INFO:tasks.workunit.client.0.trial132.stderr:+ iochild=41991 2026-02-26T03:44:47.264 INFO:tasks.workunit.client.0.trial132.stderr:+ LOCKS='[]' 2026-02-26T03:44:47.264 INFO:tasks.workunit.client.0.trial132.stderr:+ '[' '[]' == '[]' ']' 2026-02-26T03:44:47.264 INFO:tasks.workunit.client.0.trial132.stderr:+ python3 /home/ubuntu/cephtest/clone.client.0/qa/workunits/rbd/../../../src/test/librbd/rbdrw.py rbdrw-image rbdrw 2026-02-26T03:44:47.264 INFO:tasks.workunit.client.0.trial132.stderr:++ rbd lock list rbdrw-image --format json 2026-02-26T03:44:47.289 INFO:tasks.workunit.client.0.trial132.stderr:2026-02-26T03:44:47.291+0000 7f1f09f4a240 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-26T03:44:47.289 INFO:tasks.workunit.client.0.trial132.stderr:2026-02-26T03:44:47.291+0000 7f1f09f4a240 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-26T03:44:47.292 INFO:tasks.workunit.client.0.trial132.stderr:2026-02-26T03:44:47.294+0000 7f1f09f4a240 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-26T03:44:47.362 INFO:tasks.workunit.client.0.trial132.stderr:+ LOCKS='[]' 2026-02-26T03:44:47.362 INFO:tasks.workunit.client.0.trial132.stderr:+ sleep 1 2026-02-26T03:44:48.363 INFO:tasks.workunit.client.0.trial132.stderr:+ '[' '[]' == '[]' ']' 2026-02-26T03:44:48.364 INFO:tasks.workunit.client.0.trial132.stderr:++ rbd lock list rbdrw-image --format json 2026-02-26T03:44:48.392 INFO:tasks.workunit.client.0.trial132.stderr:2026-02-26T03:44:48.394+0000 7fb9eb2f6240 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-26T03:44:48.393 INFO:tasks.workunit.client.0.trial132.stderr:2026-02-26T03:44:48.394+0000 7fb9eb2f6240 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-26T03:44:48.396 INFO:tasks.workunit.client.0.trial132.stderr:2026-02-26T03:44:48.398+0000 7fb9eb2f6240 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-26T03:44:48.458 INFO:tasks.workunit.client.0.trial132.stderr:+ LOCKS='[{"id":"rbdrw","locker":"client.4201","address":"10.20.193.132:0/2411911472"}]' 2026-02-26T03:44:48.459 INFO:tasks.workunit.client.0.trial132.stderr:+ sleep 1 2026-02-26T03:44:49.459 INFO:tasks.workunit.client.0.trial132.stderr:+ '[' '[{"id":"rbdrw","locker":"client.4201","address":"10.20.193.132:0/2411911472"}]' == '[]' ']' 2026-02-26T03:44:49.460 INFO:tasks.workunit.client.0.trial132.stderr:++ rbd lock list rbdrw-image 2026-02-26T03:44:49.460 INFO:tasks.workunit.client.0.trial132.stderr:++ tail -1 2026-02-26T03:44:49.460 INFO:tasks.workunit.client.0.trial132.stderr:++ awk '{print $NF;}' 2026-02-26T03:44:49.487 INFO:tasks.workunit.client.0.trial132.stderr:2026-02-26T03:44:49.489+0000 7f38c1017240 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-26T03:44:49.487 INFO:tasks.workunit.client.0.trial132.stderr:2026-02-26T03:44:49.489+0000 7f38c1017240 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-26T03:44:49.491 INFO:tasks.workunit.client.0.trial132.stderr:2026-02-26T03:44:49.493+0000 7f38c1017240 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-26T03:44:49.554 INFO:tasks.workunit.client.0.trial132.stderr:+ clientaddr=10.20.193.132:0/2411911472 2026-02-26T03:44:49.555 INFO:tasks.workunit.client.0.trial132.stderr:++ rbd lock list rbdrw-image 2026-02-26T03:44:49.555 INFO:tasks.workunit.client.0.trial132.stderr:++ tail -1 2026-02-26T03:44:49.555 INFO:tasks.workunit.client.0.trial132.stderr:++ awk '{print $1;}' 2026-02-26T03:44:49.582 INFO:tasks.workunit.client.0.trial132.stderr:2026-02-26T03:44:49.584+0000 7f3317a23240 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-26T03:44:49.582 INFO:tasks.workunit.client.0.trial132.stderr:2026-02-26T03:44:49.584+0000 7f3317a23240 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-26T03:44:49.586 INFO:tasks.workunit.client.0.trial132.stderr:2026-02-26T03:44:49.587+0000 7f3317a23240 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-26T03:44:49.665 INFO:tasks.workunit.client.0.trial132.stderr:+ clientid=client.4201 2026-02-26T03:44:49.666 INFO:tasks.workunit.client.0.trial132.stdout:clientaddr: 10.20.193.132:0/2411911472 2026-02-26T03:44:49.666 INFO:tasks.workunit.client.0.trial132.stdout:clientid: client.4201 2026-02-26T03:44:49.666 INFO:tasks.workunit.client.0.trial132.stderr:+ echo 'clientaddr: 10.20.193.132:0/2411911472' 2026-02-26T03:44:49.667 INFO:tasks.workunit.client.0.trial132.stderr:+ echo 'clientid: client.4201' 2026-02-26T03:44:49.667 INFO:tasks.workunit.client.0.trial132.stderr:+ ceph osd blocklist add 10.20.193.132:0/2411911472 2026-02-26T03:44:49.724 INFO:tasks.workunit.client.0.trial132.stderr:2026-02-26T03:44:49.721+0000 7fa17125a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-26T03:44:49.726 INFO:tasks.workunit.client.0.trial132.stderr:2026-02-26T03:44:49.728+0000 7fa17125a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-26T03:44:51.536 INFO:tasks.workunit.client.0.trial132.stderr:blocklisting 10.20.193.132:0/2411911472 until 2026-02-26T04:44:50.602843+0000 (3600 sec) 2026-02-26T03:44:51.548 INFO:tasks.workunit.client.0.trial132.stderr:+ wait 41991 2026-02-26T03:44:51.548 INFO:tasks.workunit.client.0.trial132.stderr:+ rbdrw_exitcode=108 2026-02-26T03:44:51.548 INFO:tasks.workunit.client.0.trial132.stderr:+ '[' 108 '!=' 108 ']' 2026-02-26T03:44:51.549 INFO:tasks.workunit.client.0.trial132.stdout:rbdrw stopped with ESHUTDOWN 2026-02-26T03:44:51.549 INFO:tasks.workunit.client.0.trial132.stderr:+ echo 'rbdrw stopped with ESHUTDOWN' 2026-02-26T03:44:51.549 INFO:tasks.workunit.client.0.trial132.stderr:+ set -e 2026-02-26T03:44:51.549 INFO:tasks.workunit.client.0.trial132.stderr:+ ceph osd blocklist rm 10.20.193.132:0/2411911472 2026-02-26T03:44:51.601 INFO:tasks.workunit.client.0.trial132.stderr:2026-02-26T03:44:51.600+0000 7f2b8ae5a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-26T03:44:51.604 INFO:tasks.workunit.client.0.trial132.stderr:2026-02-26T03:44:51.606+0000 7f2b8ae5a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-26T03:44:52.607 INFO:tasks.workunit.client.0.trial132.stderr:10.20.193.132:0/2411911472 isn't blocklisted 2026-02-26T03:44:52.615 INFO:tasks.workunit.client.0.trial132.stderr:+ rbd lock remove rbdrw-image rbdrw client.4201 2026-02-26T03:44:52.642 INFO:tasks.workunit.client.0.trial132.stderr:2026-02-26T03:44:52.643+0000 7fdadfef1240 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-26T03:44:52.642 INFO:tasks.workunit.client.0.trial132.stderr:2026-02-26T03:44:52.643+0000 7fdadfef1240 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-26T03:44:52.645 INFO:tasks.workunit.client.0.trial132.stderr:2026-02-26T03:44:52.647+0000 7fdadfef1240 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-26T03:44:53.587 INFO:tasks.workunit.client.0.trial132.stderr:+ sleep 30 2026-02-26T03:45:23.588 INFO:tasks.workunit.client.0.trial132.stderr:+ rbd rm rbdrw-image 2026-02-26T03:45:23.615 INFO:tasks.workunit.client.0.trial132.stderr:2026-02-26T03:45:23.616+0000 7f09ae8e0240 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-26T03:45:23.615 INFO:tasks.workunit.client.0.trial132.stderr:2026-02-26T03:45:23.616+0000 7f09ae8e0240 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-26T03:45:23.618 INFO:tasks.workunit.client.0.trial132.stderr:2026-02-26T03:45:23.619+0000 7f09ae8e0240 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-26T03:45:23.945 INFO:tasks.workunit.client.0.trial132.stderr: Removing image: 33% complete... Removing image: 66% complete... Removing image: 100% complete...done. 2026-02-26T03:45:23.949 INFO:tasks.workunit.client.0.trial132.stdout:OK 2026-02-26T03:45:23.950 INFO:tasks.workunit.client.0.trial132.stderr:+ echo OK 2026-02-26T03:45:23.951 INFO:teuthology.orchestra.run:Running command with timeout 3600 2026-02-26T03:45:23.951 DEBUG:teuthology.orchestra.run.trial132:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2026-02-26T03:45:24.015 INFO:tasks.workunit:Stopping ['rbd/test_lock_fence.sh'] on client.0... 2026-02-26T03:45:24.016 DEBUG:teuthology.orchestra.run.trial132:> sudo rm -rf -- /home/ubuntu/cephtest/workunits.list.client.0 /home/ubuntu/cephtest/clone.client.0 2026-02-26T03:45:24.406 DEBUG:teuthology.parallel:result is None 2026-02-26T03:45:24.406 DEBUG:teuthology.orchestra.run.trial132:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0 2026-02-26T03:45:24.430 INFO:tasks.workunit:Deleted dir /home/ubuntu/cephtest/mnt.0/client.0 2026-02-26T03:45:24.431 DEBUG:teuthology.orchestra.run.trial132:> rmdir -- /home/ubuntu/cephtest/mnt.0 2026-02-26T03:45:24.487 INFO:tasks.workunit:Deleted artificial mount point /home/ubuntu/cephtest/mnt.0/client.0 2026-02-26T03:45:24.488 DEBUG:teuthology.run_tasks:Unwinding manager ceph 2026-02-26T03:45:24.501 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2026-02-26T03:45:24.502 DEBUG:teuthology.orchestra.run.trial132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-26T03:45:24.605 INFO:teuthology.orchestra.run.trial132.stderr:2026-02-26T03:45:24.604+0000 7ff09545a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-26T03:45:24.607 INFO:teuthology.orchestra.run.trial132.stderr:2026-02-26T03:45:24.609+0000 7ff09545a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-26T03:45:24.716 INFO:teuthology.orchestra.run.trial132.stdout: 2026-02-26T03:45:24.716 INFO:teuthology.orchestra.run.trial132.stderr:dumped all 2026-02-26T03:45:24.725 INFO:teuthology.orchestra.run.trial132.stdout:{"pg_ready":true,"pg_map":{"version":50,"stamp":"2026-02-26T03:45:24.476381+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":594499,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":494,"num_read_kb":1109,"num_write":343,"num_write_kb":2139,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":320,"ondisk_log_size":320,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":3,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":156499968,"kb_used":3324,"kb_used_data":3324,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":156496644,"statfs":{"total":160255967232,"available":160252563456,"internally_reserved":0,"allocated":3403776,"data_stored":3403776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.000831"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":27,"reported_epoch":21,"state":"active+clean","last_fresh":"2026-02-26T03:44:53.553427+0000","last_change":"2026-02-26T03:44:21.244227+0000","last_active":"2026-02-26T03:44:53.553427+0000","last_peered":"2026-02-26T03:44:53.553427+0000","last_clean":"2026-02-26T03:44:53.553427+0000","last_became_active":"2026-02-26T03:44:19.263201+0000","last_became_peered":"2026-02-26T03:44:19.263201+0000","last_unstale":"2026-02-26T03:44:53.553427+0000","last_undegraded":"2026-02-26T03:44:53.553427+0000","last_fullsized":"2026-02-26T03:44:53.553427+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-26T03:44:18.219221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-26T03:44:18.219221+0000","last_clean_scrub_stamp":"2026-02-26T03:44:18.219221+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":27,"reported_epoch":21,"state":"active+clean","last_fresh":"2026-02-26T03:44:53.554539+0000","last_change":"2026-02-26T03:44:21.247517+0000","last_active":"2026-02-26T03:44:53.554539+0000","last_peered":"2026-02-26T03:44:53.554539+0000","last_clean":"2026-02-26T03:44:53.554539+0000","last_became_active":"2026-02-26T03:44:19.261313+0000","last_became_peered":"2026-02-26T03:44:19.261313+0000","last_unstale":"2026-02-26T03:44:53.554539+0000","last_undegraded":"2026-02-26T03:44:53.554539+0000","last_fullsized":"2026-02-26T03:44:53.554539+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-26T03:44:18.219221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-26T03:44:18.219221+0000","last_clean_scrub_stamp":"2026-02-26T03:44:18.219221+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"17'1","reported_seq":40,"reported_epoch":21,"state":"active+clean","last_fresh":"2026-02-26T03:44:53.554503+0000","last_change":"2026-02-26T03:44:21.246959+0000","last_active":"2026-02-26T03:44:53.554503+0000","last_peered":"2026-02-26T03:44:53.554503+0000","last_clean":"2026-02-26T03:44:53.554503+0000","last_became_active":"2026-02-26T03:44:19.268173+0000","last_became_peered":"2026-02-26T03:44:19.268173+0000","last_unstale":"2026-02-26T03:44:53.554503+0000","last_undegraded":"2026-02-26T03:44:53.554503+0000","last_fullsized":"2026-02-26T03:44:53.554503+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-26T03:44:18.219221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-26T03:44:18.219221+0000","last_clean_scrub_stamp":"2026-02-26T03:44:18.219221+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":20,"num_read_kb":6,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"21'7","reported_seq":93,"reported_epoch":21,"state":"active+clean","last_fresh":"2026-02-26T03:44:53.569010+0000","last_change":"2026-02-26T03:44:21.247143+0000","last_active":"2026-02-26T03:44:53.569010+0000","last_peered":"2026-02-26T03:44:53.569010+0000","last_clean":"2026-02-26T03:44:53.569010+0000","last_became_active":"2026-02-26T03:44:19.271270+0000","last_became_peered":"2026-02-26T03:44:19.271270+0000","last_unstale":"2026-02-26T03:44:53.569010+0000","last_undegraded":"2026-02-26T03:44:53.569010+0000","last_fullsized":"2026-02-26T03:44:53.569010+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-26T03:44:18.219221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-26T03:44:18.219221+0000","last_clean_scrub_stamp":"2026-02-26T03:44:18.219221+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":163,"num_read_kb":28,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"17'2","reported_seq":38,"reported_epoch":21,"state":"active+clean","last_fresh":"2026-02-26T03:44:53.554023+0000","last_change":"2026-02-26T03:44:21.245047+0000","last_active":"2026-02-26T03:44:53.554023+0000","last_peered":"2026-02-26T03:44:53.554023+0000","last_clean":"2026-02-26T03:44:53.554023+0000","last_became_active":"2026-02-26T03:44:19.276841+0000","last_became_peered":"2026-02-26T03:44:19.276841+0000","last_unstale":"2026-02-26T03:44:53.554023+0000","last_undegraded":"2026-02-26T03:44:53.554023+0000","last_fullsized":"2026-02-26T03:44:53.554023+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-26T03:44:18.219221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-26T03:44:18.219221+0000","last_clean_scrub_stamp":"2026-02-26T03:44:18.219221+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":27,"reported_epoch":21,"state":"active+clean","last_fresh":"2026-02-26T03:44:53.553308+0000","last_change":"2026-02-26T03:44:21.243284+0000","last_active":"2026-02-26T03:44:53.553308+0000","last_peered":"2026-02-26T03:44:53.553308+0000","last_clean":"2026-02-26T03:44:53.553308+0000","last_became_active":"2026-02-26T03:44:19.283254+0000","last_became_peered":"2026-02-26T03:44:19.283254+0000","last_unstale":"2026-02-26T03:44:53.553308+0000","last_undegraded":"2026-02-26T03:44:53.553308+0000","last_fullsized":"2026-02-26T03:44:53.553308+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-26T03:44:18.219221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-26T03:44:18.219221+0000","last_clean_scrub_stamp":"2026-02-26T03:44:18.219221+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":27,"reported_epoch":21,"state":"active+clean","last_fresh":"2026-02-26T03:44:53.552483+0000","last_change":"2026-02-26T03:44:21.240663+0000","last_active":"2026-02-26T03:44:53.552483+0000","last_peered":"2026-02-26T03:44:53.552483+0000","last_clean":"2026-02-26T03:44:53.552483+0000","last_became_active":"2026-02-26T03:44:19.283733+0000","last_became_peered":"2026-02-26T03:44:19.283733+0000","last_unstale":"2026-02-26T03:44:53.552483+0000","last_undegraded":"2026-02-26T03:44:53.552483+0000","last_fullsized":"2026-02-26T03:44:53.552483+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-26T03:44:18.219221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-26T03:44:18.219221+0000","last_clean_scrub_stamp":"2026-02-26T03:44:18.219221+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"17'258","reported_seq":542,"reported_epoch":21,"state":"active+clean","last_fresh":"2026-02-26T03:44:53.554705+0000","last_change":"2026-02-26T03:44:21.246671+0000","last_active":"2026-02-26T03:44:53.554705+0000","last_peered":"2026-02-26T03:44:53.554705+0000","last_clean":"2026-02-26T03:44:53.554705+0000","last_became_active":"2026-02-26T03:44:19.259527+0000","last_became_peered":"2026-02-26T03:44:19.259527+0000","last_unstale":"2026-02-26T03:44:53.554705+0000","last_undegraded":"2026-02-26T03:44:53.554705+0000","last_fullsized":"2026-02-26T03:44:53.554705+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-26T03:44:18.219221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-26T03:44:18.219221+0000","last_clean_scrub_stamp":"2026-02-26T03:44:18.219221+0000","objects_scrubbed":0,"log_size":258,"log_dups_size":0,"ondisk_log_size":258,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4096,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":257,"num_read_kb":1028,"num_write":257,"num_write_kb":1028,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"12'52","reported_seq":102,"reported_epoch":21,"state":"active+clean","last_fresh":"2026-02-26T03:44:53.553040+0000","last_change":"2026-02-26T03:44:16.250074+0000","last_active":"2026-02-26T03:44:53.553040+0000","last_peered":"2026-02-26T03:44:53.553040+0000","last_clean":"2026-02-26T03:44:53.553040+0000","last_became_active":"2026-02-26T03:44:16.240654+0000","last_became_peered":"2026-02-26T03:44:16.240654+0000","last_unstale":"2026-02-26T03:44:53.553040+0000","last_undegraded":"2026-02-26T03:44:53.553040+0000","last_fullsized":"2026-02-26T03:44:53.553040+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-26T03:44:15.206229+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-26T03:44:15.206229+0000","last_clean_scrub_stamp":"2026-02-26T03:44:15.206229+0000","objects_scrubbed":0,"log_size":52,"log_dups_size":0,"ondisk_log_size":52,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":53,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":4131,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":441,"num_read_kb":1063,"num_write":264,"num_write_kb":1033,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":160255967232,"available":160252563456,"internally_reserved":0,"allocated":3403776,"data_stored":3403776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":268,"ondisk_log_size":268,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":53,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":53418655744,"available":53417119744,"internally_reserved":0,"allocated":1536000,"data_stored":1536000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":52,"ondisk_log_size":52,"up":2,"acting":2,"num_store_stats":1}],"osd_stats":[{"osd":2,"up_from":10,"seq":42949673029,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52166656,"kb_used":496,"kb_used_data":496,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":52166160,"statfs":{"total":53418655744,"available":53418147840,"internally_reserved":0,"allocated":507904,"data_stored":507904,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":10,"seq":42949673029,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52166656,"kb_used":1328,"kb_used_data":1328,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":52165328,"statfs":{"total":53418655744,"available":53417295872,"internally_reserved":0,"allocated":1359872,"data_stored":1359872,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":9,"seq":38654705733,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52166656,"kb_used":1500,"kb_used_data":1500,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":52165156,"statfs":{"total":53418655744,"available":53417119744,"internally_reserved":0,"allocated":1536000,"data_stored":1536000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":1,"total":53418655744,"available":53417119744,"internally_reserved":0,"allocated":1536000,"data_stored":1536000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":53418655744,"available":53417295872,"internally_reserved":0,"allocated":1359872,"data_stored":1359872,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":53418655744,"available":53417119744,"internally_reserved":0,"allocated":1536000,"data_stored":1536000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":53418655744,"available":53418147840,"internally_reserved":0,"allocated":507904,"data_stored":507904,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-26T03:45:24.726 DEBUG:teuthology.orchestra.run.trial132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-26T03:45:24.803 INFO:teuthology.orchestra.run.trial132.stderr:2026-02-26T03:45:24.802+0000 7f1abe1e4640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-26T03:45:24.805 INFO:teuthology.orchestra.run.trial132.stderr:2026-02-26T03:45:24.807+0000 7f1abe1e4640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-26T03:45:24.913 INFO:teuthology.orchestra.run.trial132.stdout: 2026-02-26T03:45:24.914 INFO:teuthology.orchestra.run.trial132.stderr:dumped all 2026-02-26T03:45:24.922 INFO:teuthology.orchestra.run.trial132.stdout:{"pg_ready":true,"pg_map":{"version":50,"stamp":"2026-02-26T03:45:24.476381+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":594499,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":494,"num_read_kb":1109,"num_write":343,"num_write_kb":2139,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":320,"ondisk_log_size":320,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":3,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":156499968,"kb_used":3324,"kb_used_data":3324,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":156496644,"statfs":{"total":160255967232,"available":160252563456,"internally_reserved":0,"allocated":3403776,"data_stored":3403776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.000831"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":27,"reported_epoch":21,"state":"active+clean","last_fresh":"2026-02-26T03:44:53.553427+0000","last_change":"2026-02-26T03:44:21.244227+0000","last_active":"2026-02-26T03:44:53.553427+0000","last_peered":"2026-02-26T03:44:53.553427+0000","last_clean":"2026-02-26T03:44:53.553427+0000","last_became_active":"2026-02-26T03:44:19.263201+0000","last_became_peered":"2026-02-26T03:44:19.263201+0000","last_unstale":"2026-02-26T03:44:53.553427+0000","last_undegraded":"2026-02-26T03:44:53.553427+0000","last_fullsized":"2026-02-26T03:44:53.553427+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-26T03:44:18.219221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-26T03:44:18.219221+0000","last_clean_scrub_stamp":"2026-02-26T03:44:18.219221+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":27,"reported_epoch":21,"state":"active+clean","last_fresh":"2026-02-26T03:44:53.554539+0000","last_change":"2026-02-26T03:44:21.247517+0000","last_active":"2026-02-26T03:44:53.554539+0000","last_peered":"2026-02-26T03:44:53.554539+0000","last_clean":"2026-02-26T03:44:53.554539+0000","last_became_active":"2026-02-26T03:44:19.261313+0000","last_became_peered":"2026-02-26T03:44:19.261313+0000","last_unstale":"2026-02-26T03:44:53.554539+0000","last_undegraded":"2026-02-26T03:44:53.554539+0000","last_fullsized":"2026-02-26T03:44:53.554539+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-26T03:44:18.219221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-26T03:44:18.219221+0000","last_clean_scrub_stamp":"2026-02-26T03:44:18.219221+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"17'1","reported_seq":40,"reported_epoch":21,"state":"active+clean","last_fresh":"2026-02-26T03:44:53.554503+0000","last_change":"2026-02-26T03:44:21.246959+0000","last_active":"2026-02-26T03:44:53.554503+0000","last_peered":"2026-02-26T03:44:53.554503+0000","last_clean":"2026-02-26T03:44:53.554503+0000","last_became_active":"2026-02-26T03:44:19.268173+0000","last_became_peered":"2026-02-26T03:44:19.268173+0000","last_unstale":"2026-02-26T03:44:53.554503+0000","last_undegraded":"2026-02-26T03:44:53.554503+0000","last_fullsized":"2026-02-26T03:44:53.554503+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-26T03:44:18.219221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-26T03:44:18.219221+0000","last_clean_scrub_stamp":"2026-02-26T03:44:18.219221+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":20,"num_read_kb":6,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"21'7","reported_seq":93,"reported_epoch":21,"state":"active+clean","last_fresh":"2026-02-26T03:44:53.569010+0000","last_change":"2026-02-26T03:44:21.247143+0000","last_active":"2026-02-26T03:44:53.569010+0000","last_peered":"2026-02-26T03:44:53.569010+0000","last_clean":"2026-02-26T03:44:53.569010+0000","last_became_active":"2026-02-26T03:44:19.271270+0000","last_became_peered":"2026-02-26T03:44:19.271270+0000","last_unstale":"2026-02-26T03:44:53.569010+0000","last_undegraded":"2026-02-26T03:44:53.569010+0000","last_fullsized":"2026-02-26T03:44:53.569010+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-26T03:44:18.219221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-26T03:44:18.219221+0000","last_clean_scrub_stamp":"2026-02-26T03:44:18.219221+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":163,"num_read_kb":28,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"17'2","reported_seq":38,"reported_epoch":21,"state":"active+clean","last_fresh":"2026-02-26T03:44:53.554023+0000","last_change":"2026-02-26T03:44:21.245047+0000","last_active":"2026-02-26T03:44:53.554023+0000","last_peered":"2026-02-26T03:44:53.554023+0000","last_clean":"2026-02-26T03:44:53.554023+0000","last_became_active":"2026-02-26T03:44:19.276841+0000","last_became_peered":"2026-02-26T03:44:19.276841+0000","last_unstale":"2026-02-26T03:44:53.554023+0000","last_undegraded":"2026-02-26T03:44:53.554023+0000","last_fullsized":"2026-02-26T03:44:53.554023+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-26T03:44:18.219221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-26T03:44:18.219221+0000","last_clean_scrub_stamp":"2026-02-26T03:44:18.219221+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":27,"reported_epoch":21,"state":"active+clean","last_fresh":"2026-02-26T03:44:53.553308+0000","last_change":"2026-02-26T03:44:21.243284+0000","last_active":"2026-02-26T03:44:53.553308+0000","last_peered":"2026-02-26T03:44:53.553308+0000","last_clean":"2026-02-26T03:44:53.553308+0000","last_became_active":"2026-02-26T03:44:19.283254+0000","last_became_peered":"2026-02-26T03:44:19.283254+0000","last_unstale":"2026-02-26T03:44:53.553308+0000","last_undegraded":"2026-02-26T03:44:53.553308+0000","last_fullsized":"2026-02-26T03:44:53.553308+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-26T03:44:18.219221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-26T03:44:18.219221+0000","last_clean_scrub_stamp":"2026-02-26T03:44:18.219221+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":27,"reported_epoch":21,"state":"active+clean","last_fresh":"2026-02-26T03:44:53.552483+0000","last_change":"2026-02-26T03:44:21.240663+0000","last_active":"2026-02-26T03:44:53.552483+0000","last_peered":"2026-02-26T03:44:53.552483+0000","last_clean":"2026-02-26T03:44:53.552483+0000","last_became_active":"2026-02-26T03:44:19.283733+0000","last_became_peered":"2026-02-26T03:44:19.283733+0000","last_unstale":"2026-02-26T03:44:53.552483+0000","last_undegraded":"2026-02-26T03:44:53.552483+0000","last_fullsized":"2026-02-26T03:44:53.552483+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-26T03:44:18.219221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-26T03:44:18.219221+0000","last_clean_scrub_stamp":"2026-02-26T03:44:18.219221+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"17'258","reported_seq":542,"reported_epoch":21,"state":"active+clean","last_fresh":"2026-02-26T03:44:53.554705+0000","last_change":"2026-02-26T03:44:21.246671+0000","last_active":"2026-02-26T03:44:53.554705+0000","last_peered":"2026-02-26T03:44:53.554705+0000","last_clean":"2026-02-26T03:44:53.554705+0000","last_became_active":"2026-02-26T03:44:19.259527+0000","last_became_peered":"2026-02-26T03:44:19.259527+0000","last_unstale":"2026-02-26T03:44:53.554705+0000","last_undegraded":"2026-02-26T03:44:53.554705+0000","last_fullsized":"2026-02-26T03:44:53.554705+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-26T03:44:18.219221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-26T03:44:18.219221+0000","last_clean_scrub_stamp":"2026-02-26T03:44:18.219221+0000","objects_scrubbed":0,"log_size":258,"log_dups_size":0,"ondisk_log_size":258,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4096,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":257,"num_read_kb":1028,"num_write":257,"num_write_kb":1028,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"12'52","reported_seq":102,"reported_epoch":21,"state":"active+clean","last_fresh":"2026-02-26T03:44:53.553040+0000","last_change":"2026-02-26T03:44:16.250074+0000","last_active":"2026-02-26T03:44:53.553040+0000","last_peered":"2026-02-26T03:44:53.553040+0000","last_clean":"2026-02-26T03:44:53.553040+0000","last_became_active":"2026-02-26T03:44:16.240654+0000","last_became_peered":"2026-02-26T03:44:16.240654+0000","last_unstale":"2026-02-26T03:44:53.553040+0000","last_undegraded":"2026-02-26T03:44:53.553040+0000","last_fullsized":"2026-02-26T03:44:53.553040+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-26T03:44:15.206229+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-26T03:44:15.206229+0000","last_clean_scrub_stamp":"2026-02-26T03:44:15.206229+0000","objects_scrubbed":0,"log_size":52,"log_dups_size":0,"ondisk_log_size":52,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":53,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":4131,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":441,"num_read_kb":1063,"num_write":264,"num_write_kb":1033,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":160255967232,"available":160252563456,"internally_reserved":0,"allocated":3403776,"data_stored":3403776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":268,"ondisk_log_size":268,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":53,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":53418655744,"available":53417119744,"internally_reserved":0,"allocated":1536000,"data_stored":1536000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":52,"ondisk_log_size":52,"up":2,"acting":2,"num_store_stats":1}],"osd_stats":[{"osd":2,"up_from":10,"seq":42949673029,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52166656,"kb_used":496,"kb_used_data":496,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":52166160,"statfs":{"total":53418655744,"available":53418147840,"internally_reserved":0,"allocated":507904,"data_stored":507904,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":10,"seq":42949673029,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52166656,"kb_used":1328,"kb_used_data":1328,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":52165328,"statfs":{"total":53418655744,"available":53417295872,"internally_reserved":0,"allocated":1359872,"data_stored":1359872,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":9,"seq":38654705733,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52166656,"kb_used":1500,"kb_used_data":1500,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":52165156,"statfs":{"total":53418655744,"available":53417119744,"internally_reserved":0,"allocated":1536000,"data_stored":1536000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":1,"total":53418655744,"available":53417119744,"internally_reserved":0,"allocated":1536000,"data_stored":1536000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":53418655744,"available":53417295872,"internally_reserved":0,"allocated":1359872,"data_stored":1359872,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":53418655744,"available":53417119744,"internally_reserved":0,"allocated":1536000,"data_stored":1536000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":53418655744,"available":53418147840,"internally_reserved":0,"allocated":507904,"data_stored":507904,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-26T03:45:24.923 INFO:tasks.ceph.ceph_manager.ceph:clean! 2026-02-26T03:45:24.923 DEBUG:teuthology.orchestra.run.trial132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-26T03:45:24.999 INFO:teuthology.orchestra.run.trial132.stderr:2026-02-26T03:45:24.999+0000 7fadfd90b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-26T03:45:25.001 INFO:teuthology.orchestra.run.trial132.stderr:2026-02-26T03:45:25.003+0000 7fadfd90b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-26T03:45:25.113 INFO:teuthology.orchestra.run.trial132.stdout: 2026-02-26T03:45:25.114 INFO:teuthology.orchestra.run.trial132.stderr:dumped all 2026-02-26T03:45:25.123 INFO:teuthology.orchestra.run.trial132.stdout:{"pg_ready":true,"pg_map":{"version":50,"stamp":"2026-02-26T03:45:24.476381+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":594499,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":494,"num_read_kb":1109,"num_write":343,"num_write_kb":2139,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":320,"ondisk_log_size":320,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":3,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":156499968,"kb_used":3324,"kb_used_data":3324,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":156496644,"statfs":{"total":160255967232,"available":160252563456,"internally_reserved":0,"allocated":3403776,"data_stored":3403776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.000831"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":27,"reported_epoch":21,"state":"active+clean","last_fresh":"2026-02-26T03:44:53.553427+0000","last_change":"2026-02-26T03:44:21.244227+0000","last_active":"2026-02-26T03:44:53.553427+0000","last_peered":"2026-02-26T03:44:53.553427+0000","last_clean":"2026-02-26T03:44:53.553427+0000","last_became_active":"2026-02-26T03:44:19.263201+0000","last_became_peered":"2026-02-26T03:44:19.263201+0000","last_unstale":"2026-02-26T03:44:53.553427+0000","last_undegraded":"2026-02-26T03:44:53.553427+0000","last_fullsized":"2026-02-26T03:44:53.553427+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-26T03:44:18.219221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-26T03:44:18.219221+0000","last_clean_scrub_stamp":"2026-02-26T03:44:18.219221+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":27,"reported_epoch":21,"state":"active+clean","last_fresh":"2026-02-26T03:44:53.554539+0000","last_change":"2026-02-26T03:44:21.247517+0000","last_active":"2026-02-26T03:44:53.554539+0000","last_peered":"2026-02-26T03:44:53.554539+0000","last_clean":"2026-02-26T03:44:53.554539+0000","last_became_active":"2026-02-26T03:44:19.261313+0000","last_became_peered":"2026-02-26T03:44:19.261313+0000","last_unstale":"2026-02-26T03:44:53.554539+0000","last_undegraded":"2026-02-26T03:44:53.554539+0000","last_fullsized":"2026-02-26T03:44:53.554539+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-26T03:44:18.219221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-26T03:44:18.219221+0000","last_clean_scrub_stamp":"2026-02-26T03:44:18.219221+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"17'1","reported_seq":40,"reported_epoch":21,"state":"active+clean","last_fresh":"2026-02-26T03:44:53.554503+0000","last_change":"2026-02-26T03:44:21.246959+0000","last_active":"2026-02-26T03:44:53.554503+0000","last_peered":"2026-02-26T03:44:53.554503+0000","last_clean":"2026-02-26T03:44:53.554503+0000","last_became_active":"2026-02-26T03:44:19.268173+0000","last_became_peered":"2026-02-26T03:44:19.268173+0000","last_unstale":"2026-02-26T03:44:53.554503+0000","last_undegraded":"2026-02-26T03:44:53.554503+0000","last_fullsized":"2026-02-26T03:44:53.554503+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-26T03:44:18.219221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-26T03:44:18.219221+0000","last_clean_scrub_stamp":"2026-02-26T03:44:18.219221+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":20,"num_read_kb":6,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"21'7","reported_seq":93,"reported_epoch":21,"state":"active+clean","last_fresh":"2026-02-26T03:44:53.569010+0000","last_change":"2026-02-26T03:44:21.247143+0000","last_active":"2026-02-26T03:44:53.569010+0000","last_peered":"2026-02-26T03:44:53.569010+0000","last_clean":"2026-02-26T03:44:53.569010+0000","last_became_active":"2026-02-26T03:44:19.271270+0000","last_became_peered":"2026-02-26T03:44:19.271270+0000","last_unstale":"2026-02-26T03:44:53.569010+0000","last_undegraded":"2026-02-26T03:44:53.569010+0000","last_fullsized":"2026-02-26T03:44:53.569010+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-26T03:44:18.219221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-26T03:44:18.219221+0000","last_clean_scrub_stamp":"2026-02-26T03:44:18.219221+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":163,"num_read_kb":28,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"17'2","reported_seq":38,"reported_epoch":21,"state":"active+clean","last_fresh":"2026-02-26T03:44:53.554023+0000","last_change":"2026-02-26T03:44:21.245047+0000","last_active":"2026-02-26T03:44:53.554023+0000","last_peered":"2026-02-26T03:44:53.554023+0000","last_clean":"2026-02-26T03:44:53.554023+0000","last_became_active":"2026-02-26T03:44:19.276841+0000","last_became_peered":"2026-02-26T03:44:19.276841+0000","last_unstale":"2026-02-26T03:44:53.554023+0000","last_undegraded":"2026-02-26T03:44:53.554023+0000","last_fullsized":"2026-02-26T03:44:53.554023+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-26T03:44:18.219221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-26T03:44:18.219221+0000","last_clean_scrub_stamp":"2026-02-26T03:44:18.219221+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":27,"reported_epoch":21,"state":"active+clean","last_fresh":"2026-02-26T03:44:53.553308+0000","last_change":"2026-02-26T03:44:21.243284+0000","last_active":"2026-02-26T03:44:53.553308+0000","last_peered":"2026-02-26T03:44:53.553308+0000","last_clean":"2026-02-26T03:44:53.553308+0000","last_became_active":"2026-02-26T03:44:19.283254+0000","last_became_peered":"2026-02-26T03:44:19.283254+0000","last_unstale":"2026-02-26T03:44:53.553308+0000","last_undegraded":"2026-02-26T03:44:53.553308+0000","last_fullsized":"2026-02-26T03:44:53.553308+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-26T03:44:18.219221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-26T03:44:18.219221+0000","last_clean_scrub_stamp":"2026-02-26T03:44:18.219221+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":27,"reported_epoch":21,"state":"active+clean","last_fresh":"2026-02-26T03:44:53.552483+0000","last_change":"2026-02-26T03:44:21.240663+0000","last_active":"2026-02-26T03:44:53.552483+0000","last_peered":"2026-02-26T03:44:53.552483+0000","last_clean":"2026-02-26T03:44:53.552483+0000","last_became_active":"2026-02-26T03:44:19.283733+0000","last_became_peered":"2026-02-26T03:44:19.283733+0000","last_unstale":"2026-02-26T03:44:53.552483+0000","last_undegraded":"2026-02-26T03:44:53.552483+0000","last_fullsized":"2026-02-26T03:44:53.552483+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-26T03:44:18.219221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-26T03:44:18.219221+0000","last_clean_scrub_stamp":"2026-02-26T03:44:18.219221+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"17'258","reported_seq":542,"reported_epoch":21,"state":"active+clean","last_fresh":"2026-02-26T03:44:53.554705+0000","last_change":"2026-02-26T03:44:21.246671+0000","last_active":"2026-02-26T03:44:53.554705+0000","last_peered":"2026-02-26T03:44:53.554705+0000","last_clean":"2026-02-26T03:44:53.554705+0000","last_became_active":"2026-02-26T03:44:19.259527+0000","last_became_peered":"2026-02-26T03:44:19.259527+0000","last_unstale":"2026-02-26T03:44:53.554705+0000","last_undegraded":"2026-02-26T03:44:53.554705+0000","last_fullsized":"2026-02-26T03:44:53.554705+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-26T03:44:18.219221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-26T03:44:18.219221+0000","last_clean_scrub_stamp":"2026-02-26T03:44:18.219221+0000","objects_scrubbed":0,"log_size":258,"log_dups_size":0,"ondisk_log_size":258,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4096,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":257,"num_read_kb":1028,"num_write":257,"num_write_kb":1028,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"12'52","reported_seq":102,"reported_epoch":21,"state":"active+clean","last_fresh":"2026-02-26T03:44:53.553040+0000","last_change":"2026-02-26T03:44:16.250074+0000","last_active":"2026-02-26T03:44:53.553040+0000","last_peered":"2026-02-26T03:44:53.553040+0000","last_clean":"2026-02-26T03:44:53.553040+0000","last_became_active":"2026-02-26T03:44:16.240654+0000","last_became_peered":"2026-02-26T03:44:16.240654+0000","last_unstale":"2026-02-26T03:44:53.553040+0000","last_undegraded":"2026-02-26T03:44:53.553040+0000","last_fullsized":"2026-02-26T03:44:53.553040+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-26T03:44:15.206229+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-26T03:44:15.206229+0000","last_clean_scrub_stamp":"2026-02-26T03:44:15.206229+0000","objects_scrubbed":0,"log_size":52,"log_dups_size":0,"ondisk_log_size":52,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":53,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":4131,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":441,"num_read_kb":1063,"num_write":264,"num_write_kb":1033,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":160255967232,"available":160252563456,"internally_reserved":0,"allocated":3403776,"data_stored":3403776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":268,"ondisk_log_size":268,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":53,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":53418655744,"available":53417119744,"internally_reserved":0,"allocated":1536000,"data_stored":1536000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":52,"ondisk_log_size":52,"up":2,"acting":2,"num_store_stats":1}],"osd_stats":[{"osd":2,"up_from":10,"seq":42949673029,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52166656,"kb_used":496,"kb_used_data":496,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":52166160,"statfs":{"total":53418655744,"available":53418147840,"internally_reserved":0,"allocated":507904,"data_stored":507904,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":10,"seq":42949673029,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52166656,"kb_used":1328,"kb_used_data":1328,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":52165328,"statfs":{"total":53418655744,"available":53417295872,"internally_reserved":0,"allocated":1359872,"data_stored":1359872,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":9,"seq":38654705733,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52166656,"kb_used":1500,"kb_used_data":1500,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":52165156,"statfs":{"total":53418655744,"available":53417119744,"internally_reserved":0,"allocated":1536000,"data_stored":1536000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":1,"total":53418655744,"available":53417119744,"internally_reserved":0,"allocated":1536000,"data_stored":1536000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":53418655744,"available":53417295872,"internally_reserved":0,"allocated":1359872,"data_stored":1359872,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":53418655744,"available":53417119744,"internally_reserved":0,"allocated":1536000,"data_stored":1536000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":53418655744,"available":53418147840,"internally_reserved":0,"allocated":507904,"data_stored":507904,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-26T03:45:25.124 DEBUG:teuthology.orchestra.run.trial132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2026-02-26T03:45:25.200 INFO:teuthology.orchestra.run.trial132.stderr:2026-02-26T03:45:25.199+0000 7f14b8b0b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-26T03:45:25.202 INFO:teuthology.orchestra.run.trial132.stderr:2026-02-26T03:45:25.204+0000 7f14b8b0b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-26T03:45:25.315 INFO:teuthology.orchestra.run.trial132.stdout: 2026-02-26T03:45:25.315 INFO:teuthology.orchestra.run.trial132.stdout:{"epoch":21,"fsid":"109d3300-c629-4972-9fab-d8a7123a9741","created":"2026-02-26T03:44:02.016900+0000","modified":"2026-02-26T03:44:53.542554+0000","last_up_change":"2026-02-26T03:44:13.195912+0000","last_in_change":"2026-02-26T03:44:03.779206+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-26T03:44:14.486080+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"13","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.9900000095367432,"score_stable":2.9900000095367432,"optimal_score":0.67000001668930054,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2026-02-26T03:44:17.770703+0000","flags":270337,"flags_names":"hashpspool,selfmanaged_snaps,crimson","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":2,"snap_epoch":17,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"de6b87e2-9687-4f88-b0cb-1e4799f76b1c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":10,"up_thru":14,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.132:6805","nonce":2385043468}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.132:6804","nonce":2385043468}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.132:6809","nonce":2385043468}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.132:6808","nonce":2385043468}]},"public_addr":"10.20.193.132:6805/2385043468","cluster_addr":"10.20.193.132:6804/2385043468","heartbeat_back_addr":"10.20.193.132:6809/2385043468","heartbeat_front_addr":"10.20.193.132:6808/2385043468","state":["exists","up"]},{"osd":1,"uuid":"7d7e4e45-9f58-40d0-b37a-0973e1bc7504","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":14,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.132:6803","nonce":546026106}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.132:6802","nonce":546026106}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.132:6815","nonce":546026106}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.132:6816","nonce":546026106}]},"public_addr":"10.20.193.132:6803/546026106","cluster_addr":"10.20.193.132:6802/546026106","heartbeat_back_addr":"10.20.193.132:6815/546026106","heartbeat_front_addr":"10.20.193.132:6816/546026106","state":["exists","up"]},{"osd":2,"uuid":"f83e5653-ffd0-41dd-b067-fa02dc7934e4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":10,"up_thru":14,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.132:6810","nonce":798308673}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.132:6811","nonce":798308673}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.132:6807","nonce":798308673}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.132:6806","nonce":798308673}]},"public_addr":"10.20.193.132:6810/798308673","cluster_addr":"10.20.193.132:6811/798308673","heartbeat_back_addr":"10.20.193.132:6807/798308673","heartbeat_front_addr":"10.20.193.132:6806/798308673","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.132:0/2411911472":"2026-02-26T04:44:52.712323+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":2,"snaps":[{"begin":2,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-26T03:45:26.326 INFO:tasks.ceph:Scrubbing osd.0 2026-02-26T03:45:26.326 DEBUG:teuthology.orchestra.run.trial132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 config set osd_debug_deep_scrub_sleep 0 2026-02-26T03:45:26.407 INFO:teuthology.orchestra.run.trial132.stderr:2026-02-26T03:45:26.404+0000 7f3d2da5a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-26T03:45:26.409 INFO:teuthology.orchestra.run.trial132.stderr:2026-02-26T03:45:26.411+0000 7f3d2da5a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-26T03:45:26.420 INFO:teuthology.orchestra.run.trial132.stdout:{ 2026-02-26T03:45:26.420 INFO:teuthology.orchestra.run.trial132.stdout: "success": "" 2026-02-26T03:45:26.421 INFO:teuthology.orchestra.run.trial132.stdout:} 2026-02-26T03:45:26.428 DEBUG:teuthology.orchestra.run.trial132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 0 2026-02-26T03:45:26.504 INFO:teuthology.orchestra.run.trial132.stderr:2026-02-26T03:45:26.503+0000 7fdeafd0b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-26T03:45:26.506 INFO:teuthology.orchestra.run.trial132.stderr:2026-02-26T03:45:26.507+0000 7fdeafd0b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-26T03:45:26.616 INFO:teuthology.orchestra.run.trial132.stderr:instructed osd(s) 0 to deep-scrub 2026-02-26T03:45:26.628 INFO:tasks.ceph:Scrubbing osd.1 2026-02-26T03:45:26.628 DEBUG:teuthology.orchestra.run.trial132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 config set osd_debug_deep_scrub_sleep 0 2026-02-26T03:45:26.705 INFO:teuthology.orchestra.run.trial132.stderr:2026-02-26T03:45:26.703+0000 7fdadd48b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-26T03:45:26.707 INFO:teuthology.orchestra.run.trial132.stderr:2026-02-26T03:45:26.709+0000 7fdadd48b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-26T03:45:26.716 INFO:teuthology.orchestra.run.trial132.stdout:{ 2026-02-26T03:45:26.717 INFO:teuthology.orchestra.run.trial132.stdout: "success": "" 2026-02-26T03:45:26.717 INFO:teuthology.orchestra.run.trial132.stdout:} 2026-02-26T03:45:26.725 DEBUG:teuthology.orchestra.run.trial132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 1 2026-02-26T03:45:26.801 INFO:teuthology.orchestra.run.trial132.stderr:2026-02-26T03:45:26.800+0000 7fc08745a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-26T03:45:26.803 INFO:teuthology.orchestra.run.trial132.stderr:2026-02-26T03:45:26.805+0000 7fc08745a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-26T03:45:26.913 INFO:teuthology.orchestra.run.trial132.stderr:instructed osd(s) 1 to deep-scrub 2026-02-26T03:45:26.926 INFO:tasks.ceph:Scrubbing osd.2 2026-02-26T03:45:26.927 DEBUG:teuthology.orchestra.run.trial132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 config set osd_debug_deep_scrub_sleep 0 2026-02-26T03:45:27.005 INFO:teuthology.orchestra.run.trial132.stderr:2026-02-26T03:45:27.002+0000 7fcbd28cb640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-26T03:45:27.007 INFO:teuthology.orchestra.run.trial132.stderr:2026-02-26T03:45:27.008+0000 7fcbd28cb640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-26T03:45:27.016 INFO:teuthology.orchestra.run.trial132.stdout:{ 2026-02-26T03:45:27.016 INFO:teuthology.orchestra.run.trial132.stdout: "success": "" 2026-02-26T03:45:27.016 INFO:teuthology.orchestra.run.trial132.stdout:} 2026-02-26T03:45:27.023 DEBUG:teuthology.orchestra.run.trial132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 2 2026-02-26T03:45:27.100 INFO:teuthology.orchestra.run.trial132.stderr:2026-02-26T03:45:27.098+0000 7fbb479b8640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-26T03:45:27.102 INFO:teuthology.orchestra.run.trial132.stderr:2026-02-26T03:45:27.103+0000 7fbb479b8640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-26T03:45:27.213 INFO:teuthology.orchestra.run.trial132.stderr:instructed osd(s) 2 to deep-scrub 2026-02-26T03:45:27.224 DEBUG:teuthology.orchestra.run.trial132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-26T03:45:27.300 INFO:teuthology.orchestra.run.trial132.stderr:2026-02-26T03:45:27.299+0000 7fa48ebfd640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-26T03:45:27.302 INFO:teuthology.orchestra.run.trial132.stderr:2026-02-26T03:45:27.303+0000 7fa48ebfd640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-26T03:45:27.412 INFO:teuthology.orchestra.run.trial132.stdout: 2026-02-26T03:45:27.413 INFO:teuthology.orchestra.run.trial132.stderr:dumped all 2026-02-26T03:45:27.421 INFO:teuthology.orchestra.run.trial132.stdout:{"pg_ready":true,"pg_map":{"version":51,"stamp":"2026-02-26T03:45:26.476620+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":594499,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":494,"num_read_kb":1109,"num_write":343,"num_write_kb":2139,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":320,"ondisk_log_size":320,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":3,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":156499968,"kb_used":3324,"kb_used_data":3324,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":156496644,"statfs":{"total":160255967232,"available":160252563456,"internally_reserved":0,"allocated":3403776,"data_stored":3403776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.000927"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":27,"reported_epoch":21,"state":"active+clean","last_fresh":"2026-02-26T03:44:53.553427+0000","last_change":"2026-02-26T03:44:21.244227+0000","last_active":"2026-02-26T03:44:53.553427+0000","last_peered":"2026-02-26T03:44:53.553427+0000","last_clean":"2026-02-26T03:44:53.553427+0000","last_became_active":"2026-02-26T03:44:19.263201+0000","last_became_peered":"2026-02-26T03:44:19.263201+0000","last_unstale":"2026-02-26T03:44:53.553427+0000","last_undegraded":"2026-02-26T03:44:53.553427+0000","last_fullsized":"2026-02-26T03:44:53.553427+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-26T03:44:18.219221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-26T03:44:18.219221+0000","last_clean_scrub_stamp":"2026-02-26T03:44:18.219221+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":27,"reported_epoch":21,"state":"active+clean","last_fresh":"2026-02-26T03:44:53.554539+0000","last_change":"2026-02-26T03:44:21.247517+0000","last_active":"2026-02-26T03:44:53.554539+0000","last_peered":"2026-02-26T03:44:53.554539+0000","last_clean":"2026-02-26T03:44:53.554539+0000","last_became_active":"2026-02-26T03:44:19.261313+0000","last_became_peered":"2026-02-26T03:44:19.261313+0000","last_unstale":"2026-02-26T03:44:53.554539+0000","last_undegraded":"2026-02-26T03:44:53.554539+0000","last_fullsized":"2026-02-26T03:44:53.554539+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-26T03:44:18.219221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-26T03:44:18.219221+0000","last_clean_scrub_stamp":"2026-02-26T03:44:18.219221+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"17'1","reported_seq":40,"reported_epoch":21,"state":"active+clean","last_fresh":"2026-02-26T03:44:53.554503+0000","last_change":"2026-02-26T03:44:21.246959+0000","last_active":"2026-02-26T03:44:53.554503+0000","last_peered":"2026-02-26T03:44:53.554503+0000","last_clean":"2026-02-26T03:44:53.554503+0000","last_became_active":"2026-02-26T03:44:19.268173+0000","last_became_peered":"2026-02-26T03:44:19.268173+0000","last_unstale":"2026-02-26T03:44:53.554503+0000","last_undegraded":"2026-02-26T03:44:53.554503+0000","last_fullsized":"2026-02-26T03:44:53.554503+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-26T03:44:18.219221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-26T03:44:18.219221+0000","last_clean_scrub_stamp":"2026-02-26T03:44:18.219221+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":20,"num_read_kb":6,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"21'7","reported_seq":93,"reported_epoch":21,"state":"active+clean","last_fresh":"2026-02-26T03:44:53.569010+0000","last_change":"2026-02-26T03:44:21.247143+0000","last_active":"2026-02-26T03:44:53.569010+0000","last_peered":"2026-02-26T03:44:53.569010+0000","last_clean":"2026-02-26T03:44:53.569010+0000","last_became_active":"2026-02-26T03:44:19.271270+0000","last_became_peered":"2026-02-26T03:44:19.271270+0000","last_unstale":"2026-02-26T03:44:53.569010+0000","last_undegraded":"2026-02-26T03:44:53.569010+0000","last_fullsized":"2026-02-26T03:44:53.569010+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-26T03:44:18.219221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-26T03:44:18.219221+0000","last_clean_scrub_stamp":"2026-02-26T03:44:18.219221+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":163,"num_read_kb":28,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"17'2","reported_seq":38,"reported_epoch":21,"state":"active+clean","last_fresh":"2026-02-26T03:44:53.554023+0000","last_change":"2026-02-26T03:44:21.245047+0000","last_active":"2026-02-26T03:44:53.554023+0000","last_peered":"2026-02-26T03:44:53.554023+0000","last_clean":"2026-02-26T03:44:53.554023+0000","last_became_active":"2026-02-26T03:44:19.276841+0000","last_became_peered":"2026-02-26T03:44:19.276841+0000","last_unstale":"2026-02-26T03:44:53.554023+0000","last_undegraded":"2026-02-26T03:44:53.554023+0000","last_fullsized":"2026-02-26T03:44:53.554023+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-26T03:44:18.219221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-26T03:44:18.219221+0000","last_clean_scrub_stamp":"2026-02-26T03:44:18.219221+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":27,"reported_epoch":21,"state":"active+clean","last_fresh":"2026-02-26T03:44:53.553308+0000","last_change":"2026-02-26T03:44:21.243284+0000","last_active":"2026-02-26T03:44:53.553308+0000","last_peered":"2026-02-26T03:44:53.553308+0000","last_clean":"2026-02-26T03:44:53.553308+0000","last_became_active":"2026-02-26T03:44:19.283254+0000","last_became_peered":"2026-02-26T03:44:19.283254+0000","last_unstale":"2026-02-26T03:44:53.553308+0000","last_undegraded":"2026-02-26T03:44:53.553308+0000","last_fullsized":"2026-02-26T03:44:53.553308+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-26T03:44:18.219221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-26T03:44:18.219221+0000","last_clean_scrub_stamp":"2026-02-26T03:44:18.219221+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":27,"reported_epoch":21,"state":"active+clean","last_fresh":"2026-02-26T03:44:53.552483+0000","last_change":"2026-02-26T03:44:21.240663+0000","last_active":"2026-02-26T03:44:53.552483+0000","last_peered":"2026-02-26T03:44:53.552483+0000","last_clean":"2026-02-26T03:44:53.552483+0000","last_became_active":"2026-02-26T03:44:19.283733+0000","last_became_peered":"2026-02-26T03:44:19.283733+0000","last_unstale":"2026-02-26T03:44:53.552483+0000","last_undegraded":"2026-02-26T03:44:53.552483+0000","last_fullsized":"2026-02-26T03:44:53.552483+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-26T03:44:18.219221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-26T03:44:18.219221+0000","last_clean_scrub_stamp":"2026-02-26T03:44:18.219221+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"17'258","reported_seq":542,"reported_epoch":21,"state":"active+clean","last_fresh":"2026-02-26T03:44:53.554705+0000","last_change":"2026-02-26T03:44:21.246671+0000","last_active":"2026-02-26T03:44:53.554705+0000","last_peered":"2026-02-26T03:44:53.554705+0000","last_clean":"2026-02-26T03:44:53.554705+0000","last_became_active":"2026-02-26T03:44:19.259527+0000","last_became_peered":"2026-02-26T03:44:19.259527+0000","last_unstale":"2026-02-26T03:44:53.554705+0000","last_undegraded":"2026-02-26T03:44:53.554705+0000","last_fullsized":"2026-02-26T03:44:53.554705+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-26T03:44:18.219221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-26T03:44:18.219221+0000","last_clean_scrub_stamp":"2026-02-26T03:44:18.219221+0000","objects_scrubbed":0,"log_size":258,"log_dups_size":0,"ondisk_log_size":258,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4096,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":257,"num_read_kb":1028,"num_write":257,"num_write_kb":1028,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"12'52","reported_seq":102,"reported_epoch":21,"state":"active+clean","last_fresh":"2026-02-26T03:44:53.553040+0000","last_change":"2026-02-26T03:44:16.250074+0000","last_active":"2026-02-26T03:44:53.553040+0000","last_peered":"2026-02-26T03:44:53.553040+0000","last_clean":"2026-02-26T03:44:53.553040+0000","last_became_active":"2026-02-26T03:44:16.240654+0000","last_became_peered":"2026-02-26T03:44:16.240654+0000","last_unstale":"2026-02-26T03:44:53.553040+0000","last_undegraded":"2026-02-26T03:44:53.553040+0000","last_fullsized":"2026-02-26T03:44:53.553040+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-26T03:44:15.206229+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-26T03:44:15.206229+0000","last_clean_scrub_stamp":"2026-02-26T03:44:15.206229+0000","objects_scrubbed":0,"log_size":52,"log_dups_size":0,"ondisk_log_size":52,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":53,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":4131,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":441,"num_read_kb":1063,"num_write":264,"num_write_kb":1033,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":160255967232,"available":160252563456,"internally_reserved":0,"allocated":3403776,"data_stored":3403776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":268,"ondisk_log_size":268,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":53,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":53418655744,"available":53417119744,"internally_reserved":0,"allocated":1536000,"data_stored":1536000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":52,"ondisk_log_size":52,"up":2,"acting":2,"num_store_stats":1}],"osd_stats":[{"osd":2,"up_from":10,"seq":42949673029,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52166656,"kb_used":496,"kb_used_data":496,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":52166160,"statfs":{"total":53418655744,"available":53418147840,"internally_reserved":0,"allocated":507904,"data_stored":507904,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":10,"seq":42949673029,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52166656,"kb_used":1328,"kb_used_data":1328,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":52165328,"statfs":{"total":53418655744,"available":53417295872,"internally_reserved":0,"allocated":1359872,"data_stored":1359872,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":9,"seq":38654705733,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52166656,"kb_used":1500,"kb_used_data":1500,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":52165156,"statfs":{"total":53418655744,"available":53417119744,"internally_reserved":0,"allocated":1536000,"data_stored":1536000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":1,"total":53418655744,"available":53417119744,"internally_reserved":0,"allocated":1536000,"data_stored":1536000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":53418655744,"available":53417295872,"internally_reserved":0,"allocated":1359872,"data_stored":1359872,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":53418655744,"available":53417119744,"internally_reserved":0,"allocated":1536000,"data_stored":1536000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":53418655744,"available":53418147840,"internally_reserved":0,"allocated":507904,"data_stored":507904,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-26T03:45:27.426 INFO:tasks.ceph:pgid 2.7 last_scrub_stamp 2026-02-26T03:44:18.219221+0000 time.struct_time(tm_year=2026, tm_mon=2, tm_mday=26, tm_hour=3, tm_min=44, tm_sec=18, tm_wday=3, tm_yday=57, tm_isdst=-1) <= time.struct_time(tm_year=2026, tm_mon=2, tm_mday=26, tm_hour=3, tm_min=45, tm_sec=25, tm_wday=3, tm_yday=57, tm_isdst=0) 2026-02-26T03:45:27.427 INFO:tasks.ceph:pgid 2.6 last_scrub_stamp 2026-02-26T03:44:18.219221+0000 time.struct_time(tm_year=2026, tm_mon=2, tm_mday=26, tm_hour=3, tm_min=44, tm_sec=18, tm_wday=3, tm_yday=57, tm_isdst=-1) <= time.struct_time(tm_year=2026, tm_mon=2, tm_mday=26, tm_hour=3, tm_min=45, tm_sec=25, tm_wday=3, tm_yday=57, tm_isdst=0) 2026-02-26T03:45:27.427 INFO:tasks.ceph:pgid 2.5 last_scrub_stamp 2026-02-26T03:44:18.219221+0000 time.struct_time(tm_year=2026, tm_mon=2, tm_mday=26, tm_hour=3, tm_min=44, tm_sec=18, tm_wday=3, tm_yday=57, tm_isdst=-1) <= time.struct_time(tm_year=2026, tm_mon=2, tm_mday=26, tm_hour=3, tm_min=45, tm_sec=25, tm_wday=3, tm_yday=57, tm_isdst=0) 2026-02-26T03:45:27.427 INFO:tasks.ceph:pgid 2.4 last_scrub_stamp 2026-02-26T03:44:18.219221+0000 time.struct_time(tm_year=2026, tm_mon=2, tm_mday=26, tm_hour=3, tm_min=44, tm_sec=18, tm_wday=3, tm_yday=57, tm_isdst=-1) <= time.struct_time(tm_year=2026, tm_mon=2, tm_mday=26, tm_hour=3, tm_min=45, tm_sec=25, tm_wday=3, tm_yday=57, tm_isdst=0) 2026-02-26T03:45:27.427 INFO:tasks.ceph:pgid 2.2 last_scrub_stamp 2026-02-26T03:44:18.219221+0000 time.struct_time(tm_year=2026, tm_mon=2, tm_mday=26, tm_hour=3, tm_min=44, tm_sec=18, tm_wday=3, tm_yday=57, tm_isdst=-1) <= time.struct_time(tm_year=2026, tm_mon=2, tm_mday=26, tm_hour=3, tm_min=45, tm_sec=25, tm_wday=3, tm_yday=57, tm_isdst=0) 2026-02-26T03:45:27.427 INFO:tasks.ceph:pgid 2.1 last_scrub_stamp 2026-02-26T03:44:18.219221+0000 time.struct_time(tm_year=2026, tm_mon=2, tm_mday=26, tm_hour=3, tm_min=44, tm_sec=18, tm_wday=3, tm_yday=57, tm_isdst=-1) <= time.struct_time(tm_year=2026, tm_mon=2, tm_mday=26, tm_hour=3, tm_min=45, tm_sec=25, tm_wday=3, tm_yday=57, tm_isdst=0) 2026-02-26T03:45:27.427 INFO:tasks.ceph:pgid 2.0 last_scrub_stamp 2026-02-26T03:44:18.219221+0000 time.struct_time(tm_year=2026, tm_mon=2, tm_mday=26, tm_hour=3, tm_min=44, tm_sec=18, tm_wday=3, tm_yday=57, tm_isdst=-1) <= time.struct_time(tm_year=2026, tm_mon=2, tm_mday=26, tm_hour=3, tm_min=45, tm_sec=25, tm_wday=3, tm_yday=57, tm_isdst=0) 2026-02-26T03:45:27.428 INFO:tasks.ceph:pgid 2.3 last_scrub_stamp 2026-02-26T03:44:18.219221+0000 time.struct_time(tm_year=2026, tm_mon=2, tm_mday=26, tm_hour=3, tm_min=44, tm_sec=18, tm_wday=3, tm_yday=57, tm_isdst=-1) <= time.struct_time(tm_year=2026, tm_mon=2, tm_mday=26, tm_hour=3, tm_min=45, tm_sec=25, tm_wday=3, tm_yday=57, tm_isdst=0) 2026-02-26T03:45:27.428 INFO:tasks.ceph:pgid 1.0 last_scrub_stamp 2026-02-26T03:44:15.206229+0000 time.struct_time(tm_year=2026, tm_mon=2, tm_mday=26, tm_hour=3, tm_min=44, tm_sec=15, tm_wday=3, tm_yday=57, tm_isdst=-1) <= time.struct_time(tm_year=2026, tm_mon=2, tm_mday=26, tm_hour=3, tm_min=45, tm_sec=25, tm_wday=3, tm_yday=57, tm_isdst=0) 2026-02-26T03:45:27.428 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2026-02-26T03:45:47.428 DEBUG:teuthology.orchestra.run.trial132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-26T03:45:47.511 INFO:teuthology.orchestra.run.trial132.stderr:2026-02-26T03:45:47.511+0000 7f4ec015c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-26T03:45:47.514 INFO:teuthology.orchestra.run.trial132.stderr:2026-02-26T03:45:47.515+0000 7f4ec015c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-26T03:45:47.625 INFO:teuthology.orchestra.run.trial132.stdout: 2026-02-26T03:45:47.626 INFO:teuthology.orchestra.run.trial132.stderr:dumped all 2026-02-26T03:45:47.635 INFO:teuthology.orchestra.run.trial132.stdout:{"pg_ready":true,"pg_map":{"version":61,"stamp":"2026-02-26T03:45:46.480136+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":579,"num_read_kb":1121,"num_write":349,"num_write_kb":2142,"num_scrub_errors":0,"num_shallow_scrub_errors":2,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":340,"ondisk_log_size":340,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":3,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":156499968,"kb_used":3180,"kb_used_data":3180,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":156496788,"statfs":{"total":160255967232,"available":160252710912,"internally_reserved":0,"allocated":3256320,"data_stored":3256320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002343"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":29,"reported_epoch":21,"state":"active+clean","last_fresh":"2026-02-26T03:45:26.919219+0000","last_change":"2026-02-26T03:45:26.930468+0000","last_active":"2026-02-26T03:45:26.919219+0000","last_peered":"2026-02-26T03:45:26.919219+0000","last_clean":"2026-02-26T03:45:26.919219+0000","last_became_active":"2026-02-26T03:44:19.263201+0000","last_became_peered":"2026-02-26T03:44:19.263201+0000","last_unstale":"2026-02-26T03:45:26.919219+0000","last_undegraded":"2026-02-26T03:45:26.919219+0000","last_fullsized":"2026-02-26T03:45:26.919219+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-26T03:45:26.930400+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-26T03:45:26.930400+0000","last_clean_scrub_stamp":"2026-02-26T03:44:18.219221+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"21'2","reported_seq":29,"reported_epoch":21,"state":"active+clean","last_fresh":"2026-02-26T03:45:26.919559+0000","last_change":"2026-02-26T03:45:26.935064+0000","last_active":"2026-02-26T03:45:26.919559+0000","last_peered":"2026-02-26T03:45:26.919559+0000","last_clean":"2026-02-26T03:45:26.919559+0000","last_became_active":"2026-02-26T03:44:19.261313+0000","last_became_peered":"2026-02-26T03:44:19.261313+0000","last_unstale":"2026-02-26T03:45:26.919559+0000","last_undegraded":"2026-02-26T03:45:26.919559+0000","last_fullsized":"2026-02-26T03:45:26.919559+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"21'2","last_scrub_stamp":"2026-02-26T03:45:26.935011+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-26T03:45:26.935011+0000","last_clean_scrub_stamp":"2026-02-26T03:44:18.219221+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"21'3","reported_seq":43,"reported_epoch":21,"state":"active+clean","last_fresh":"2026-02-26T03:45:26.919037+0000","last_change":"2026-02-26T03:45:26.931609+0000","last_active":"2026-02-26T03:45:26.919037+0000","last_peered":"2026-02-26T03:45:26.919037+0000","last_clean":"2026-02-26T03:45:26.919037+0000","last_became_active":"2026-02-26T03:44:19.268173+0000","last_became_peered":"2026-02-26T03:44:19.268173+0000","last_unstale":"2026-02-26T03:45:26.919037+0000","last_undegraded":"2026-02-26T03:45:26.919037+0000","last_fullsized":"2026-02-26T03:45:26.919037+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"21'3","last_scrub_stamp":"2026-02-26T03:45:26.931545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-26T03:45:26.931545+0000","last_clean_scrub_stamp":"2026-02-26T03:44:18.219221+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":20,"num_read_kb":6,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"21'17","reported_seq":134,"reported_epoch":21,"state":"active+clean","last_fresh":"2026-02-26T03:45:26.918282+0000","last_change":"2026-02-26T03:45:26.933131+0000","last_active":"2026-02-26T03:45:26.918282+0000","last_peered":"2026-02-26T03:45:26.918282+0000","last_clean":"2026-02-26T03:45:26.918282+0000","last_became_active":"2026-02-26T03:44:19.271270+0000","last_became_peered":"2026-02-26T03:44:19.271270+0000","last_unstale":"2026-02-26T03:45:26.918282+0000","last_undegraded":"2026-02-26T03:45:26.918282+0000","last_fullsized":"2026-02-26T03:45:26.918282+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"21'17","last_scrub_stamp":"2026-02-26T03:45:26.933051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-26T03:45:26.933051+0000","last_clean_scrub_stamp":"2026-02-26T03:44:18.219221+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":242,"num_read_kb":40,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":1,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"17'2","reported_seq":44,"reported_epoch":21,"state":"active+clean","last_fresh":"2026-02-26T03:45:26.620394+0000","last_change":"2026-02-26T03:45:26.630558+0000","last_active":"2026-02-26T03:45:26.620394+0000","last_peered":"2026-02-26T03:45:26.620394+0000","last_clean":"2026-02-26T03:45:26.620394+0000","last_became_active":"2026-02-26T03:44:19.276841+0000","last_became_peered":"2026-02-26T03:44:19.276841+0000","last_unstale":"2026-02-26T03:45:26.620394+0000","last_undegraded":"2026-02-26T03:45:26.620394+0000","last_fullsized":"2026-02-26T03:45:26.620394+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"17'2","last_scrub_stamp":"2026-02-26T03:45:26.630489+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-26T03:45:26.630489+0000","last_clean_scrub_stamp":"2026-02-26T03:44:18.219221+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":29,"reported_epoch":21,"state":"active+clean","last_fresh":"2026-02-26T03:45:27.217392+0000","last_change":"2026-02-26T03:45:27.225173+0000","last_active":"2026-02-26T03:45:27.217392+0000","last_peered":"2026-02-26T03:45:27.217392+0000","last_clean":"2026-02-26T03:45:27.217392+0000","last_became_active":"2026-02-26T03:44:19.283254+0000","last_became_peered":"2026-02-26T03:44:19.283254+0000","last_unstale":"2026-02-26T03:45:27.217392+0000","last_undegraded":"2026-02-26T03:45:27.217392+0000","last_fullsized":"2026-02-26T03:45:27.217392+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-26T03:45:27.225081+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-26T03:45:27.225081+0000","last_clean_scrub_stamp":"2026-02-26T03:44:18.219221+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":29,"reported_epoch":21,"state":"active+clean","last_fresh":"2026-02-26T03:45:27.216983+0000","last_change":"2026-02-26T03:45:27.224258+0000","last_active":"2026-02-26T03:45:27.216983+0000","last_peered":"2026-02-26T03:45:27.216983+0000","last_clean":"2026-02-26T03:45:27.216983+0000","last_became_active":"2026-02-26T03:44:19.283733+0000","last_became_peered":"2026-02-26T03:44:19.283733+0000","last_unstale":"2026-02-26T03:45:27.216983+0000","last_undegraded":"2026-02-26T03:45:27.216983+0000","last_fullsized":"2026-02-26T03:45:27.216983+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-26T03:45:27.224193+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-26T03:45:27.224193+0000","last_clean_scrub_stamp":"2026-02-26T03:44:18.219221+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"21'264","reported_seq":553,"reported_epoch":21,"state":"active+clean","last_fresh":"2026-02-26T03:45:26.918282+0000","last_change":"2026-02-26T03:45:26.932248+0000","last_active":"2026-02-26T03:45:26.918282+0000","last_peered":"2026-02-26T03:45:26.918282+0000","last_clean":"2026-02-26T03:45:26.918282+0000","last_became_active":"2026-02-26T03:44:19.259527+0000","last_became_peered":"2026-02-26T03:44:19.259527+0000","last_unstale":"2026-02-26T03:45:26.918282+0000","last_undegraded":"2026-02-26T03:45:26.918282+0000","last_fullsized":"2026-02-26T03:45:26.918282+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"21'264","last_scrub_stamp":"2026-02-26T03:45:26.932179+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-26T03:45:26.932179+0000","last_clean_scrub_stamp":"2026-02-26T03:44:18.219221+0000","objects_scrubbed":0,"log_size":264,"log_dups_size":0,"ondisk_log_size":264,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":263,"num_read_kb":1028,"num_write":261,"num_write_kb":1031,"num_scrub_errors":0,"num_shallow_scrub_errors":1,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"12'52","reported_seq":104,"reported_epoch":21,"state":"active+clean","last_fresh":"2026-02-26T03:45:26.917782+0000","last_change":"2026-02-26T03:45:26.942680+0000","last_active":"2026-02-26T03:45:26.917782+0000","last_peered":"2026-02-26T03:45:26.917782+0000","last_clean":"2026-02-26T03:45:26.917782+0000","last_became_active":"2026-02-26T03:44:16.240654+0000","last_became_peered":"2026-02-26T03:44:16.240654+0000","last_unstale":"2026-02-26T03:45:26.917782+0000","last_undegraded":"2026-02-26T03:45:26.917782+0000","last_fullsized":"2026-02-26T03:45:26.917782+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"12'52","last_scrub_stamp":"2026-02-26T03:45:26.942612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-26T03:45:26.942612+0000","last_clean_scrub_stamp":"2026-02-26T03:44:15.206229+0000","objects_scrubbed":0,"log_size":52,"log_dups_size":0,"ondisk_log_size":52,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":53,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":526,"num_read_kb":1075,"num_write":270,"num_write_kb":1036,"num_scrub_errors":0,"num_shallow_scrub_errors":2,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":160255967232,"available":160252710912,"internally_reserved":0,"allocated":3256320,"data_stored":3256320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":288,"ondisk_log_size":288,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":53,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":53418655744,"available":53417193472,"internally_reserved":0,"allocated":1462272,"data_stored":1462272,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":52,"ondisk_log_size":52,"up":2,"acting":2,"num_store_stats":1}],"osd_stats":[{"osd":2,"up_from":10,"seq":42949673049,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52166656,"kb_used":556,"kb_used_data":556,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":52166100,"statfs":{"total":53418655744,"available":53418086400,"internally_reserved":0,"allocated":569344,"data_stored":569344,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":10,"seq":42949673049,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52166656,"kb_used":1196,"kb_used_data":1196,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":52165460,"statfs":{"total":53418655744,"available":53417431040,"internally_reserved":0,"allocated":1224704,"data_stored":1224704,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":9,"seq":38654705753,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52166656,"kb_used":1428,"kb_used_data":1428,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":52165228,"statfs":{"total":53418655744,"available":53417193472,"internally_reserved":0,"allocated":1462272,"data_stored":1462272,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":1,"total":53418655744,"available":53417193472,"internally_reserved":0,"allocated":1462272,"data_stored":1462272,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":53418655744,"available":53417431040,"internally_reserved":0,"allocated":1224704,"data_stored":1224704,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":53418655744,"available":53417193472,"internally_reserved":0,"allocated":1462272,"data_stored":1462272,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":53418655744,"available":53418086400,"internally_reserved":0,"allocated":569344,"data_stored":569344,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-26T03:45:47.637 DEBUG:teuthology.orchestra.run.trial132:> sudo ceph --cluster ceph config set global mon_health_to_clog false 2026-02-26T03:45:47.712 INFO:teuthology.orchestra.run.trial132.stderr:2026-02-26T03:45:47.709+0000 7f2dbc851640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-26T03:45:47.714 INFO:teuthology.orchestra.run.trial132.stderr:2026-02-26T03:45:47.716+0000 7f2dbc851640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-26T03:45:47.839 INFO:teuthology.misc:Shutting down mds daemons... 2026-02-26T03:45:47.839 INFO:teuthology.misc:Shutting down osd daemons... 2026-02-26T03:45:47.839 DEBUG:tasks.ceph.osd.0:waiting for process to exit 2026-02-26T03:45:47.839 INFO:teuthology.orchestra.run:waiting for 300 2026-02-26T03:45:47.904 INFO:tasks.ceph.osd.0:Stopped 2026-02-26T03:45:47.905 DEBUG:tasks.ceph.osd.1:waiting for process to exit 2026-02-26T03:45:47.905 INFO:teuthology.orchestra.run:waiting for 300 2026-02-26T03:45:47.968 INFO:tasks.ceph.osd.1:Stopped 2026-02-26T03:45:47.968 DEBUG:tasks.ceph.osd.2:waiting for process to exit 2026-02-26T03:45:47.968 INFO:teuthology.orchestra.run:waiting for 300 2026-02-26T03:45:48.036 INFO:tasks.ceph.osd.2:Stopped 2026-02-26T03:45:48.036 INFO:teuthology.misc:Shutting down mgr daemons... 2026-02-26T03:45:48.037 DEBUG:tasks.ceph.mgr.x:waiting for process to exit 2026-02-26T03:45:48.037 INFO:teuthology.orchestra.run:waiting for 300 2026-02-26T03:45:48.072 INFO:tasks.ceph.mgr.x:Stopped 2026-02-26T03:45:48.072 INFO:teuthology.misc:Shutting down mon daemons... 2026-02-26T03:45:48.073 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2026-02-26T03:45:48.073 INFO:teuthology.orchestra.run:waiting for 300 2026-02-26T03:45:48.122 INFO:tasks.ceph.mon.a:Stopped 2026-02-26T03:45:48.122 INFO:tasks.ceph:Checking cluster log for badness... 2026-02-26T03:45:48.123 DEBUG:teuthology.orchestra.run.trial132:> sudo grep -E '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/ceph.log | grep -E -v '\(MDS_ALL_DOWN\)' | grep -E -v '\(MDS_UP_LESS_THAN_MAX\)' | grep -E -v '\(POOL_APP_NOT_ENABLED\)' | head -n 1 2026-02-26T03:45:48.190 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-0 on ubuntu@trial132.front.sepia.ceph.com 2026-02-26T03:45:48.190 DEBUG:teuthology.orchestra.run.trial132:> sync && sudo umount -f /var/lib/ceph/osd/ceph-0 2026-02-26T03:45:48.261 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-1 on ubuntu@trial132.front.sepia.ceph.com 2026-02-26T03:45:48.261 DEBUG:teuthology.orchestra.run.trial132:> sync && sudo umount -f /var/lib/ceph/osd/ceph-1 2026-02-26T03:45:48.378 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-2 on ubuntu@trial132.front.sepia.ceph.com 2026-02-26T03:45:48.378 DEBUG:teuthology.orchestra.run.trial132:> sync && sudo umount -f /var/lib/ceph/osd/ceph-2 2026-02-26T03:45:48.406 INFO:tasks.ceph:Archiving mon data... 2026-02-26T03:45:48.407 DEBUG:teuthology.misc:Transferring archived files from trial132:/var/lib/ceph/mon/ceph-a to /home/teuthworker/mnt/teuthology/shraddhaag-2026-02-26_02:22:56-crimson-rados-main-distro-debug-trial/71215/data/mon.a.tgz 2026-02-26T03:45:48.408 DEBUG:teuthology.orchestra.run.trial132:> mktemp 2026-02-26T03:45:48.463 INFO:teuthology.orchestra.run.trial132.stdout:/tmp/tmp.HWg3z9kkbd 2026-02-26T03:45:48.463 DEBUG:teuthology.orchestra.run.trial132:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-a -- . > /tmp/tmp.HWg3z9kkbd 2026-02-26T03:45:48.548 DEBUG:teuthology.orchestra.run.trial132:> sudo chmod 0666 /tmp/tmp.HWg3z9kkbd 2026-02-26T03:45:48.630 DEBUG:teuthology.orchestra.remote:trial132:/tmp/tmp.HWg3z9kkbd is 176KB 2026-02-26T03:45:48.683 DEBUG:teuthology.orchestra.run.trial132:> rm -fr /tmp/tmp.HWg3z9kkbd 2026-02-26T03:45:48.699 INFO:tasks.ceph:Cleaning ceph cluster... 2026-02-26T03:45:48.699 DEBUG:teuthology.orchestra.run.trial132:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2026-02-26T03:45:48.820 INFO:teuthology.util.scanner:summary_data or yaml_file is empty! 2026-02-26T03:45:48.821 INFO:tasks.ceph:Archiving crash dumps... 2026-02-26T03:45:48.823 DEBUG:teuthology.misc:Transferring archived files from trial132:/var/lib/ceph/crash to /home/teuthworker/mnt/teuthology/shraddhaag-2026-02-26_02:22:56-crimson-rados-main-distro-debug-trial/71215/remote/trial132/crash 2026-02-26T03:45:48.824 DEBUG:teuthology.orchestra.run.trial132:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2026-02-26T03:45:48.884 INFO:tasks.ceph:Compressing logs... 2026-02-26T03:45:48.884 DEBUG:teuthology.orchestra.run.trial132:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2026-02-26T03:45:48.949 INFO:teuthology.orchestra.run.trial132.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.2.log 2026-02-26T03:45:48.949 INFO:teuthology.orchestra.run.trial132.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42866.log 2026-02-26T03:45:48.949 INFO:teuthology.orchestra.run.trial132.stderr:/var/log/ceph/ceph-osd.2.log: gzip -5 --verbose -- /var/log/ceph/ceph-mon.a.log 2026-02-26T03:45:48.950 INFO:teuthology.orchestra.run.trial132.stderr:/var/log/ceph/ceph-client.admin.42866.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41209.log 2026-02-26T03:45:48.950 INFO:teuthology.orchestra.run.trial132.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.42866.log.gz 2026-02-26T03:45:48.950 INFO:teuthology.orchestra.run.trial132.stderr:/var/log/ceph/ceph-mon.a.log: gzip -5 --verbose -- /var/log/ceph/ceph.tmp-client.admin.38803.log 2026-02-26T03:45:48.950 INFO:teuthology.orchestra.run.trial132.stderr:/var/log/ceph/ceph-client.admin.41209.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.41209.log.gz 2026-02-26T03:45:48.950 INFO:teuthology.orchestra.run.trial132.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.0.log 2026-02-26T03:45:48.950 INFO:teuthology.orchestra.run.trial132.stderr:/var/log/ceph/ceph.tmp-client.admin.38803.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40381.log 2026-02-26T03:45:48.951 INFO:teuthology.orchestra.run.trial132.stderr: 68.0% -- replaced with /var/log/ceph/ceph.tmp-client.admin.38803.log.gz 2026-02-26T03:45:48.951 INFO:teuthology.orchestra.run.trial132.stderr:/var/log/ceph/ceph-osd.0.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41361.log 2026-02-26T03:45:48.951 INFO:teuthology.orchestra.run.trial132.stderr:/var/log/ceph/ceph-client.admin.40381.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42559.log 2026-02-26T03:45:48.951 INFO:teuthology.orchestra.run.trial132.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.40381.log.gz 2026-02-26T03:45:48.951 INFO:teuthology.orchestra.run.trial132.stderr:/var/log/ceph/ceph-client.admin.41361.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40197.log 2026-02-26T03:45:48.952 INFO:teuthology.orchestra.run.trial132.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.41361.log.gz 2026-02-26T03:45:48.952 INFO:teuthology.orchestra.run.trial132.stderr:/var/log/ceph/ceph-client.admin.42559.log: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.42559.log.gz 2026-02-26T03:45:48.952 INFO:teuthology.orchestra.run.trial132.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40694.log 2026-02-26T03:45:48.952 INFO:teuthology.orchestra.run.trial132.stderr:/var/log/ceph/ceph-client.admin.40197.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42804.log 2026-02-26T03:45:48.952 INFO:teuthology.orchestra.run.trial132.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.40197.log.gz 2026-02-26T03:45:48.952 INFO:teuthology.orchestra.run.trial132.stderr:/var/log/ceph/ceph-client.admin.40694.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42099.log 2026-02-26T03:45:48.952 INFO:teuthology.orchestra.run.trial132.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.40694.log.gz 2026-02-26T03:45:48.953 INFO:teuthology.orchestra.run.trial132.stderr:/var/log/ceph/ceph-client.admin.42804.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42657.log 2026-02-26T03:45:48.953 INFO:teuthology.orchestra.run.trial132.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.42804.log.gz 2026-02-26T03:45:48.953 INFO:teuthology.orchestra.run.trial132.stderr:/var/log/ceph/ceph-client.admin.42099.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42510.log 2026-02-26T03:45:48.953 INFO:teuthology.orchestra.run.trial132.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.42099.log.gz 2026-02-26T03:45:48.953 INFO:teuthology.orchestra.run.trial132.stderr:/var/log/ceph/ceph-client.admin.42657.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41601.log 2026-02-26T03:45:48.953 INFO:teuthology.orchestra.run.trial132.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.42657.log.gz 2026-02-26T03:45:48.953 INFO:teuthology.orchestra.run.trial132.stderr:/var/log/ceph/ceph-client.admin.42510.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41974.log 2026-02-26T03:45:48.953 INFO:teuthology.orchestra.run.trial132.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.42510.log.gz 2026-02-26T03:45:48.954 INFO:teuthology.orchestra.run.trial132.stderr:/var/log/ceph/ceph-client.admin.41601.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42706.log 2026-02-26T03:45:48.954 INFO:teuthology.orchestra.run.trial132.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.41601.log.gz 2026-02-26T03:45:48.954 INFO:teuthology.orchestra.run.trial132.stderr:/var/log/ceph/ceph-client.admin.41974.log: 66.7% -- replaced with /var/log/ceph/ceph-client.admin.41974.log.gz 2026-02-26T03:45:48.954 INFO:teuthology.orchestra.run.trial132.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41699.log 2026-02-26T03:45:48.954 INFO:teuthology.orchestra.run.trial132.stderr:/var/log/ceph/ceph-client.admin.42706.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40951.log 2026-02-26T03:45:48.954 INFO:teuthology.orchestra.run.trial132.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.42706.log.gz 2026-02-26T03:45:48.954 INFO:teuthology.orchestra.run.trial132.stderr:/var/log/ceph/ceph-client.admin.41699.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41046.log 2026-02-26T03:45:48.954 INFO:teuthology.orchestra.run.trial132.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.41699.log.gz 2026-02-26T03:45:48.955 INFO:teuthology.orchestra.run.trial132.stderr:/var/log/ceph/ceph-client.admin.40951.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40244.log 2026-02-26T03:45:48.955 INFO:teuthology.orchestra.run.trial132.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.40951.log.gz 2026-02-26T03:45:48.955 INFO:teuthology.orchestra.run.trial132.stderr:/var/log/ceph/ceph-client.admin.41046.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42608.log 2026-02-26T03:45:48.955 INFO:teuthology.orchestra.run.trial132.stderr: 66.7% -- replaced with /var/log/ceph/ceph-client.admin.41046.log.gz 2026-02-26T03:45:48.955 INFO:teuthology.orchestra.run.trial132.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41507.log 2026-02-26T03:45:48.955 INFO:teuthology.orchestra.run.trial132.stderr:/var/log/ceph/ceph-client.admin.40244.log: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.40244.log.gz 2026-02-26T03:45:48.955 INFO:teuthology.orchestra.run.trial132.stderr:/var/log/ceph/ceph-client.admin.42608.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42169.log 2026-02-26T03:45:48.955 INFO:teuthology.orchestra.run.trial132.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.42608.log.gz 2026-02-26T03:45:48.956 INFO:teuthology.orchestra.run.trial132.stderr:/var/log/ceph/ceph-client.admin.41507.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40475.log 2026-02-26T03:45:48.956 INFO:teuthology.orchestra.run.trial132.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.41507.log.gz 2026-02-26T03:45:48.956 INFO:teuthology.orchestra.run.trial132.stderr:gzip -5 --verbose/var/log/ceph/ceph-client.admin.42169.log: -- /var/log/ceph/ceph-client.admin.41650.log 2026-02-26T03:45:48.956 INFO:teuthology.orchestra.run.trial132.stderr: 66.9% -- replaced with /var/log/ceph/ceph-client.admin.42169.log.gz 2026-02-26T03:45:48.956 INFO:teuthology.orchestra.run.trial132.stderr:/var/log/ceph/ceph-client.admin.40475.log: gzip 50.0% -5 -- replaced with /var/log/ceph/ceph-client.admin.40475.log.gz --verbose 2026-02-26T03:45:48.956 INFO:teuthology.orchestra.run.trial132.stderr: -- /var/log/ceph/ceph-client.admin.42123.log 2026-02-26T03:45:48.956 INFO:teuthology.orchestra.run.trial132.stderr:/var/log/ceph/ceph-client.admin.41650.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42461.log 2026-02-26T03:45:48.956 INFO:teuthology.orchestra.run.trial132.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.41650.log.gz 2026-02-26T03:45:48.957 INFO:teuthology.orchestra.run.trial132.stderr:/var/log/ceph/ceph-client.admin.42123.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40102.log 2026-02-26T03:45:48.957 INFO:teuthology.orchestra.run.trial132.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.42123.log.gz 2026-02-26T03:45:48.957 INFO:teuthology.orchestra.run.trial132.stderr:/var/log/ceph/ceph-client.admin.42461.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42363.log 2026-02-26T03:45:48.957 INFO:teuthology.orchestra.run.trial132.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.42461.log.gz 2026-02-26T03:45:48.957 INFO:teuthology.orchestra.run.trial132.stderr:/var/log/ceph/ceph-client.admin.40102.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40883.log 2026-02-26T03:45:48.957 INFO:teuthology.orchestra.run.trial132.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.40102.log.gz 2026-02-26T03:45:48.957 INFO:teuthology.orchestra.run.trial132.stderr:/var/log/ceph/ceph-client.admin.42363.log: gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2026-02-26T03:45:48.957 INFO:teuthology.orchestra.run.trial132.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.42363.log.gz 2026-02-26T03:45:48.957 INFO:teuthology.orchestra.run.trial132.stderr:/var/log/ceph/ceph-client.admin.40883.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41111.log 2026-02-26T03:45:48.957 INFO:teuthology.orchestra.run.trial132.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.40883.log.gz 2026-02-26T03:45:48.957 INFO:teuthology.orchestra.run.trial132.stderr:gzip -5 --verbose/var/log/ceph/ceph.audit.log: -- /var/log/ceph/ceph-client.admin.41991.log 2026-02-26T03:45:48.958 INFO:teuthology.orchestra.run.trial132.stderr:/var/log/ceph/ceph-client.admin.41111.log: 86.8% -- replaced with /var/log/ceph/ceph.audit.log.gz 2026-02-26T03:45:48.958 INFO:teuthology.orchestra.run.trial132.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42054.log 2026-02-26T03:45:48.958 INFO:teuthology.orchestra.run.trial132.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.41111.log.gz 2026-02-26T03:45:48.958 INFO:teuthology.orchestra.run.trial132.stderr:/var/log/ceph/ceph-client.admin.41991.log: 58.5% -- replaced with /var/log/ceph/ceph-client.admin.41991.log.gz 2026-02-26T03:45:48.958 INFO:teuthology.orchestra.run.trial132.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41021.log 2026-02-26T03:45:48.958 INFO:teuthology.orchestra.run.trial132.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.1.log 2026-02-26T03:45:48.958 INFO:teuthology.orchestra.run.trial132.stderr:/var/log/ceph/ceph-client.admin.42054.log: /var/log/ceph/ceph-client.admin.41021.log: 66.4% -- replaced with /var/log/ceph/ceph-client.admin.42054.log.gz 2026-02-26T03:45:48.958 INFO:teuthology.orchestra.run.trial132.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40725.log 2026-02-26T03:45:48.958 INFO:teuthology.orchestra.run.trial132.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.41021.log.gz 2026-02-26T03:45:48.959 INFO:teuthology.orchestra.run.trial132.stderr:/var/log/ceph/ceph-osd.1.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41508.log 2026-02-26T03:45:48.959 INFO:teuthology.orchestra.run.trial132.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41359.log 2026-02-26T03:45:48.959 INFO:teuthology.orchestra.run.trial132.stderr:/var/log/ceph/ceph-client.admin.40725.log: /var/log/ceph/ceph-client.admin.41508.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.40725.log.gz 2026-02-26T03:45:48.959 INFO:teuthology.orchestra.run.trial132.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.41508.log.gz 2026-02-26T03:45:48.959 INFO:teuthology.orchestra.run.trial132.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41506.log 2026-02-26T03:45:48.959 INFO:teuthology.orchestra.run.trial132.stderr:/var/log/ceph/ceph-client.admin.41359.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42412.log 2026-02-26T03:45:48.959 INFO:teuthology.orchestra.run.trial132.stderr: 50.8% -- replaced with /var/log/ceph/ceph-client.admin.41359.log.gz 2026-02-26T03:45:48.959 INFO:teuthology.orchestra.run.trial132.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mgr.x.log 2026-02-26T03:45:48.960 INFO:teuthology.orchestra.run.trial132.stderr:/var/log/ceph/ceph-client.admin.41506.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.41506.log.gz 2026-02-26T03:45:48.960 INFO:teuthology.orchestra.run.trial132.stderr:/var/log/ceph/ceph-client.admin.42412.log: 49.2% -- replaced with /var/log/ceph/ceph-client.admin.42412.log.gz 2026-02-26T03:45:48.960 INFO:teuthology.orchestra.run.trial132.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42913.log 2026-02-26T03:45:48.960 INFO:teuthology.orchestra.run.trial132.stderr:/var/log/ceph/ceph-mgr.x.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41360.log 2026-02-26T03:45:48.960 INFO:teuthology.orchestra.run.trial132.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42755.log 2026-02-26T03:45:48.960 INFO:teuthology.orchestra.run.trial132.stderr:/var/log/ceph/ceph-client.admin.42913.log: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.42913.log.gz 2026-02-26T03:45:48.961 INFO:teuthology.orchestra.run.trial132.stderr:/var/log/ceph/ceph-client.admin.41360.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42314.log 2026-02-26T03:45:48.961 INFO:teuthology.orchestra.run.trial132.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.41360.log.gz 2026-02-26T03:45:48.961 INFO:teuthology.orchestra.run.trial132.stderr:/var/log/ceph/ceph-client.admin.42755.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41992.log 2026-02-26T03:45:48.961 INFO:teuthology.orchestra.run.trial132.stderr: 49.2% -- replaced with /var/log/ceph/ceph-client.admin.42755.log.gz 2026-02-26T03:45:48.961 INFO:teuthology.orchestra.run.trial132.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.log 2026-02-26T03:45:48.961 INFO:teuthology.orchestra.run.trial132.stderr:/var/log/ceph/ceph-client.admin.42314.log: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.42314.log.gz 2026-02-26T03:45:48.961 INFO:teuthology.orchestra.run.trial132.stderr:/var/log/ceph/ceph-client.admin.41992.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42077.log 2026-02-26T03:45:48.961 INFO:teuthology.orchestra.run.trial132.stderr: 66.9% -- replaced with /var/log/ceph/ceph-client.admin.41992.log.gz 2026-02-26T03:45:48.961 INFO:teuthology.orchestra.run.trial132.stderr:/var/log/ceph/ceph.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42148.log 2026-02-26T03:45:48.962 INFO:teuthology.orchestra.run.trial132.stderr: 84.4% -- replaced with /var/log/ceph/ceph.log.gz 2026-02-26T03:45:48.962 INFO:teuthology.orchestra.run.trial132.stderr:/var/log/ceph/ceph-client.admin.42077.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42032.log 2026-02-26T03:45:48.962 INFO:teuthology.orchestra.run.trial132.stderr: 66.7% -- replaced with /var/log/ceph/ceph-client.admin.42077.log.gz 2026-02-26T03:45:48.962 INFO:teuthology.orchestra.run.trial132.stderr:gzip -5 --verbose/var/log/ceph/ceph-client.admin.42148.log: -- /var/log/ceph/ceph-client.admin.41160.log 2026-02-26T03:45:48.962 INFO:teuthology.orchestra.run.trial132.stderr: 66.9% -- replaced with /var/log/ceph/ceph-client.admin.42148.log.gz 2026-02-26T03:45:48.962 INFO:teuthology.orchestra.run.trial132.stderr:/var/log/ceph/ceph-client.admin.42032.log: 66.7% -- replaced with /var/log/ceph/ceph-client.admin.42032.log.gz 2026-02-26T03:45:48.962 INFO:teuthology.orchestra.run.trial132.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40428.log 2026-02-26T03:45:48.962 INFO:teuthology.orchestra.run.trial132.stderr:/var/log/ceph/ceph-client.admin.41160.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41258.log 2026-02-26T03:45:48.962 INFO:teuthology.orchestra.run.trial132.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.41160.log.gz 2026-02-26T03:45:48.962 INFO:teuthology.orchestra.run.trial132.stderr:/var/log/ceph/ceph-client.admin.40428.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.40428.log.gz 2026-02-26T03:45:48.963 INFO:teuthology.orchestra.run.trial132.stderr:/var/log/ceph/ceph-client.admin.41258.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.41258.log.gz 2026-02-26T03:45:48.969 INFO:teuthology.orchestra.run.trial132.stderr: 91.3% -- replaced with /var/log/ceph/ceph-mgr.x.log.gz 2026-02-26T03:45:48.978 INFO:teuthology.orchestra.run.trial132.stderr: 91.0% -- replaced with /var/log/ceph/ceph-mon.a.log.gz 2026-02-26T03:45:49.238 INFO:teuthology.orchestra.run.trial132.stderr: 94.8% -- replaced with /var/log/ceph/ceph-osd.0.log.gz 2026-02-26T03:45:49.283 INFO:teuthology.orchestra.run.trial132.stderr: 94.3% -- replaced with /var/log/ceph/ceph-osd.2.log.gz 2026-02-26T03:45:49.416 INFO:teuthology.orchestra.run.trial132.stderr: 94.1% -- replaced with /var/log/ceph/ceph-osd.1.log.gz 2026-02-26T03:45:49.417 INFO:teuthology.orchestra.run.trial132.stderr: 2026-02-26T03:45:49.417 INFO:teuthology.orchestra.run.trial132.stderr:real 0m0.476s 2026-02-26T03:45:49.417 INFO:teuthology.orchestra.run.trial132.stderr:user 0m1.072s 2026-02-26T03:45:49.417 INFO:teuthology.orchestra.run.trial132.stderr:sys 0m0.093s 2026-02-26T03:45:49.418 INFO:tasks.ceph:Archiving logs... 2026-02-26T03:45:49.419 DEBUG:teuthology.misc:Transferring archived files from trial132:/var/log/ceph to /home/teuthworker/mnt/teuthology/shraddhaag-2026-02-26_02:22:56-crimson-rados-main-distro-debug-trial/71215/remote/trial132/log 2026-02-26T03:45:49.420 DEBUG:teuthology.orchestra.run.trial132:> sudo tar c -f - -C /var/log/ceph -- . 2026-02-26T03:45:49.773 DEBUG:teuthology.run_tasks:Unwinding manager install 2026-02-26T03:45:49.837 DEBUG:teuthology.run_tasks:Unwinding manager install 2026-02-26T03:45:49.849 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2026-02-26T03:45:49.849 DEBUG:teuthology.orchestra.run.trial132:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2026-02-26T03:45:49.875 DEBUG:teuthology.run_tasks:Unwinding manager clock 2026-02-26T03:45:49.888 INFO:teuthology.task.clock:Checking final clock skew... 2026-02-26T03:45:49.888 DEBUG:teuthology.orchestra.run.trial132:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2026-02-26T03:45:49.930 INFO:teuthology.orchestra.run.trial132.stderr:bash: line 1: ntpq: command not found 2026-02-26T03:45:49.947 INFO:teuthology.orchestra.run.trial132.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2026-02-26T03:45:49.947 INFO:teuthology.orchestra.run.trial132.stdout:=============================================================================== 2026-02-26T03:45:49.947 INFO:teuthology.orchestra.run.trial132.stdout:^* 96-19-94-82.cpe.sparklig> 1 7 377 1 +3332us[+3452us] +/- 33ms 2026-02-26T03:45:49.947 INFO:teuthology.orchestra.run.trial132.stdout:^+ ntp11.kernfusion.at 2 6 377 2 +3298us[+3418us] +/- 72ms 2026-02-26T03:45:49.947 INFO:teuthology.orchestra.run.trial132.stdout:^+ 170-187-142-180.ip.linod> 2 7 377 1 -8920us[-8920us] +/- 83ms 2026-02-26T03:45:49.947 INFO:teuthology.orchestra.run.trial132.stdout:^+ 172-232-15-202.ip.linode> 2 6 377 4 -2588us[-2468us] +/- 75ms 2026-02-26T03:45:49.948 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2026-02-26T03:45:49.960 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2026-02-26T03:45:49.960 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2026-02-26T03:45:49.976 DEBUG:teuthology.orchestra.run.trial132:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2026-02-26T03:45:50.043 DEBUG:teuthology.orchestra.run.trial132:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 2026-02-26T03:45:50.070 DEBUG:teuthology.orchestra.run:got remote process result: 1 2026-02-26T03:45:50.070 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2026-02-26T03:45:50.087 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2026-02-26T03:45:50.103 INFO:teuthology.task.internal:Duration was 1674.829093 seconds 2026-02-26T03:45:50.103 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2026-02-26T03:45:50.119 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2026-02-26T03:45:50.119 DEBUG:teuthology.orchestra.run.trial132:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2026-02-26T03:45:50.156 INFO:teuthology.orchestra.run.trial132.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2026-02-26T03:45:50.506 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2026-02-26T03:45:50.506 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@trial132.front.sepia.ceph.com 2026-02-26T03:45:50.506 DEBUG:teuthology.orchestra.run.trial132:> grep -E --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | grep -E -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | grep -E -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2026-02-26T03:45:50.564 INFO:teuthology.task.internal.syslog:Gathering journactl... 2026-02-26T03:45:50.564 DEBUG:teuthology.orchestra.run.trial132:> sudo journalctl > /home/ubuntu/cephtest/archive/syslog/journalctl.log 2026-02-26T03:45:50.666 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2026-02-26T03:45:50.666 DEBUG:teuthology.orchestra.run.trial132:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --max-args=1 --max-procs=0 --verbose --no-run-if-empty -- gzip -5 --verbose -- 2026-02-26T03:45:50.690 INFO:teuthology.orchestra.run.trial132.stderr:gzip -5 --verbose -- /home/ubuntu/cephtest/archive/syslog/misc.log 2026-02-26T03:45:50.690 INFO:teuthology.orchestra.run.trial132.stderr:gzip -5 --verbose -- /home/ubuntu/cephtest/archive/syslog/kern.log 2026-02-26T03:45:50.690 INFO:teuthology.orchestra.run.trial132.stderr:/home/ubuntu/cephtest/archive/syslog/misc.log: 0.0% -- replaced with /home/ubuntu/cephtest/archive/syslog/misc.log.gz 2026-02-26T03:45:50.690 INFO:teuthology.orchestra.run.trial132.stderr:gzip -5 --verbose -- /home/ubuntu/cephtest/archive/syslog/journalctl.log 2026-02-26T03:45:50.691 INFO:teuthology.orchestra.run.trial132.stderr:/home/ubuntu/cephtest/archive/syslog/kern.log: 0.0% -- replaced with /home/ubuntu/cephtest/archive/syslog/kern.log.gz 2026-02-26T03:45:50.701 INFO:teuthology.orchestra.run.trial132.stderr:/home/ubuntu/cephtest/archive/syslog/journalctl.log: 82.2% -- replaced with /home/ubuntu/cephtest/archive/syslog/journalctl.log.gz 2026-02-26T03:45:50.703 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2026-02-26T03:45:50.720 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2026-02-26T03:45:50.721 DEBUG:teuthology.orchestra.run.trial132:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2026-02-26T03:45:50.768 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2026-02-26T03:45:50.787 DEBUG:teuthology.orchestra.run.trial132:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2026-02-26T03:45:50.832 INFO:teuthology.orchestra.run.trial132.stdout:kernel.core_pattern = core 2026-02-26T03:45:50.847 DEBUG:teuthology.orchestra.run.trial132:> test -e /home/ubuntu/cephtest/archive/coredump 2026-02-26T03:45:50.903 DEBUG:teuthology.orchestra.run:got remote process result: 1 2026-02-26T03:45:50.904 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2026-02-26T03:45:50.916 INFO:teuthology.task.internal:Transferring archived files... 2026-02-26T03:45:50.917 DEBUG:teuthology.misc:Transferring archived files from trial132:/home/ubuntu/cephtest/archive to /home/teuthworker/mnt/teuthology/shraddhaag-2026-02-26_02:22:56-crimson-rados-main-distro-debug-trial/71215/remote/trial132 2026-02-26T03:45:50.917 DEBUG:teuthology.orchestra.run.trial132:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2026-02-26T03:45:50.980 INFO:teuthology.task.internal:Removing archive directory... 2026-02-26T03:45:50.981 DEBUG:teuthology.orchestra.run.trial132:> rm -rf -- /home/ubuntu/cephtest/archive 2026-02-26T03:45:51.027 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2026-02-26T03:45:51.039 INFO:teuthology.task.internal:Not uploading archives. 2026-02-26T03:45:51.039 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2026-02-26T03:45:51.070 INFO:teuthology.task.internal:Tidying up after the test... 2026-02-26T03:45:51.070 DEBUG:teuthology.orchestra.run.trial132:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2026-02-26T03:45:51.088 INFO:teuthology.orchestra.run.trial132.stdout: 921600 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Feb 26 03:45 /home/ubuntu/cephtest 2026-02-26T03:45:51.089 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2026-02-26T03:45:51.107 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2026-02-26T03:45:51.144 INFO:teuthology.run:Summary data: description: crimson-rados/rbd/{0-crimson_install clusters/crimson-fixed-1 crimson-supported-all-distro/centos_latest crimson_qa_overrides deploy/ceph objectstore/seastore/segmented$/{crimson_seastore_segmented_2q} tasks/rbd_lock_and_fence} duration: 1674.829092502594 flavor: debug owner: scheduled_shraddhaag@soko04.front.sepia.ceph.com success: true 2026-02-26T03:45:51.144 DEBUG:teuthology.report:Pushing job info to https://paddles-paddles.apps.pok.os.sepia.ceph.com 2026-02-26T03:45:51.256 INFO:teuthology.run:pass