2026-02-23T10:46:53.429 INFO:root:teuthology version: 1.2.4.dev9+gfa17720d0 2026-02-23T10:46:53.444 DEBUG:teuthology.report:Pushing job info to https://paddles-paddles.apps.pok.os.sepia.ceph.com 2026-02-23T10:46:53.519 INFO:teuthology.run:Config: archive_path: /home/teuthworker/mnt/teuthology/shraddhaag-2026-02-23_09:58:48-crimson-rados-main-distro-debug-trial/66138 branch: main description: crimson-rados/rbd/{0-crimson_install clusters/crimson-fixed-1 crimson-supported-all-distro/centos_latest crimson_qa_overrides deploy/ceph objectstore/crimson_bluestore tasks/rbd_cls_tests} email: null first_in_suite: false flavor: debug job_id: '66138' kernel: branch: distro kdb: true sha1: distro last_in_suite: false machine_type: trial name: shraddhaag-2026-02-23_09:58:48-crimson-rados-main-distro-debug-trial no_nested_subset: false os_type: centos os_version: 9.stream overrides: admin_socket: branch: main ceph: conf: global: enable experimental unrecoverable data corrupting features: crimson ms client mode: crc ms cluster mode: crc ms mon client mode: crc ms mon cluster mode: crc ms mon service mode: crc ms service mode: crc mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 osd pool default crimson: true osd: bluestore block size: 96636764160 bluestore compression algorithm: snappy bluestore compression mode: aggressive bluestore fsck on mount: true bluestore rocksdb cf: false crimson cpu num: 4 crimson osd obc lru size: 10 debug alienstore: 20 debug bluefs: 20 debug bluestore: 20 debug ms: 20 debug osd: 20 debug rocksdb: 10 err to stderr: true log flush on exit: true log to file: false log to stderr: true mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore osd shutdown pgref assert: true seastore cachepin size pershard: 64M seastore max concurrent transactions: 8 crimson_compat: true flavor: debug fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - \(POOL_APP_NOT_ENABLED\) sha1: e371433fcd1390a0b82cfb18effc422c1f5d8c1d ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log global: osd crush chooseleaf type: 0 osd pool default pg num: 128 osd pool default pgp num: 128 osd pool default size: 2 mon: {} install: ceph: flavor: debug sha1: e371433fcd1390a0b82cfb18effc422c1f5d8c1d workunit: branch: wip-shraddhaag-enable-cephadm-tests env: CRIMSON_COMPAT: '1' sha1: e371433fcd1390a0b82cfb18effc422c1f5d8c1d owner: scheduled_shraddhaag@soko04.front.sepia.ceph.com priority: 49 repo: https://git.ceph.com/ceph-ci.git roles: - - mon.a - mgr.x - osd.0 - osd.1 - osd.2 - client.0 seed: 66 sha1: e371433fcd1390a0b82cfb18effc422c1f5d8c1d sleep_before_teardown: 0 suite: crimson-rados suite_branch: wip-shraddhaag-enable-cephadm-tests suite_path: /home/teuthworker/src/github.com_shraddhaag_ceph_e371433fcd1390a0b82cfb18effc422c1f5d8c1d/qa suite_relpath: qa suite_repo: https://github.com/shraddhaag/ceph.git suite_sha1: e371433fcd1390a0b82cfb18effc422c1f5d8c1d targets: trial198.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBHtvwhffeaME518p75g+XtQwTPPiA5DEOAj0oujaqjsIvUXvQpVvVpvPzoBOuIouJhlPNX1VUcy0PDrX8gWDbq0= tasks: - install: extra_packages: deb: - ceph-osd-crimson - ceph-osd-crimson-dbg rpm: - ceph-osd-crimson - ceph-osd-crimson-debuginfo - pexec: all: - sudo update-alternatives --set ceph-osd /usr/bin/ceph-osd-crimson - install: null - ceph: conf: mon: mon min osdmap epochs: 50 mon osdmap full prune interval: 2 mon osdmap full prune min: 15 mon osdmap full prune txsize: 2 paxos service trim min: 10 osd: debug monc: 20 - workunit: clients: client.0: - cls/test_cls_rbd.sh - cls/test_cls_lock.sh - cls/test_cls_journal.sh teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: fa17720d0088c3ac28e473468bfc79eeaff5cd38 timestamp: 2026-02-23_09:58:48 tube: trial user: shraddhaag verbose: false worker_log: /home/teuthworker/mnt/teuthology/worker_logs/dispatcher.trial.3318653 2026-02-23T10:46:53.520 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/github.com_shraddhaag_ceph_e371433fcd1390a0b82cfb18effc422c1f5d8c1d/qa; will attempt to use it 2026-02-23T10:46:53.520 INFO:teuthology.run:Found tasks at /home/teuthworker/src/github.com_shraddhaag_ceph_e371433fcd1390a0b82cfb18effc422c1f5d8c1d/qa/tasks 2026-02-23T10:46:53.520 INFO:teuthology.run_tasks:Running task internal.check_packages... 2026-02-23T10:46:53.522 INFO:teuthology.task.internal:Checking packages... 2026-02-23T10:46:53.522 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'debug' and ceph hash 'e371433fcd1390a0b82cfb18effc422c1f5d8c1d' 2026-02-23T10:46:53.522 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2026-02-23T10:46:53.522 INFO:teuthology.packaging:ref: None 2026-02-23T10:46:53.522 INFO:teuthology.packaging:tag: None 2026-02-23T10:46:53.522 INFO:teuthology.packaging:branch: main 2026-02-23T10:46:53.522 INFO:teuthology.packaging:sha1: e371433fcd1390a0b82cfb18effc422c1f5d8c1d 2026-02-23T10:46:53.522 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=debug&distros=centos%2F9%2Fx86_64&ref=main 2026-02-23T10:46:53.667 INFO:teuthology.task.internal:Found packages for ceph version 20.3.0-5253.g5ccdd6ad 2026-02-23T10:46:53.668 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2026-02-23T10:46:53.680 INFO:teuthology.task.internal:no buildpackages task found 2026-02-23T10:46:53.680 INFO:teuthology.run_tasks:Running task internal.save_config... 2026-02-23T10:46:53.693 INFO:teuthology.task.internal:Saving configuration 2026-02-23T10:46:53.706 INFO:teuthology.run_tasks:Running task internal.check_lock... 2026-02-23T10:46:53.718 INFO:teuthology.task.internal.check_lock:Checking locks... 2026-02-23T10:46:53.797 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'trial198.front.sepia.ceph.com', 'description': '/home/teuthworker/mnt/teuthology/shraddhaag-2026-02-23_09:58:48-crimson-rados-main-distro-debug-trial/66138', 'up': True, 'machine_type': 'trial', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9.stream', 'arch': 'x86_64', 'locked': True, 'locked_since': '2026-02-23 10:43:27.226604', 'locked_by': 'scheduled_shraddhaag@soko04.front.sepia.ceph.com', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBHtvwhffeaME518p75g+XtQwTPPiA5DEOAj0oujaqjsIvUXvQpVvVpvPzoBOuIouJhlPNX1VUcy0PDrX8gWDbq0='} 2026-02-23T10:46:53.797 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2026-02-23T10:46:53.809 INFO:teuthology.task.internal:roles: ubuntu@trial198.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'osd.0', 'osd.1', 'osd.2', 'client.0'] 2026-02-23T10:46:53.809 INFO:teuthology.run_tasks:Running task console_log... 2026-02-23T10:46:53.894 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7ce15a6d5260>, signals=[15]) 2026-02-23T10:46:53.894 INFO:teuthology.run_tasks:Running task internal.connect... 2026-02-23T10:46:53.905 INFO:teuthology.task.internal:Opening connections... 2026-02-23T10:46:53.905 DEBUG:teuthology.task.internal:connecting to ubuntu@trial198.front.sepia.ceph.com 2026-02-23T10:46:53.906 DEBUG:teuthology.orchestra.connection:{'hostname': 'trial198.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2026-02-23T10:46:53.965 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2026-02-23T10:46:53.977 DEBUG:teuthology.orchestra.run.trial198:> uname -m 2026-02-23T10:46:54.044 INFO:teuthology.orchestra.run.trial198.stdout:x86_64 2026-02-23T10:46:54.044 DEBUG:teuthology.orchestra.run.trial198:> cat /etc/os-release 2026-02-23T10:46:54.100 INFO:teuthology.orchestra.run.trial198.stdout:NAME="CentOS Stream" 2026-02-23T10:46:54.100 INFO:teuthology.orchestra.run.trial198.stdout:VERSION="9" 2026-02-23T10:46:54.100 INFO:teuthology.orchestra.run.trial198.stdout:ID="centos" 2026-02-23T10:46:54.100 INFO:teuthology.orchestra.run.trial198.stdout:ID_LIKE="rhel fedora" 2026-02-23T10:46:54.100 INFO:teuthology.orchestra.run.trial198.stdout:VERSION_ID="9" 2026-02-23T10:46:54.100 INFO:teuthology.orchestra.run.trial198.stdout:PLATFORM_ID="platform:el9" 2026-02-23T10:46:54.100 INFO:teuthology.orchestra.run.trial198.stdout:PRETTY_NAME="CentOS Stream 9" 2026-02-23T10:46:54.100 INFO:teuthology.orchestra.run.trial198.stdout:ANSI_COLOR="0;31" 2026-02-23T10:46:54.100 INFO:teuthology.orchestra.run.trial198.stdout:LOGO="fedora-logo-icon" 2026-02-23T10:46:54.100 INFO:teuthology.orchestra.run.trial198.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2026-02-23T10:46:54.101 INFO:teuthology.orchestra.run.trial198.stdout:HOME_URL="https://centos.org/" 2026-02-23T10:46:54.101 INFO:teuthology.orchestra.run.trial198.stdout:BUG_REPORT_URL="https://issues.redhat.com/" 2026-02-23T10:46:54.101 INFO:teuthology.orchestra.run.trial198.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2026-02-23T10:46:54.101 INFO:teuthology.orchestra.run.trial198.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2026-02-23T10:46:54.102 INFO:teuthology.lock.ops:Updating trial198.front.sepia.ceph.com on lock server 2026-02-23T10:46:54.194 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2026-02-23T10:46:54.207 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2026-02-23T10:46:54.218 INFO:teuthology.task.internal:Checking for old test directory... 2026-02-23T10:46:54.219 DEBUG:teuthology.orchestra.run.trial198:> test '!' -e /home/ubuntu/cephtest 2026-02-23T10:46:54.234 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2026-02-23T10:46:54.245 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2026-02-23T10:46:54.245 DEBUG:teuthology.orchestra.run.trial198:> test -z $(ls -A /var/lib/ceph) 2026-02-23T10:46:54.290 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2026-02-23T10:46:54.374 INFO:teuthology.run_tasks:Running task kernel... 2026-02-23T10:46:54.394 INFO:teuthology.task.kernel:normalize config orig: {'branch': 'distro', 'kdb': True, 'sha1': 'distro'} 2026-02-23T10:46:54.394 INFO:teuthology.task.kernel:config {'mon.a': {'branch': 'distro', 'kdb': True, 'sha1': 'distro'}}, timeout 300 2026-02-23T10:46:54.395 DEBUG:teuthology.orchestra.run.trial198:> test -f /run/.containerenv -o -f /.dockerenv 2026-02-23T10:46:54.409 DEBUG:teuthology.orchestra.run:got remote process result: 1 2026-02-23T10:46:54.409 DEBUG:teuthology.orchestra.run.trial198:> uname -r 2026-02-23T10:46:54.464 INFO:teuthology.orchestra.run.trial198.stdout:5.14.0-681.el9.x86_64 2026-02-23T10:46:54.464 INFO:teuthology.task.kernel:Running kernel on trial198: 5.14.0-681.el9.x86_64 2026-02-23T10:46:54.464 DEBUG:teuthology.orchestra.run.trial198:> sudo yum install -y kernel 2026-02-23T10:46:55.696 INFO:teuthology.orchestra.run.trial198.stdout:CentOS Stream 9 - BaseOS 10 MB/s | 8.9 MB 00:00 2026-02-23T10:46:59.173 INFO:teuthology.orchestra.run.trial198.stdout:CentOS Stream 9 - AppStream 9.1 MB/s | 27 MB 00:02 2026-02-23T10:47:03.461 INFO:teuthology.orchestra.run.trial198.stdout:CentOS Stream 9 - CRB 4.0 MB/s | 7.8 MB 00:01 2026-02-23T10:47:05.280 INFO:teuthology.orchestra.run.trial198.stdout:CentOS Stream 9 - Extras packages 18 kB/s | 20 kB 00:01 2026-02-23T10:47:05.953 INFO:teuthology.orchestra.run.trial198.stdout:Extra Packages for Enterprise Linux 33 MB/s | 20 MB 00:00 2026-02-23T10:47:09.167 INFO:teuthology.orchestra.run.trial198.stdout:lab-extras 1.7 MB/s | 50 kB 00:00 2026-02-23T10:47:10.176 INFO:teuthology.orchestra.run.trial198.stdout:Package kernel-5.14.0-645.el9.x86_64 is already installed. 2026-02-23T10:47:10.176 INFO:teuthology.orchestra.run.trial198.stdout:Package kernel-5.14.0-681.el9.x86_64 is already installed. 2026-02-23T10:47:10.195 INFO:teuthology.orchestra.run.trial198.stdout:Dependencies resolved. 2026-02-23T10:47:10.196 INFO:teuthology.orchestra.run.trial198.stdout:Nothing to do. 2026-02-23T10:47:10.196 INFO:teuthology.orchestra.run.trial198.stdout:Complete! 2026-02-23T10:47:10.220 DEBUG:teuthology.orchestra.run.trial198:> echo no | sudo yum reinstall kernel || true 2026-02-23T10:47:10.513 INFO:teuthology.orchestra.run.trial198.stdout:Last metadata expiration check: 0:00:01 ago on Mon 23 Feb 2026 10:47:09 AM UTC. 2026-02-23T10:47:10.594 INFO:teuthology.orchestra.run.trial198.stdout:Dependencies resolved. 2026-02-23T10:47:10.595 INFO:teuthology.orchestra.run.trial198.stdout:================================================================================ 2026-02-23T10:47:10.595 INFO:teuthology.orchestra.run.trial198.stdout: Package Architecture Version Repository Size 2026-02-23T10:47:10.595 INFO:teuthology.orchestra.run.trial198.stdout:================================================================================ 2026-02-23T10:47:10.595 INFO:teuthology.orchestra.run.trial198.stdout:Reinstalling: 2026-02-23T10:47:10.595 INFO:teuthology.orchestra.run.trial198.stdout: kernel x86_64 5.14.0-681.el9 baseos 803 k 2026-02-23T10:47:10.595 INFO:teuthology.orchestra.run.trial198.stdout: 2026-02-23T10:47:10.595 INFO:teuthology.orchestra.run.trial198.stdout:Transaction Summary 2026-02-23T10:47:10.595 INFO:teuthology.orchestra.run.trial198.stdout:================================================================================ 2026-02-23T10:47:10.595 INFO:teuthology.orchestra.run.trial198.stdout: 2026-02-23T10:47:10.595 INFO:teuthology.orchestra.run.trial198.stdout:Total download size: 803 k 2026-02-23T10:47:10.596 INFO:teuthology.orchestra.run.trial198.stdout:Installed size: 0 2026-02-23T10:47:10.596 INFO:teuthology.orchestra.run.trial198.stderr:Operation aborted. 2026-02-23T10:47:10.614 INFO:teuthology.orchestra.run.trial198.stdout:Is this ok [y/N]: 2026-02-23T10:47:10.614 DEBUG:teuthology.orchestra.run.trial198:> sudo yum reinstall -y kernel || true 2026-02-23T10:47:10.911 INFO:teuthology.orchestra.run.trial198.stdout:Last metadata expiration check: 0:00:01 ago on Mon 23 Feb 2026 10:47:09 AM UTC. 2026-02-23T10:47:10.991 INFO:teuthology.orchestra.run.trial198.stdout:Dependencies resolved. 2026-02-23T10:47:10.991 INFO:teuthology.orchestra.run.trial198.stdout:================================================================================ 2026-02-23T10:47:10.991 INFO:teuthology.orchestra.run.trial198.stdout: Package Architecture Version Repository Size 2026-02-23T10:47:10.991 INFO:teuthology.orchestra.run.trial198.stdout:================================================================================ 2026-02-23T10:47:10.992 INFO:teuthology.orchestra.run.trial198.stdout:Reinstalling: 2026-02-23T10:47:10.992 INFO:teuthology.orchestra.run.trial198.stdout: kernel x86_64 5.14.0-681.el9 baseos 803 k 2026-02-23T10:47:10.992 INFO:teuthology.orchestra.run.trial198.stdout: 2026-02-23T10:47:10.992 INFO:teuthology.orchestra.run.trial198.stdout:Transaction Summary 2026-02-23T10:47:10.992 INFO:teuthology.orchestra.run.trial198.stdout:================================================================================ 2026-02-23T10:47:10.992 INFO:teuthology.orchestra.run.trial198.stdout: 2026-02-23T10:47:10.992 INFO:teuthology.orchestra.run.trial198.stdout:Total download size: 803 k 2026-02-23T10:47:10.992 INFO:teuthology.orchestra.run.trial198.stdout:Installed size: 0 2026-02-23T10:47:10.992 INFO:teuthology.orchestra.run.trial198.stdout:Downloading Packages: 2026-02-23T10:47:12.000 INFO:teuthology.orchestra.run.trial198.stdout:kernel-5.14.0-681.el9.x86_64.rpm 899 kB/s | 803 kB 00:00 2026-02-23T10:47:12.000 INFO:teuthology.orchestra.run.trial198.stdout:-------------------------------------------------------------------------------- 2026-02-23T10:47:12.000 INFO:teuthology.orchestra.run.trial198.stdout:Total 796 kB/s | 803 kB 00:01 2026-02-23T10:47:12.009 INFO:teuthology.orchestra.run.trial198.stdout:Running transaction check 2026-02-23T10:47:12.042 INFO:teuthology.orchestra.run.trial198.stdout:Transaction check succeeded. 2026-02-23T10:47:12.043 INFO:teuthology.orchestra.run.trial198.stdout:Running transaction test 2026-02-23T10:47:12.047 INFO:teuthology.orchestra.run.trial198.stdout:Transaction test succeeded. 2026-02-23T10:47:12.047 INFO:teuthology.orchestra.run.trial198.stdout:Running transaction 2026-02-23T10:47:12.071 INFO:teuthology.orchestra.run.trial198.stdout: Preparing : 1/1 2026-02-23T10:47:12.075 INFO:teuthology.orchestra.run.trial198.stdout: Reinstalling : kernel-5.14.0-681.el9.x86_64 1/2 2026-02-23T10:47:12.075 INFO:teuthology.orchestra.run.trial198.stdout: Cleanup : kernel-5.14.0-681.el9.x86_64 2/2 2026-02-23T10:47:12.182 INFO:teuthology.orchestra.run.trial198.stdout: Running scriptlet: kernel-5.14.0-681.el9.x86_64 2/2 2026-02-23T10:47:12.182 INFO:teuthology.orchestra.run.trial198.stdout: Verifying : kernel-5.14.0-681.el9.x86_64 1/2 2026-02-23T10:47:12.238 INFO:teuthology.orchestra.run.trial198.stdout: Verifying : kernel-5.14.0-681.el9.x86_64 2/2 2026-02-23T10:47:12.238 INFO:teuthology.orchestra.run.trial198.stdout: 2026-02-23T10:47:12.239 INFO:teuthology.orchestra.run.trial198.stdout:Reinstalled: 2026-02-23T10:47:12.239 INFO:teuthology.orchestra.run.trial198.stdout: kernel-5.14.0-681.el9.x86_64 2026-02-23T10:47:12.239 INFO:teuthology.orchestra.run.trial198.stdout: 2026-02-23T10:47:12.239 INFO:teuthology.orchestra.run.trial198.stdout:Complete! 2026-02-23T10:47:12.265 DEBUG:teuthology.orchestra.run.trial198:> rpm -q kernel | sort -rV | head -n 1 2026-02-23T10:47:12.288 INFO:teuthology.orchestra.run.trial198.stdout:kernel-5.14.0-681.el9.x86_64 2026-02-23T10:47:12.290 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-681.el9.x86_64 2026-02-23T10:47:12.290 INFO:teuthology.task.kernel:Newest distro kernel installed and running 2026-02-23T10:47:12.290 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2026-02-23T10:47:12.290 DEBUG:teuthology.orchestra.run.trial198:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2026-02-23T10:47:12.354 INFO:teuthology.orchestra.run.trial198.stdout:ttyS1 2026-02-23T10:47:12.380 DEBUG:teuthology.parallel:result is None 2026-02-23T10:47:12.380 INFO:teuthology.run_tasks:Running task internal.base... 2026-02-23T10:47:12.391 INFO:teuthology.task.internal:Creating test directory... 2026-02-23T10:47:12.392 DEBUG:teuthology.orchestra.run.trial198:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2026-02-23T10:47:12.435 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2026-02-23T10:47:12.447 INFO:teuthology.run_tasks:Running task internal.archive... 2026-02-23T10:47:12.460 INFO:teuthology.task.internal:Creating archive directory... 2026-02-23T10:47:12.461 DEBUG:teuthology.orchestra.run.trial198:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2026-02-23T10:47:12.497 INFO:teuthology.run_tasks:Running task internal.coredump... 2026-02-23T10:47:12.509 INFO:teuthology.task.internal:Enabling coredump saving... 2026-02-23T10:47:12.509 DEBUG:teuthology.orchestra.run.trial198:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2026-02-23T10:47:12.556 INFO:teuthology.orchestra.run.trial198.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-23T10:47:12.565 INFO:teuthology.orchestra.run.trial198.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-23T10:47:12.567 INFO:teuthology.run_tasks:Running task internal.sudo... 2026-02-23T10:47:12.579 INFO:teuthology.task.internal:Configuring sudo... 2026-02-23T10:47:12.579 DEBUG:teuthology.orchestra.run.trial198:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2026-02-23T10:47:12.631 INFO:teuthology.run_tasks:Running task internal.syslog... 2026-02-23T10:47:12.647 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2026-02-23T10:47:12.648 DEBUG:teuthology.orchestra.run.trial198:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2026-02-23T10:47:12.688 DEBUG:teuthology.orchestra.run.trial198:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2026-02-23T10:47:12.850 DEBUG:teuthology.orchestra.run.trial198:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2026-02-23T10:47:12.874 DEBUG:teuthology.orchestra.run.trial198:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2026-02-23T10:47:13.017 DEBUG:teuthology.orchestra.run.trial198:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2026-02-23T10:47:13.040 DEBUG:teuthology.orchestra.run.trial198:> set -ex 2026-02-23T10:47:13.040 DEBUG:teuthology.orchestra.run.trial198:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2026-02-23T10:47:13.103 DEBUG:teuthology.orchestra.run.trial198:> sudo service rsyslog restart 2026-02-23T10:47:13.171 INFO:teuthology.orchestra.run.trial198.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2026-02-23T10:47:13.450 INFO:teuthology.run_tasks:Running task internal.timer... 2026-02-23T10:47:13.462 INFO:teuthology.task.internal:Starting timer... 2026-02-23T10:47:13.462 INFO:teuthology.run_tasks:Running task pcp... 2026-02-23T10:47:13.480 INFO:teuthology.run_tasks:Running task selinux... 2026-02-23T10:47:13.498 DEBUG:teuthology.orchestra.run.trial198:> sudo service auditd rotate 2026-02-23T10:47:13.526 INFO:teuthology.orchestra.run.trial198.stdout:Rotating logs: 2026-02-23T10:47:13.528 DEBUG:teuthology.task.selinux:Getting current SELinux state 2026-02-23T10:47:13.528 DEBUG:teuthology.orchestra.run.trial198:> /usr/sbin/getenforce 2026-02-23T10:47:13.583 INFO:teuthology.orchestra.run.trial198.stdout:Permissive 2026-02-23T10:47:13.584 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@trial198.front.sepia.ceph.com': 'permissive'} 2026-02-23T10:47:13.584 DEBUG:teuthology.orchestra.run.trial198:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 2026-02-23T10:47:13.648 DEBUG:teuthology.orchestra.run:got remote process result: 1 2026-02-23T10:47:13.649 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2026-02-23T10:47:13.649 DEBUG:teuthology.orchestra.run.trial198:> sudo /usr/sbin/setenforce permissive 2026-02-23T10:47:13.713 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2026-02-23T10:47:13.731 DEBUG:teuthology.repo_utils:Setting repo remote to https://git.ceph.com/ceph-cm-ansible.git 2026-02-23T10:47:13.734 INFO:teuthology.repo_utils:Fetching git.ceph.com_ceph-cm-ansible_main from origin 2026-02-23T10:47:13.857 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2026-02-23T10:47:13.868 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2026-02-23T10:47:13.869 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit trial198.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2026-02-23T10:49:54.949 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@trial198.front.sepia.ceph.com')] 2026-02-23T10:49:54.951 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@trial198.front.sepia.ceph.com' 2026-02-23T10:49:54.953 DEBUG:teuthology.orchestra.connection:{'hostname': 'trial198.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2026-02-23T10:49:55.064 DEBUG:teuthology.orchestra.run.trial198:> true 2026-02-23T10:49:55.136 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@trial198.front.sepia.ceph.com' 2026-02-23T10:49:55.136 INFO:teuthology.run_tasks:Running task clock... 2026-02-23T10:49:55.149 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2026-02-23T10:49:55.149 INFO:teuthology.orchestra.run:Running command with timeout 360 2026-02-23T10:49:55.150 DEBUG:teuthology.orchestra.run.trial198:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2026-02-23T10:49:55.205 INFO:teuthology.orchestra.run.trial198.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2026-02-23T10:49:55.218 INFO:teuthology.orchestra.run.trial198.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2026-02-23T10:49:55.239 INFO:teuthology.orchestra.run.trial198.stderr:sudo: ntpd: command not found 2026-02-23T10:49:55.249 INFO:teuthology.orchestra.run.trial198.stdout:506 Cannot talk to daemon 2026-02-23T10:49:55.260 INFO:teuthology.orchestra.run.trial198.stderr:Failed to start ntp.service: Unit ntp.service not found. 2026-02-23T10:49:55.273 INFO:teuthology.orchestra.run.trial198.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2026-02-23T10:49:55.318 INFO:teuthology.orchestra.run.trial198.stderr:bash: line 1: ntpq: command not found 2026-02-23T10:49:55.320 INFO:teuthology.orchestra.run.trial198.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2026-02-23T10:49:55.320 INFO:teuthology.orchestra.run.trial198.stdout:=============================================================================== 2026-02-23T10:49:55.320 INFO:teuthology.run_tasks:Running task install... 2026-02-23T10:49:55.334 DEBUG:teuthology.task.install:project ceph 2026-02-23T10:49:55.334 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'debug', 'sha1': 'e371433fcd1390a0b82cfb18effc422c1f5d8c1d'}} 2026-02-23T10:49:55.335 DEBUG:teuthology.task.install:config {'extra_packages': {'deb': ['ceph-osd-crimson', 'ceph-osd-crimson-dbg'], 'rpm': ['ceph-osd-crimson', 'ceph-osd-crimson-debuginfo']}, 'flavor': 'debug', 'sha1': 'e371433fcd1390a0b82cfb18effc422c1f5d8c1d'} 2026-02-23T10:49:55.335 INFO:teuthology.task.install:Using flavor: debug 2026-02-23T10:49:55.339 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-osd', 'ceph-osd-classic', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-osd', 'ceph-osd-classic', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2026-02-23T10:49:55.339 INFO:teuthology.task.install:extra packages: {'deb': ['ceph-osd-crimson', 'ceph-osd-crimson-dbg'], 'rpm': ['ceph-osd-crimson', 'ceph-osd-crimson-debuginfo']} 2026-02-23T10:49:55.339 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['ceph-osd-crimson', 'ceph-osd-crimson-dbg'], 'rpm': ['ceph-osd-crimson', 'ceph-osd-crimson-debuginfo']}, 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'debug', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': 'e371433fcd1390a0b82cfb18effc422c1f5d8c1d', 'tag': None, 'wait_for_package': False} 2026-02-23T10:49:55.339 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=debug&distros=centos%2F9%2Fx86_64&sha1=e371433fcd1390a0b82cfb18effc422c1f5d8c1d 2026-02-23T10:49:55.507 INFO:teuthology.task.install.rpm:Pulling from https://3.chacra.ceph.com/r/ceph/wip-shraddhaag-enable-cephadm-tests/e371433fcd1390a0b82cfb18effc422c1f5d8c1d/centos/9/flavors/debug/ 2026-02-23T10:49:55.507 INFO:teuthology.task.install.rpm:Package version is 20.3.0-5439.ge371433f 2026-02-23T10:49:55.653 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://3.chacra.ceph.com/r/ceph/wip-shraddhaag-enable-cephadm-tests/e371433fcd1390a0b82cfb18effc422c1f5d8c1d/centos/9/flavors/debug/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://3.chacra.ceph.com/r/ceph/wip-shraddhaag-enable-cephadm-tests/e371433fcd1390a0b82cfb18effc422c1f5d8c1d/centos/9/flavors/debug/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://3.chacra.ceph.com/r/ceph/wip-shraddhaag-enable-cephadm-tests/e371433fcd1390a0b82cfb18effc422c1f5d8c1d/centos/9/flavors/debug/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2026-02-23T10:49:55.653 DEBUG:teuthology.orchestra.run.trial198:> set -ex 2026-02-23T10:49:55.653 DEBUG:teuthology.orchestra.run.trial198:> sudo dd of=/etc/yum.repos.d/ceph.repo 2026-02-23T10:49:55.679 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-osd, ceph-osd-classic, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, ceph-osd-crimson, ceph-osd-crimson-debuginfo on remote rpm x86_64 2026-02-23T10:49:55.679 DEBUG:teuthology.orchestra.run.trial198:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/e371433fcd1390a0b82cfb18effc422c1f5d8c1d/;g' /etc/yum.repos.d/ceph.repo ; fi 2026-02-23T10:49:55.743 DEBUG:teuthology.orchestra.run.trial198:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2026-02-23T10:49:55.816 DEBUG:teuthology.orchestra.run.trial198:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2026-02-23T10:49:55.880 INFO:teuthology.orchestra.run.trial198.stdout:check_obsoletes = 1 2026-02-23T10:49:55.881 DEBUG:teuthology.orchestra.run.trial198:> sudo yum clean all 2026-02-23T10:49:56.039 INFO:teuthology.orchestra.run.trial198.stdout:41 files removed 2026-02-23T10:49:56.055 DEBUG:teuthology.orchestra.run.trial198:> sudo yum -y install ceph-radosgw 2026-02-23T10:49:56.328 INFO:teuthology.orchestra.run.trial198.stdout:ceph packages for x86_64 715 kB/s | 92 kB 00:00 2026-02-23T10:49:56.474 INFO:teuthology.orchestra.run.trial198.stdout:ceph noarch packages 199 kB/s | 24 kB 00:00 2026-02-23T10:49:56.602 INFO:teuthology.orchestra.run.trial198.stdout:ceph source packages 19 kB/s | 2.1 kB 00:00 2026-02-23T10:49:58.245 INFO:teuthology.orchestra.run.trial198.stdout:CentOS Stream 9 - BaseOS 5.5 MB/s | 8.9 MB 00:01 2026-02-23T10:50:25.280 INFO:teuthology.orchestra.run.trial198.stdout:CentOS Stream 9 - AppStream 1.0 MB/s | 27 MB 00:26 2026-02-23T10:50:27.891 INFO:teuthology.orchestra.run.trial198.stdout:CentOS Stream 9 - CRB 36 MB/s | 7.8 MB 00:00 2026-02-23T10:50:29.204 INFO:teuthology.orchestra.run.trial198.stdout:CentOS Stream 9 - Extras packages 35 kB/s | 20 kB 00:00 2026-02-23T10:50:29.778 INFO:teuthology.orchestra.run.trial198.stdout:Extra Packages for Enterprise Linux 40 MB/s | 20 MB 00:00 2026-02-23T10:50:32.976 INFO:teuthology.orchestra.run.trial198.stdout:lab-extras 2.6 MB/s | 50 kB 00:00 2026-02-23T10:50:34.175 INFO:teuthology.orchestra.run.trial198.stdout:Dependencies resolved. 2026-02-23T10:50:34.176 INFO:teuthology.orchestra.run.trial198.stdout:================================================================================ 2026-02-23T10:50:34.176 INFO:teuthology.orchestra.run.trial198.stdout: Package Arch Version Repository Size 2026-02-23T10:50:34.176 INFO:teuthology.orchestra.run.trial198.stdout:================================================================================ 2026-02-23T10:50:34.176 INFO:teuthology.orchestra.run.trial198.stdout:Installing: 2026-02-23T10:50:34.177 INFO:teuthology.orchestra.run.trial198.stdout: ceph-radosgw x86_64 2:20.3.0-5439.ge371433f.el9 ceph 11 M 2026-02-23T10:50:34.177 INFO:teuthology.orchestra.run.trial198.stdout:Upgrading: 2026-02-23T10:50:34.177 INFO:teuthology.orchestra.run.trial198.stdout: librados2 x86_64 2:20.3.0-5439.ge371433f.el9 ceph 3.3 M 2026-02-23T10:50:34.177 INFO:teuthology.orchestra.run.trial198.stdout: librbd1 x86_64 2:20.3.0-5439.ge371433f.el9 ceph 2.7 M 2026-02-23T10:50:34.177 INFO:teuthology.orchestra.run.trial198.stdout:Installing dependencies: 2026-02-23T10:50:34.177 INFO:teuthology.orchestra.run.trial198.stdout: boost-program-options x86_64 1.75.0-13.el9 appstream 104 k 2026-02-23T10:50:34.177 INFO:teuthology.orchestra.run.trial198.stdout: ceph-base x86_64 2:20.3.0-5439.ge371433f.el9 ceph 5.1 M 2026-02-23T10:50:34.177 INFO:teuthology.orchestra.run.trial198.stdout: ceph-common x86_64 2:20.3.0-5439.ge371433f.el9 ceph 21 M 2026-02-23T10:50:34.177 INFO:teuthology.orchestra.run.trial198.stdout: ceph-selinux x86_64 2:20.3.0-5439.ge371433f.el9 ceph 25 k 2026-02-23T10:50:34.177 INFO:teuthology.orchestra.run.trial198.stdout: gperftools-libs x86_64 2.9.1-3.el9 epel 308 k 2026-02-23T10:50:34.177 INFO:teuthology.orchestra.run.trial198.stdout: libarrow x86_64 9.0.0-15.el9 epel 4.4 M 2026-02-23T10:50:34.178 INFO:teuthology.orchestra.run.trial198.stdout: libarrow-doc noarch 9.0.0-15.el9 epel 25 k 2026-02-23T10:50:34.178 INFO:teuthology.orchestra.run.trial198.stdout: libcephfs2 x86_64 2:20.3.0-5439.ge371433f.el9 ceph 860 k 2026-02-23T10:50:34.178 INFO:teuthology.orchestra.run.trial198.stdout: libnbd x86_64 1.20.3-4.el9 appstream 164 k 2026-02-23T10:50:34.178 INFO:teuthology.orchestra.run.trial198.stdout: liboath x86_64 2.6.12-1.el9 epel 49 k 2026-02-23T10:50:34.178 INFO:teuthology.orchestra.run.trial198.stdout: libpmemobj x86_64 1.12.1-1.el9 appstream 160 k 2026-02-23T10:50:34.178 INFO:teuthology.orchestra.run.trial198.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2026-02-23T10:50:34.178 INFO:teuthology.orchestra.run.trial198.stdout: libradosstriper1 x86_64 2:20.3.0-5439.ge371433f.el9 ceph 263 k 2026-02-23T10:50:34.178 INFO:teuthology.orchestra.run.trial198.stdout: librdkafka x86_64 1.6.1-102.el9 appstream 662 k 2026-02-23T10:50:34.178 INFO:teuthology.orchestra.run.trial198.stdout: librgw2 x86_64 2:20.3.0-5439.ge371433f.el9 ceph 5.6 M 2026-02-23T10:50:34.178 INFO:teuthology.orchestra.run.trial198.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2026-02-23T10:50:34.178 INFO:teuthology.orchestra.run.trial198.stdout: lttng-ust x86_64 2.12.0-6.el9 appstream 292 k 2026-02-23T10:50:34.179 INFO:teuthology.orchestra.run.trial198.stdout: mailcap noarch 2.1.49-5.el9 baseos 33 k 2026-02-23T10:50:34.179 INFO:teuthology.orchestra.run.trial198.stdout: parquet-libs x86_64 9.0.0-15.el9 epel 838 k 2026-02-23T10:50:34.179 INFO:teuthology.orchestra.run.trial198.stdout: pciutils x86_64 3.7.0-7.el9 baseos 93 k 2026-02-23T10:50:34.179 INFO:teuthology.orchestra.run.trial198.stdout: python3-ceph-argparse x86_64 2:20.3.0-5439.ge371433f.el9 ceph 45 k 2026-02-23T10:50:34.179 INFO:teuthology.orchestra.run.trial198.stdout: python3-ceph-common x86_64 2:20.3.0-5439.ge371433f.el9 ceph 186 k 2026-02-23T10:50:34.179 INFO:teuthology.orchestra.run.trial198.stdout: python3-cephfs x86_64 2:20.3.0-5439.ge371433f.el9 ceph 204 k 2026-02-23T10:50:34.179 INFO:teuthology.orchestra.run.trial198.stdout: python3-rados x86_64 2:20.3.0-5439.ge371433f.el9 ceph 324 k 2026-02-23T10:50:34.179 INFO:teuthology.orchestra.run.trial198.stdout: python3-rbd x86_64 2:20.3.0-5439.ge371433f.el9 ceph 305 k 2026-02-23T10:50:34.179 INFO:teuthology.orchestra.run.trial198.stdout: python3-rgw x86_64 2:20.3.0-5439.ge371433f.el9 ceph 100 k 2026-02-23T10:50:34.179 INFO:teuthology.orchestra.run.trial198.stdout: qatlib x86_64 25.08.0-2.el9 appstream 240 k 2026-02-23T10:50:34.179 INFO:teuthology.orchestra.run.trial198.stdout: qatzip-libs x86_64 1.3.1-1.el9 appstream 66 k 2026-02-23T10:50:34.180 INFO:teuthology.orchestra.run.trial198.stdout: re2 x86_64 1:20211101-20.el9 epel 191 k 2026-02-23T10:50:34.180 INFO:teuthology.orchestra.run.trial198.stdout: thrift x86_64 0.15.0-4.el9 epel 1.6 M 2026-02-23T10:50:34.180 INFO:teuthology.orchestra.run.trial198.stdout:Installing weak dependencies: 2026-02-23T10:50:34.180 INFO:teuthology.orchestra.run.trial198.stdout: qatlib-service x86_64 25.08.0-2.el9 appstream 37 k 2026-02-23T10:50:34.180 INFO:teuthology.orchestra.run.trial198.stdout: 2026-02-23T10:50:34.180 INFO:teuthology.orchestra.run.trial198.stdout:Transaction Summary 2026-02-23T10:50:34.180 INFO:teuthology.orchestra.run.trial198.stdout:================================================================================ 2026-02-23T10:50:34.180 INFO:teuthology.orchestra.run.trial198.stdout:Install 32 Packages 2026-02-23T10:50:34.180 INFO:teuthology.orchestra.run.trial198.stdout:Upgrade 2 Packages 2026-02-23T10:50:34.180 INFO:teuthology.orchestra.run.trial198.stdout: 2026-02-23T10:50:34.181 INFO:teuthology.orchestra.run.trial198.stdout:Total download size: 60 M 2026-02-23T10:50:34.181 INFO:teuthology.orchestra.run.trial198.stdout:Downloading Packages: 2026-02-23T10:50:36.175 INFO:teuthology.orchestra.run.trial198.stdout:(1/34): ceph-base-20.3.0-5439.ge371433f.el9.x86 3.4 MB/s | 5.1 MB 00:01 2026-02-23T10:50:36.229 INFO:teuthology.orchestra.run.trial198.stdout:(2/34): ceph-selinux-20.3.0-5439.ge371433f.el9. 466 kB/s | 25 kB 00:00 2026-02-23T10:50:36.410 INFO:teuthology.orchestra.run.trial198.stdout:(3/34): libcephfs2-20.3.0-5439.ge371433f.el9.x8 4.6 MB/s | 860 kB 00:00 2026-02-23T10:50:36.472 INFO:teuthology.orchestra.run.trial198.stdout:(4/34): libradosstriper1-20.3.0-5439.ge371433f. 4.1 MB/s | 263 kB 00:00 2026-02-23T10:50:37.551 INFO:teuthology.orchestra.run.trial198.stdout:(5/34): librgw2-20.3.0-5439.ge371433f.el9.x86_6 5.2 MB/s | 5.6 MB 00:01 2026-02-23T10:50:37.594 INFO:teuthology.orchestra.run.trial198.stdout:(6/34): python3-ceph-argparse-20.3.0-5439.ge371 1.0 MB/s | 45 kB 00:00 2026-02-23T10:50:37.643 INFO:teuthology.orchestra.run.trial198.stdout:(7/34): python3-ceph-common-20.3.0-5439.ge37143 3.8 MB/s | 186 kB 00:00 2026-02-23T10:50:37.693 INFO:teuthology.orchestra.run.trial198.stdout:(8/34): python3-cephfs-20.3.0-5439.ge371433f.el 4.0 MB/s | 204 kB 00:00 2026-02-23T10:50:37.748 INFO:teuthology.orchestra.run.trial198.stdout:(9/34): python3-rados-20.3.0-5439.ge371433f.el9 5.8 MB/s | 324 kB 00:00 2026-02-23T10:50:37.802 INFO:teuthology.orchestra.run.trial198.stdout:(10/34): python3-rbd-20.3.0-5439.ge371433f.el9. 5.5 MB/s | 305 kB 00:00 2026-02-23T10:50:37.848 INFO:teuthology.orchestra.run.trial198.stdout:(11/34): python3-rgw-20.3.0-5439.ge371433f.el9. 2.1 MB/s | 100 kB 00:00 2026-02-23T10:50:37.931 INFO:teuthology.orchestra.run.trial198.stdout:(12/34): mailcap-2.1.49-5.el9.noarch.rpm 401 kB/s | 33 kB 00:00 2026-02-23T10:50:37.979 INFO:teuthology.orchestra.run.trial198.stdout:(13/34): pciutils-3.7.0-7.el9.x86_64.rpm 1.9 MB/s | 93 kB 00:00 2026-02-23T10:50:38.379 INFO:teuthology.orchestra.run.trial198.stdout:(14/34): boost-program-options-1.75.0-13.el9.x8 260 kB/s | 104 kB 00:00 2026-02-23T10:50:38.558 INFO:teuthology.orchestra.run.trial198.stdout:(15/34): libnbd-1.20.3-4.el9.x86_64.rpm 913 kB/s | 164 kB 00:00 2026-02-23T10:50:38.672 INFO:teuthology.orchestra.run.trial198.stdout:(16/34): libpmemobj-1.12.1-1.el9.x86_64.rpm 1.4 MB/s | 160 kB 00:00 2026-02-23T10:50:38.760 INFO:teuthology.orchestra.run.trial198.stdout:(17/34): librabbitmq-0.11.0-7.el9.x86_64.rpm 517 kB/s | 45 kB 00:00 2026-02-23T10:50:38.985 INFO:teuthology.orchestra.run.trial198.stdout:(18/34): librdkafka-1.6.1-102.el9.x86_64.rpm 2.9 MB/s | 662 kB 00:00 2026-02-23T10:50:39.085 INFO:teuthology.orchestra.run.trial198.stdout:(19/34): lttng-ust-2.12.0-6.el9.x86_64.rpm 2.9 MB/s | 292 kB 00:00 2026-02-23T10:50:39.179 INFO:teuthology.orchestra.run.trial198.stdout:(20/34): qatlib-25.08.0-2.el9.x86_64.rpm 2.5 MB/s | 240 kB 00:00 2026-02-23T10:50:39.260 INFO:teuthology.orchestra.run.trial198.stdout:(21/34): qatlib-service-25.08.0-2.el9.x86_64.rp 465 kB/s | 37 kB 00:00 2026-02-23T10:50:39.342 INFO:teuthology.orchestra.run.trial198.stdout:(22/34): qatzip-libs-1.3.1-1.el9.x86_64.rpm 804 kB/s | 66 kB 00:00 2026-02-23T10:50:39.576 INFO:teuthology.orchestra.run.trial198.stdout:(23/34): gperftools-libs-2.9.1-3.el9.x86_64.rpm 1.3 MB/s | 308 kB 00:00 2026-02-23T10:50:39.691 INFO:teuthology.orchestra.run.trial198.stdout:(24/34): ceph-radosgw-20.3.0-5439.ge371433f.el9 2.2 MB/s | 11 MB 00:05 2026-02-23T10:50:39.738 INFO:teuthology.orchestra.run.trial198.stdout:(25/34): libarrow-9.0.0-15.el9.x86_64.rpm 27 MB/s | 4.4 MB 00:00 2026-02-23T10:50:39.769 INFO:teuthology.orchestra.run.trial198.stdout:(26/34): liboath-2.6.12-1.el9.x86_64.rpm 1.6 MB/s | 49 kB 00:00 2026-02-23T10:50:39.800 INFO:teuthology.orchestra.run.trial198.stdout:(27/34): libunwind-1.6.2-1.el9.x86_64.rpm 2.2 MB/s | 67 kB 00:00 2026-02-23T10:50:39.820 INFO:teuthology.orchestra.run.trial198.stdout:(28/34): libarrow-doc-9.0.0-15.el9.noarch.rpm 192 kB/s | 25 kB 00:00 2026-02-23T10:50:39.834 INFO:teuthology.orchestra.run.trial198.stdout:(29/34): parquet-libs-9.0.0-15.el9.x86_64.rpm 24 MB/s | 838 kB 00:00 2026-02-23T10:50:39.884 INFO:teuthology.orchestra.run.trial198.stdout:(30/34): thrift-0.15.0-4.el9.x86_64.rpm 32 MB/s | 1.6 MB 00:00 2026-02-23T10:50:39.930 INFO:teuthology.orchestra.run.trial198.stdout:(31/34): re2-20211101-20.el9.x86_64.rpm 1.7 MB/s | 191 kB 00:00 2026-02-23T10:50:40.280 INFO:teuthology.orchestra.run.trial198.stdout:(32/34): ceph-common-20.3.0-5439.ge371433f.el9. 3.7 MB/s | 21 MB 00:05 2026-02-23T10:50:40.396 INFO:teuthology.orchestra.run.trial198.stdout:(33/34): librados2-20.3.0-5439.ge371433f.el9.x8 6.4 MB/s | 3.3 MB 00:00 2026-02-23T10:50:40.595 INFO:teuthology.orchestra.run.trial198.stdout:(34/34): librbd1-20.3.0-5439.ge371433f.el9.x86_ 4.0 MB/s | 2.7 MB 00:00 2026-02-23T10:50:40.600 INFO:teuthology.orchestra.run.trial198.stdout:-------------------------------------------------------------------------------- 2026-02-23T10:50:40.600 INFO:teuthology.orchestra.run.trial198.stdout:Total 9.4 MB/s | 60 MB 00:06 2026-02-23T10:50:40.685 INFO:teuthology.orchestra.run.trial198.stdout:Running transaction check 2026-02-23T10:50:40.702 INFO:teuthology.orchestra.run.trial198.stdout:Transaction check succeeded. 2026-02-23T10:50:40.702 INFO:teuthology.orchestra.run.trial198.stdout:Running transaction test 2026-02-23T10:50:40.946 INFO:teuthology.orchestra.run.trial198.stdout:Transaction test succeeded. 2026-02-23T10:50:40.946 INFO:teuthology.orchestra.run.trial198.stdout:Running transaction 2026-02-23T10:50:41.226 INFO:teuthology.orchestra.run.trial198.stdout: Preparing : 1/1 2026-02-23T10:50:41.368 INFO:teuthology.orchestra.run.trial198.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/36 2026-02-23T10:50:41.373 INFO:teuthology.orchestra.run.trial198.stdout: Upgrading : librados2-2:20.3.0-5439.ge371433f.el9.x86_64 2/36 2026-02-23T10:50:41.400 INFO:teuthology.orchestra.run.trial198.stdout: Running scriptlet: librados2-2:20.3.0-5439.ge371433f.el9.x86_64 2/36 2026-02-23T10:50:41.436 INFO:teuthology.orchestra.run.trial198.stdout: Installing : python3-rados-2:20.3.0-5439.ge371433f.el9.x86_64 3/36 2026-02-23T10:50:41.437 INFO:teuthology.orchestra.run.trial198.stdout: Installing : libcephfs2-2:20.3.0-5439.ge371433f.el9.x86_64 4/36 2026-02-23T10:50:41.448 INFO:teuthology.orchestra.run.trial198.stdout: Running scriptlet: libcephfs2-2:20.3.0-5439.ge371433f.el9.x86_64 4/36 2026-02-23T10:50:41.456 INFO:teuthology.orchestra.run.trial198.stdout: Installing : liboath-2.6.12-1.el9.x86_64 5/36 2026-02-23T10:50:41.459 INFO:teuthology.orchestra.run.trial198.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 6/36 2026-02-23T10:50:41.463 INFO:teuthology.orchestra.run.trial198.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 7/36 2026-02-23T10:50:41.467 INFO:teuthology.orchestra.run.trial198.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 8/36 2026-02-23T10:50:41.478 INFO:teuthology.orchestra.run.trial198.stdout: Installing : python3-ceph-argparse-2:20.3.0-5439.ge371433f.el9. 9/36 2026-02-23T10:50:41.490 INFO:teuthology.orchestra.run.trial198.stdout: Installing : python3-cephfs-2:20.3.0-5439.ge371433f.el9.x86_64 10/36 2026-02-23T10:50:41.491 INFO:teuthology.orchestra.run.trial198.stdout: Installing : libradosstriper1-2:20.3.0-5439.ge371433f.el9.x86_6 11/36 2026-02-23T10:50:41.512 INFO:teuthology.orchestra.run.trial198.stdout: Running scriptlet: libradosstriper1-2:20.3.0-5439.ge371433f.el9.x86_6 11/36 2026-02-23T10:50:41.516 INFO:teuthology.orchestra.run.trial198.stdout: Installing : thrift-0.15.0-4.el9.x86_64 12/36 2026-02-23T10:50:41.546 INFO:teuthology.orchestra.run.trial198.stdout: Installing : re2-1:20211101-20.el9.x86_64 13/36 2026-02-23T10:50:41.551 INFO:teuthology.orchestra.run.trial198.stdout: Installing : libarrow-9.0.0-15.el9.x86_64 14/36 2026-02-23T10:50:41.558 INFO:teuthology.orchestra.run.trial198.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 15/36 2026-02-23T10:50:41.560 INFO:teuthology.orchestra.run.trial198.stdout: Installing : gperftools-libs-2.9.1-3.el9.x86_64 16/36 2026-02-23T10:50:41.565 INFO:teuthology.orchestra.run.trial198.stdout: Installing : libarrow-doc-9.0.0-15.el9.noarch 17/36 2026-02-23T10:50:41.680 INFO:teuthology.orchestra.run.trial198.stdout: Installing : libnbd-1.20.3-4.el9.x86_64 18/36 2026-02-23T10:50:41.681 INFO:teuthology.orchestra.run.trial198.stdout: Upgrading : librbd1-2:20.3.0-5439.ge371433f.el9.x86_64 19/36 2026-02-23T10:50:41.707 INFO:teuthology.orchestra.run.trial198.stdout: Running scriptlet: librbd1-2:20.3.0-5439.ge371433f.el9.x86_64 19/36 2026-02-23T10:50:41.709 INFO:teuthology.orchestra.run.trial198.stdout: Installing : python3-rbd-2:20.3.0-5439.ge371433f.el9.x86_64 20/36 2026-02-23T10:50:41.715 INFO:teuthology.orchestra.run.trial198.stdout: Installing : boost-program-options-1.75.0-13.el9.x86_64 21/36 2026-02-23T10:50:41.963 INFO:teuthology.orchestra.run.trial198.stdout: Installing : parquet-libs-9.0.0-15.el9.x86_64 22/36 2026-02-23T10:50:41.964 INFO:teuthology.orchestra.run.trial198.stdout: Installing : librgw2-2:20.3.0-5439.ge371433f.el9.x86_64 23/36 2026-02-23T10:50:41.982 INFO:teuthology.orchestra.run.trial198.stdout: Running scriptlet: librgw2-2:20.3.0-5439.ge371433f.el9.x86_64 23/36 2026-02-23T10:50:41.986 INFO:teuthology.orchestra.run.trial198.stdout: Installing : python3-rgw-2:20.3.0-5439.ge371433f.el9.x86_64 24/36 2026-02-23T10:50:41.990 INFO:teuthology.orchestra.run.trial198.stdout: Installing : pciutils-3.7.0-7.el9.x86_64 25/36 2026-02-23T10:50:41.993 INFO:teuthology.orchestra.run.trial198.stdout: Installing : qatlib-25.08.0-2.el9.x86_64 26/36 2026-02-23T10:50:41.994 INFO:teuthology.orchestra.run.trial198.stdout: Installing : qatlib-service-25.08.0-2.el9.x86_64 27/36 2026-02-23T10:50:42.012 INFO:teuthology.orchestra.run.trial198.stdout: Running scriptlet: qatlib-service-25.08.0-2.el9.x86_64 27/36 2026-02-23T10:50:42.016 INFO:teuthology.orchestra.run.trial198.stdout: Installing : qatzip-libs-1.3.1-1.el9.x86_64 28/36 2026-02-23T10:50:42.050 INFO:teuthology.orchestra.run.trial198.stdout: Installing : mailcap-2.1.49-5.el9.noarch 29/36 2026-02-23T10:50:42.052 INFO:teuthology.orchestra.run.trial198.stdout: Installing : python3-ceph-common-2:20.3.0-5439.ge371433f.el9.x8 30/36 2026-02-23T10:50:42.990 INFO:teuthology.orchestra.run.trial198.stdout: Running scriptlet: ceph-common-2:20.3.0-5439.ge371433f.el9.x86_64 31/36 2026-02-23T10:50:42.992 INFO:teuthology.orchestra.run.trial198.stdout: Installing : ceph-common-2:20.3.0-5439.ge371433f.el9.x86_64 31/36 2026-02-23T10:50:43.255 INFO:teuthology.orchestra.run.trial198.stdout: Running scriptlet: ceph-common-2:20.3.0-5439.ge371433f.el9.x86_64 31/36 2026-02-23T10:50:43.259 INFO:teuthology.orchestra.run.trial198.stdout: Installing : ceph-base-2:20.3.0-5439.ge371433f.el9.x86_64 32/36 2026-02-23T10:50:43.299 INFO:teuthology.orchestra.run.trial198.stdout: Running scriptlet: ceph-base-2:20.3.0-5439.ge371433f.el9.x86_64 32/36 2026-02-23T10:50:43.300 INFO:teuthology.orchestra.run.trial198.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2026-02-23T10:50:43.300 INFO:teuthology.orchestra.run.trial198.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2026-02-23T10:50:43.300 INFO:teuthology.orchestra.run.trial198.stdout: 2026-02-23T10:50:43.303 INFO:teuthology.orchestra.run.trial198.stdout: Installing : ceph-selinux-2:20.3.0-5439.ge371433f.el9.x86_64 33/36 2026-02-23T10:50:47.812 INFO:teuthology.orchestra.run.trial198.stdout: Running scriptlet: ceph-selinux-2:20.3.0-5439.ge371433f.el9.x86_64 33/36 2026-02-23T10:50:47.812 INFO:teuthology.orchestra.run.trial198.stdout:skipping the directory /sys 2026-02-23T10:50:47.813 INFO:teuthology.orchestra.run.trial198.stdout:skipping the directory /proc 2026-02-23T10:50:47.813 INFO:teuthology.orchestra.run.trial198.stdout:skipping the directory /mnt 2026-02-23T10:50:47.813 INFO:teuthology.orchestra.run.trial198.stdout:skipping the directory /var/tmp 2026-02-23T10:50:47.813 INFO:teuthology.orchestra.run.trial198.stdout:skipping the directory /home 2026-02-23T10:50:47.813 INFO:teuthology.orchestra.run.trial198.stdout:skipping the directory /root 2026-02-23T10:50:47.813 INFO:teuthology.orchestra.run.trial198.stdout:skipping the directory /tmp 2026-02-23T10:50:47.813 INFO:teuthology.orchestra.run.trial198.stdout: 2026-02-23T10:50:48.305 INFO:teuthology.orchestra.run.trial198.stdout: Installing : ceph-radosgw-2:20.3.0-5439.ge371433f.el9.x86_64 34/36 2026-02-23T10:50:48.328 INFO:teuthology.orchestra.run.trial198.stdout: Running scriptlet: ceph-radosgw-2:20.3.0-5439.ge371433f.el9.x86_64 34/36 2026-02-23T10:50:48.328 INFO:teuthology.orchestra.run.trial198.stdout:Glob pattern passed to enable, but globs are not supported for this. 2026-02-23T10:50:48.328 INFO:teuthology.orchestra.run.trial198.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2026-02-23T10:50:48.328 INFO:teuthology.orchestra.run.trial198.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2026-02-23T10:50:48.328 INFO:teuthology.orchestra.run.trial198.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2026-02-23T10:50:48.328 INFO:teuthology.orchestra.run.trial198.stdout: 2026-02-23T10:50:48.330 INFO:teuthology.orchestra.run.trial198.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 35/36 2026-02-23T10:50:48.342 INFO:teuthology.orchestra.run.trial198.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 35/36 2026-02-23T10:50:48.342 INFO:teuthology.orchestra.run.trial198.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 36/36 2026-02-23T10:50:48.398 INFO:teuthology.orchestra.run.trial198.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 36/36 2026-02-23T10:50:48.398 INFO:teuthology.orchestra.run.trial198.stdout:Creating group 'qat' with GID 992. 2026-02-23T10:50:48.398 INFO:teuthology.orchestra.run.trial198.stdout: 2026-02-23T10:50:48.700 INFO:teuthology.orchestra.run.trial198.stdout: Verifying : ceph-base-2:20.3.0-5439.ge371433f.el9.x86_64 1/36 2026-02-23T10:50:48.700 INFO:teuthology.orchestra.run.trial198.stdout: Verifying : ceph-common-2:20.3.0-5439.ge371433f.el9.x86_64 2/36 2026-02-23T10:50:48.700 INFO:teuthology.orchestra.run.trial198.stdout: Verifying : ceph-radosgw-2:20.3.0-5439.ge371433f.el9.x86_64 3/36 2026-02-23T10:50:48.700 INFO:teuthology.orchestra.run.trial198.stdout: Verifying : ceph-selinux-2:20.3.0-5439.ge371433f.el9.x86_64 4/36 2026-02-23T10:50:48.700 INFO:teuthology.orchestra.run.trial198.stdout: Verifying : libcephfs2-2:20.3.0-5439.ge371433f.el9.x86_64 5/36 2026-02-23T10:50:48.700 INFO:teuthology.orchestra.run.trial198.stdout: Verifying : libradosstriper1-2:20.3.0-5439.ge371433f.el9.x86_6 6/36 2026-02-23T10:50:48.700 INFO:teuthology.orchestra.run.trial198.stdout: Verifying : librgw2-2:20.3.0-5439.ge371433f.el9.x86_64 7/36 2026-02-23T10:50:48.700 INFO:teuthology.orchestra.run.trial198.stdout: Verifying : python3-ceph-argparse-2:20.3.0-5439.ge371433f.el9. 8/36 2026-02-23T10:50:48.700 INFO:teuthology.orchestra.run.trial198.stdout: Verifying : python3-ceph-common-2:20.3.0-5439.ge371433f.el9.x8 9/36 2026-02-23T10:50:48.700 INFO:teuthology.orchestra.run.trial198.stdout: Verifying : python3-cephfs-2:20.3.0-5439.ge371433f.el9.x86_64 10/36 2026-02-23T10:50:48.700 INFO:teuthology.orchestra.run.trial198.stdout: Verifying : python3-rados-2:20.3.0-5439.ge371433f.el9.x86_64 11/36 2026-02-23T10:50:48.700 INFO:teuthology.orchestra.run.trial198.stdout: Verifying : python3-rbd-2:20.3.0-5439.ge371433f.el9.x86_64 12/36 2026-02-23T10:50:48.700 INFO:teuthology.orchestra.run.trial198.stdout: Verifying : python3-rgw-2:20.3.0-5439.ge371433f.el9.x86_64 13/36 2026-02-23T10:50:48.700 INFO:teuthology.orchestra.run.trial198.stdout: Verifying : mailcap-2.1.49-5.el9.noarch 14/36 2026-02-23T10:50:48.700 INFO:teuthology.orchestra.run.trial198.stdout: Verifying : pciutils-3.7.0-7.el9.x86_64 15/36 2026-02-23T10:50:48.700 INFO:teuthology.orchestra.run.trial198.stdout: Verifying : boost-program-options-1.75.0-13.el9.x86_64 16/36 2026-02-23T10:50:48.700 INFO:teuthology.orchestra.run.trial198.stdout: Verifying : libnbd-1.20.3-4.el9.x86_64 17/36 2026-02-23T10:50:48.700 INFO:teuthology.orchestra.run.trial198.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 18/36 2026-02-23T10:50:48.700 INFO:teuthology.orchestra.run.trial198.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 19/36 2026-02-23T10:50:48.700 INFO:teuthology.orchestra.run.trial198.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 20/36 2026-02-23T10:50:48.700 INFO:teuthology.orchestra.run.trial198.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 21/36 2026-02-23T10:50:48.700 INFO:teuthology.orchestra.run.trial198.stdout: Verifying : qatlib-25.08.0-2.el9.x86_64 22/36 2026-02-23T10:50:48.701 INFO:teuthology.orchestra.run.trial198.stdout: Verifying : qatlib-service-25.08.0-2.el9.x86_64 23/36 2026-02-23T10:50:48.701 INFO:teuthology.orchestra.run.trial198.stdout: Verifying : qatzip-libs-1.3.1-1.el9.x86_64 24/36 2026-02-23T10:50:48.701 INFO:teuthology.orchestra.run.trial198.stdout: Verifying : gperftools-libs-2.9.1-3.el9.x86_64 25/36 2026-02-23T10:50:48.701 INFO:teuthology.orchestra.run.trial198.stdout: Verifying : libarrow-9.0.0-15.el9.x86_64 26/36 2026-02-23T10:50:48.701 INFO:teuthology.orchestra.run.trial198.stdout: Verifying : libarrow-doc-9.0.0-15.el9.noarch 27/36 2026-02-23T10:50:48.701 INFO:teuthology.orchestra.run.trial198.stdout: Verifying : liboath-2.6.12-1.el9.x86_64 28/36 2026-02-23T10:50:48.701 INFO:teuthology.orchestra.run.trial198.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 29/36 2026-02-23T10:50:48.701 INFO:teuthology.orchestra.run.trial198.stdout: Verifying : parquet-libs-9.0.0-15.el9.x86_64 30/36 2026-02-23T10:50:48.701 INFO:teuthology.orchestra.run.trial198.stdout: Verifying : re2-1:20211101-20.el9.x86_64 31/36 2026-02-23T10:50:48.701 INFO:teuthology.orchestra.run.trial198.stdout: Verifying : thrift-0.15.0-4.el9.x86_64 32/36 2026-02-23T10:50:48.701 INFO:teuthology.orchestra.run.trial198.stdout: Verifying : librados2-2:20.3.0-5439.ge371433f.el9.x86_64 33/36 2026-02-23T10:50:48.701 INFO:teuthology.orchestra.run.trial198.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 34/36 2026-02-23T10:50:48.701 INFO:teuthology.orchestra.run.trial198.stdout: Verifying : librbd1-2:20.3.0-5439.ge371433f.el9.x86_64 35/36 2026-02-23T10:50:48.764 INFO:teuthology.orchestra.run.trial198.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 36/36 2026-02-23T10:50:48.764 INFO:teuthology.orchestra.run.trial198.stdout: 2026-02-23T10:50:48.764 INFO:teuthology.orchestra.run.trial198.stdout:Upgraded: 2026-02-23T10:50:48.764 INFO:teuthology.orchestra.run.trial198.stdout: librados2-2:20.3.0-5439.ge371433f.el9.x86_64 2026-02-23T10:50:48.764 INFO:teuthology.orchestra.run.trial198.stdout: librbd1-2:20.3.0-5439.ge371433f.el9.x86_64 2026-02-23T10:50:48.764 INFO:teuthology.orchestra.run.trial198.stdout:Installed: 2026-02-23T10:50:48.764 INFO:teuthology.orchestra.run.trial198.stdout: boost-program-options-1.75.0-13.el9.x86_64 2026-02-23T10:50:48.765 INFO:teuthology.orchestra.run.trial198.stdout: ceph-base-2:20.3.0-5439.ge371433f.el9.x86_64 2026-02-23T10:50:48.765 INFO:teuthology.orchestra.run.trial198.stdout: ceph-common-2:20.3.0-5439.ge371433f.el9.x86_64 2026-02-23T10:50:48.765 INFO:teuthology.orchestra.run.trial198.stdout: ceph-radosgw-2:20.3.0-5439.ge371433f.el9.x86_64 2026-02-23T10:50:48.765 INFO:teuthology.orchestra.run.trial198.stdout: ceph-selinux-2:20.3.0-5439.ge371433f.el9.x86_64 2026-02-23T10:50:48.765 INFO:teuthology.orchestra.run.trial198.stdout: gperftools-libs-2.9.1-3.el9.x86_64 2026-02-23T10:50:48.765 INFO:teuthology.orchestra.run.trial198.stdout: libarrow-9.0.0-15.el9.x86_64 2026-02-23T10:50:48.765 INFO:teuthology.orchestra.run.trial198.stdout: libarrow-doc-9.0.0-15.el9.noarch 2026-02-23T10:50:48.765 INFO:teuthology.orchestra.run.trial198.stdout: libcephfs2-2:20.3.0-5439.ge371433f.el9.x86_64 2026-02-23T10:50:48.765 INFO:teuthology.orchestra.run.trial198.stdout: libnbd-1.20.3-4.el9.x86_64 2026-02-23T10:50:48.765 INFO:teuthology.orchestra.run.trial198.stdout: liboath-2.6.12-1.el9.x86_64 2026-02-23T10:50:48.766 INFO:teuthology.orchestra.run.trial198.stdout: libpmemobj-1.12.1-1.el9.x86_64 2026-02-23T10:50:48.766 INFO:teuthology.orchestra.run.trial198.stdout: librabbitmq-0.11.0-7.el9.x86_64 2026-02-23T10:50:48.766 INFO:teuthology.orchestra.run.trial198.stdout: libradosstriper1-2:20.3.0-5439.ge371433f.el9.x86_64 2026-02-23T10:50:48.766 INFO:teuthology.orchestra.run.trial198.stdout: librdkafka-1.6.1-102.el9.x86_64 2026-02-23T10:50:48.766 INFO:teuthology.orchestra.run.trial198.stdout: librgw2-2:20.3.0-5439.ge371433f.el9.x86_64 2026-02-23T10:50:48.766 INFO:teuthology.orchestra.run.trial198.stdout: libunwind-1.6.2-1.el9.x86_64 2026-02-23T10:50:48.766 INFO:teuthology.orchestra.run.trial198.stdout: lttng-ust-2.12.0-6.el9.x86_64 2026-02-23T10:50:48.766 INFO:teuthology.orchestra.run.trial198.stdout: mailcap-2.1.49-5.el9.noarch 2026-02-23T10:50:48.766 INFO:teuthology.orchestra.run.trial198.stdout: parquet-libs-9.0.0-15.el9.x86_64 2026-02-23T10:50:48.766 INFO:teuthology.orchestra.run.trial198.stdout: pciutils-3.7.0-7.el9.x86_64 2026-02-23T10:50:48.766 INFO:teuthology.orchestra.run.trial198.stdout: python3-ceph-argparse-2:20.3.0-5439.ge371433f.el9.x86_64 2026-02-23T10:50:48.767 INFO:teuthology.orchestra.run.trial198.stdout: python3-ceph-common-2:20.3.0-5439.ge371433f.el9.x86_64 2026-02-23T10:50:48.767 INFO:teuthology.orchestra.run.trial198.stdout: python3-cephfs-2:20.3.0-5439.ge371433f.el9.x86_64 2026-02-23T10:50:48.767 INFO:teuthology.orchestra.run.trial198.stdout: python3-rados-2:20.3.0-5439.ge371433f.el9.x86_64 2026-02-23T10:50:48.767 INFO:teuthology.orchestra.run.trial198.stdout: python3-rbd-2:20.3.0-5439.ge371433f.el9.x86_64 2026-02-23T10:50:48.767 INFO:teuthology.orchestra.run.trial198.stdout: python3-rgw-2:20.3.0-5439.ge371433f.el9.x86_64 2026-02-23T10:50:48.767 INFO:teuthology.orchestra.run.trial198.stdout: qatlib-25.08.0-2.el9.x86_64 2026-02-23T10:50:48.767 INFO:teuthology.orchestra.run.trial198.stdout: qatlib-service-25.08.0-2.el9.x86_64 2026-02-23T10:50:48.767 INFO:teuthology.orchestra.run.trial198.stdout: qatzip-libs-1.3.1-1.el9.x86_64 2026-02-23T10:50:48.767 INFO:teuthology.orchestra.run.trial198.stdout: re2-1:20211101-20.el9.x86_64 2026-02-23T10:50:48.767 INFO:teuthology.orchestra.run.trial198.stdout: thrift-0.15.0-4.el9.x86_64 2026-02-23T10:50:48.768 INFO:teuthology.orchestra.run.trial198.stdout: 2026-02-23T10:50:48.768 INFO:teuthology.orchestra.run.trial198.stdout:Complete! 2026-02-23T10:50:48.805 DEBUG:teuthology.orchestra.run.trial198:> sudo yum -y install ceph-test 2026-02-23T10:50:49.115 INFO:teuthology.orchestra.run.trial198.stdout:Last metadata expiration check: 0:00:17 ago on Mon 23 Feb 2026 10:50:32 AM UTC. 2026-02-23T10:50:49.198 INFO:teuthology.orchestra.run.trial198.stdout:Dependencies resolved. 2026-02-23T10:50:49.199 INFO:teuthology.orchestra.run.trial198.stdout:================================================================================ 2026-02-23T10:50:49.199 INFO:teuthology.orchestra.run.trial198.stdout: Package Arch Version Repository Size 2026-02-23T10:50:49.199 INFO:teuthology.orchestra.run.trial198.stdout:================================================================================ 2026-02-23T10:50:49.199 INFO:teuthology.orchestra.run.trial198.stdout:Installing: 2026-02-23T10:50:49.199 INFO:teuthology.orchestra.run.trial198.stdout: ceph-test x86_64 2:20.3.0-5439.ge371433f.el9 ceph 126 M 2026-02-23T10:50:49.199 INFO:teuthology.orchestra.run.trial198.stdout:Installing dependencies: 2026-02-23T10:50:49.199 INFO:teuthology.orchestra.run.trial198.stdout: hwloc-libs x86_64 2.4.1-6.el9 baseos 2.1 M 2026-02-23T10:50:49.199 INFO:teuthology.orchestra.run.trial198.stdout: libasan8 x86_64 15.2.1-7.el9 appstream 525 k 2026-02-23T10:50:49.199 INFO:teuthology.orchestra.run.trial198.stdout: libatomic x86_64 11.5.0-14.el9 baseos 23 k 2026-02-23T10:50:49.199 INFO:teuthology.orchestra.run.trial198.stdout: libcephsqlite x86_64 2:20.3.0-5439.ge371433f.el9 ceph 153 k 2026-02-23T10:50:49.199 INFO:teuthology.orchestra.run.trial198.stdout: libubsan x86_64 11.5.0-14.el9 appstream 176 k 2026-02-23T10:50:49.199 INFO:teuthology.orchestra.run.trial198.stdout: libxslt x86_64 1.1.34-12.el9 appstream 233 k 2026-02-23T10:50:49.199 INFO:teuthology.orchestra.run.trial198.stdout: lksctp-tools x86_64 1.0.19-2.el9 baseos 94 k 2026-02-23T10:50:49.199 INFO:teuthology.orchestra.run.trial198.stdout: protobuf x86_64 3.14.0-17.el9 appstream 1.0 M 2026-02-23T10:50:49.199 INFO:teuthology.orchestra.run.trial198.stdout: socat x86_64 1.7.4.1-8.el9 appstream 303 k 2026-02-23T10:50:49.199 INFO:teuthology.orchestra.run.trial198.stdout: xmlstarlet x86_64 1.6.1-20.el9 appstream 64 k 2026-02-23T10:50:49.199 INFO:teuthology.orchestra.run.trial198.stdout: yaml-cpp x86_64 0.6.3-5.el9 epel 122 k 2026-02-23T10:50:49.200 INFO:teuthology.orchestra.run.trial198.stdout: 2026-02-23T10:50:49.200 INFO:teuthology.orchestra.run.trial198.stdout:Transaction Summary 2026-02-23T10:50:49.200 INFO:teuthology.orchestra.run.trial198.stdout:================================================================================ 2026-02-23T10:50:49.200 INFO:teuthology.orchestra.run.trial198.stdout:Install 12 Packages 2026-02-23T10:50:49.200 INFO:teuthology.orchestra.run.trial198.stdout: 2026-02-23T10:50:49.200 INFO:teuthology.orchestra.run.trial198.stdout:Total download size: 131 M 2026-02-23T10:50:49.200 INFO:teuthology.orchestra.run.trial198.stdout:Installed size: 977 M 2026-02-23T10:50:49.200 INFO:teuthology.orchestra.run.trial198.stdout:Downloading Packages: 2026-02-23T10:50:50.098 INFO:teuthology.orchestra.run.trial198.stdout:(1/12): libcephsqlite-20.3.0-5439.ge371433f.el9 420 kB/s | 153 kB 00:00 2026-02-23T10:50:50.099 INFO:teuthology.orchestra.run.trial198.stdout:[MIRROR] hwloc-libs-2.4.1-6.el9.x86_64.rpm: Status code: 503 for https://ftp2.osuosl.org/pub/centos-stream/9-stream/BaseOS/x86_64/os/Packages/hwloc-libs-2.4.1-6.el9.x86_64.rpm (IP: 140.211.166.134) 2026-02-23T10:50:50.434 INFO:teuthology.orchestra.run.trial198.stdout:[MIRROR] libatomic-11.5.0-14.el9.x86_64.rpm: Status code: 503 for https://ftp2.osuosl.org/pub/centos-stream/9-stream/BaseOS/x86_64/os/Packages/libatomic-11.5.0-14.el9.x86_64.rpm (IP: 140.211.166.134) 2026-02-23T10:50:50.503 INFO:teuthology.orchestra.run.trial198.stdout:[MIRROR] hwloc-libs-2.4.1-6.el9.x86_64.rpm: Status code: 503 for https://ftp2.osuosl.org/pub/centos-stream/9-stream/BaseOS/x86_64/os/Packages/hwloc-libs-2.4.1-6.el9.x86_64.rpm (IP: 140.211.166.134) 2026-02-23T10:50:50.538 INFO:teuthology.orchestra.run.trial198.stdout:(2/12): hwloc-libs-2.4.1-6.el9.x86_64.rpm 2.6 MB/s | 2.1 MB 00:00 2026-02-23T10:50:50.774 INFO:teuthology.orchestra.run.trial198.stdout:[MIRROR] libatomic-11.5.0-14.el9.x86_64.rpm: Status code: 503 for https://ftp2.osuosl.org/pub/centos-stream/9-stream/BaseOS/x86_64/os/Packages/libatomic-11.5.0-14.el9.x86_64.rpm (IP: 140.211.166.134) 2026-02-23T10:50:50.779 INFO:teuthology.orchestra.run.trial198.stdout:(3/12): libatomic-11.5.0-14.el9.x86_64.rpm 33 kB/s | 23 kB 00:00 2026-02-23T10:50:50.876 INFO:teuthology.orchestra.run.trial198.stdout:[MIRROR] lksctp-tools-1.0.19-2.el9.x86_64.rpm: Status code: 503 for https://ftp2.osuosl.org/pub/centos-stream/9-stream/BaseOS/x86_64/os/Packages/lksctp-tools-1.0.19-2.el9.x86_64.rpm (IP: 140.211.166.134) 2026-02-23T10:50:51.446 INFO:teuthology.orchestra.run.trial198.stdout:(4/12): lksctp-tools-1.0.19-2.el9.x86_64.rpm 104 kB/s | 94 kB 00:00 2026-02-23T10:50:51.699 INFO:teuthology.orchestra.run.trial198.stdout:(5/12): libubsan-11.5.0-14.el9.x86_64.rpm 693 kB/s | 176 kB 00:00 2026-02-23T10:50:51.816 INFO:teuthology.orchestra.run.trial198.stdout:(6/12): libasan8-15.2.1-7.el9.x86_64.rpm 506 kB/s | 525 kB 00:01 2026-02-23T10:50:52.032 INFO:teuthology.orchestra.run.trial198.stdout:(7/12): libxslt-1.1.34-12.el9.x86_64.rpm 701 kB/s | 233 kB 00:00 2026-02-23T10:50:52.073 INFO:teuthology.orchestra.run.trial198.stdout:(8/12): protobuf-3.14.0-17.el9.x86_64.rpm 3.9 MB/s | 1.0 MB 00:00 2026-02-23T10:50:52.171 INFO:teuthology.orchestra.run.trial198.stdout:(9/12): xmlstarlet-1.6.1-20.el9.x86_64.rpm 648 kB/s | 64 kB 00:00 2026-02-23T10:50:52.442 INFO:teuthology.orchestra.run.trial198.stdout:(10/12): socat-1.7.4.1-8.el9.x86_64.rpm 739 kB/s | 303 kB 00:00 2026-02-23T10:50:52.517 INFO:teuthology.orchestra.run.trial198.stdout:(11/12): yaml-cpp-0.6.3-5.el9.x86_64.rpm 352 kB/s | 122 kB 00:00 2026-02-23T10:51:00.777 INFO:teuthology.orchestra.run.trial198.stdout:(12/12): ceph-test-20.3.0-5439.ge371433f.el9.x8 11 MB/s | 126 MB 00:11 2026-02-23T10:51:00.780 INFO:teuthology.orchestra.run.trial198.stdout:-------------------------------------------------------------------------------- 2026-02-23T10:51:00.780 INFO:teuthology.orchestra.run.trial198.stdout:Total 11 MB/s | 131 MB 00:11 2026-02-23T10:51:00.862 INFO:teuthology.orchestra.run.trial198.stdout:Running transaction check 2026-02-23T10:51:00.873 INFO:teuthology.orchestra.run.trial198.stdout:Transaction check succeeded. 2026-02-23T10:51:00.873 INFO:teuthology.orchestra.run.trial198.stdout:Running transaction test 2026-02-23T10:51:01.300 INFO:teuthology.orchestra.run.trial198.stdout:Transaction test succeeded. 2026-02-23T10:51:01.301 INFO:teuthology.orchestra.run.trial198.stdout:Running transaction 2026-02-23T10:51:01.756 INFO:teuthology.orchestra.run.trial198.stdout: Preparing : 1/1 2026-02-23T10:51:01.764 INFO:teuthology.orchestra.run.trial198.stdout: Installing : yaml-cpp-0.6.3-5.el9.x86_64 1/12 2026-02-23T10:51:01.773 INFO:teuthology.orchestra.run.trial198.stdout: Installing : socat-1.7.4.1-8.el9.x86_64 2/12 2026-02-23T10:51:01.778 INFO:teuthology.orchestra.run.trial198.stdout: Installing : protobuf-3.14.0-17.el9.x86_64 3/12 2026-02-23T10:51:01.782 INFO:teuthology.orchestra.run.trial198.stdout: Installing : libxslt-1.1.34-12.el9.x86_64 4/12 2026-02-23T10:51:01.784 INFO:teuthology.orchestra.run.trial198.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 5/12 2026-02-23T10:51:01.789 INFO:teuthology.orchestra.run.trial198.stdout: Installing : libubsan-11.5.0-14.el9.x86_64 6/12 2026-02-23T10:51:01.789 INFO:teuthology.orchestra.run.trial198.stdout: Installing : libasan8-15.2.1-7.el9.x86_64 7/12 2026-02-23T10:51:01.807 INFO:teuthology.orchestra.run.trial198.stdout: Running scriptlet: libasan8-15.2.1-7.el9.x86_64 7/12 2026-02-23T10:51:01.809 INFO:teuthology.orchestra.run.trial198.stdout: Installing : lksctp-tools-1.0.19-2.el9.x86_64 8/12 2026-02-23T10:51:01.820 INFO:teuthology.orchestra.run.trial198.stdout: Installing : libatomic-11.5.0-14.el9.x86_64 9/12 2026-02-23T10:51:01.829 INFO:teuthology.orchestra.run.trial198.stdout: Installing : hwloc-libs-2.4.1-6.el9.x86_64 10/12 2026-02-23T10:51:01.829 INFO:teuthology.orchestra.run.trial198.stdout: Installing : libcephsqlite-2:20.3.0-5439.ge371433f.el9.x86_64 11/12 2026-02-23T10:51:08.737 INFO:teuthology.orchestra.run.trial198.stdout: Running scriptlet: libcephsqlite-2:20.3.0-5439.ge371433f.el9.x86_64 11/12 2026-02-23T10:51:08.755 INFO:teuthology.orchestra.run.trial198.stdout: Installing : ceph-test-2:20.3.0-5439.ge371433f.el9.x86_64 12/12 2026-02-23T10:51:09.533 INFO:teuthology.orchestra.run.trial198.stdout: Running scriptlet: ceph-test-2:20.3.0-5439.ge371433f.el9.x86_64 12/12 2026-02-23T10:51:09.534 INFO:teuthology.orchestra.run.trial198.stdout: Verifying : ceph-test-2:20.3.0-5439.ge371433f.el9.x86_64 1/12 2026-02-23T10:51:09.534 INFO:teuthology.orchestra.run.trial198.stdout: Verifying : libcephsqlite-2:20.3.0-5439.ge371433f.el9.x86_64 2/12 2026-02-23T10:51:09.534 INFO:teuthology.orchestra.run.trial198.stdout: Verifying : hwloc-libs-2.4.1-6.el9.x86_64 3/12 2026-02-23T10:51:09.534 INFO:teuthology.orchestra.run.trial198.stdout: Verifying : libatomic-11.5.0-14.el9.x86_64 4/12 2026-02-23T10:51:09.534 INFO:teuthology.orchestra.run.trial198.stdout: Verifying : lksctp-tools-1.0.19-2.el9.x86_64 5/12 2026-02-23T10:51:09.534 INFO:teuthology.orchestra.run.trial198.stdout: Verifying : libasan8-15.2.1-7.el9.x86_64 6/12 2026-02-23T10:51:09.534 INFO:teuthology.orchestra.run.trial198.stdout: Verifying : libubsan-11.5.0-14.el9.x86_64 7/12 2026-02-23T10:51:09.534 INFO:teuthology.orchestra.run.trial198.stdout: Verifying : libxslt-1.1.34-12.el9.x86_64 8/12 2026-02-23T10:51:09.534 INFO:teuthology.orchestra.run.trial198.stdout: Verifying : protobuf-3.14.0-17.el9.x86_64 9/12 2026-02-23T10:51:09.534 INFO:teuthology.orchestra.run.trial198.stdout: Verifying : socat-1.7.4.1-8.el9.x86_64 10/12 2026-02-23T10:51:09.535 INFO:teuthology.orchestra.run.trial198.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 11/12 2026-02-23T10:51:09.588 INFO:teuthology.orchestra.run.trial198.stdout: Verifying : yaml-cpp-0.6.3-5.el9.x86_64 12/12 2026-02-23T10:51:09.588 INFO:teuthology.orchestra.run.trial198.stdout: 2026-02-23T10:51:09.588 INFO:teuthology.orchestra.run.trial198.stdout:Installed: 2026-02-23T10:51:09.588 INFO:teuthology.orchestra.run.trial198.stdout: ceph-test-2:20.3.0-5439.ge371433f.el9.x86_64 2026-02-23T10:51:09.589 INFO:teuthology.orchestra.run.trial198.stdout: hwloc-libs-2.4.1-6.el9.x86_64 2026-02-23T10:51:09.589 INFO:teuthology.orchestra.run.trial198.stdout: libasan8-15.2.1-7.el9.x86_64 2026-02-23T10:51:09.589 INFO:teuthology.orchestra.run.trial198.stdout: libatomic-11.5.0-14.el9.x86_64 2026-02-23T10:51:09.589 INFO:teuthology.orchestra.run.trial198.stdout: libcephsqlite-2:20.3.0-5439.ge371433f.el9.x86_64 2026-02-23T10:51:09.589 INFO:teuthology.orchestra.run.trial198.stdout: libubsan-11.5.0-14.el9.x86_64 2026-02-23T10:51:09.589 INFO:teuthology.orchestra.run.trial198.stdout: libxslt-1.1.34-12.el9.x86_64 2026-02-23T10:51:09.589 INFO:teuthology.orchestra.run.trial198.stdout: lksctp-tools-1.0.19-2.el9.x86_64 2026-02-23T10:51:09.589 INFO:teuthology.orchestra.run.trial198.stdout: protobuf-3.14.0-17.el9.x86_64 2026-02-23T10:51:09.589 INFO:teuthology.orchestra.run.trial198.stdout: socat-1.7.4.1-8.el9.x86_64 2026-02-23T10:51:09.589 INFO:teuthology.orchestra.run.trial198.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2026-02-23T10:51:09.589 INFO:teuthology.orchestra.run.trial198.stdout: yaml-cpp-0.6.3-5.el9.x86_64 2026-02-23T10:51:09.590 INFO:teuthology.orchestra.run.trial198.stdout: 2026-02-23T10:51:09.590 INFO:teuthology.orchestra.run.trial198.stdout:Complete! 2026-02-23T10:51:09.632 DEBUG:teuthology.orchestra.run.trial198:> sudo yum -y install ceph 2026-02-23T10:51:09.936 INFO:teuthology.orchestra.run.trial198.stdout:Last metadata expiration check: 0:00:37 ago on Mon 23 Feb 2026 10:50:32 AM UTC. 2026-02-23T10:51:10.020 INFO:teuthology.orchestra.run.trial198.stdout:Dependencies resolved. 2026-02-23T10:51:10.021 INFO:teuthology.orchestra.run.trial198.stdout:================================================================================ 2026-02-23T10:51:10.021 INFO:teuthology.orchestra.run.trial198.stdout: Package Arch Version Repository Size 2026-02-23T10:51:10.021 INFO:teuthology.orchestra.run.trial198.stdout:================================================================================ 2026-02-23T10:51:10.021 INFO:teuthology.orchestra.run.trial198.stdout:Installing: 2026-02-23T10:51:10.021 INFO:teuthology.orchestra.run.trial198.stdout: ceph x86_64 2:20.3.0-5439.ge371433f.el9 ceph 6.5 k 2026-02-23T10:51:10.021 INFO:teuthology.orchestra.run.trial198.stdout:Installing dependencies: 2026-02-23T10:51:10.021 INFO:teuthology.orchestra.run.trial198.stdout: ceph-mds x86_64 2:20.3.0-5439.ge371433f.el9 ceph 2.0 M 2026-02-23T10:51:10.022 INFO:teuthology.orchestra.run.trial198.stdout: ceph-mgr x86_64 2:20.3.0-5439.ge371433f.el9 ceph 937 k 2026-02-23T10:51:10.022 INFO:teuthology.orchestra.run.trial198.stdout: ceph-mgr-modules-core noarch 2:20.3.0-5439.ge371433f.el9 ceph-noarch 298 k 2026-02-23T10:51:10.022 INFO:teuthology.orchestra.run.trial198.stdout: ceph-mon x86_64 2:20.3.0-5439.ge371433f.el9 ceph 4.9 M 2026-02-23T10:51:10.022 INFO:teuthology.orchestra.run.trial198.stdout: ceph-osd x86_64 2:20.3.0-5439.ge371433f.el9 ceph 164 k 2026-02-23T10:51:10.022 INFO:teuthology.orchestra.run.trial198.stdout: ceph-osd-classic x86_64 2:20.3.0-5439.ge371433f.el9 ceph 15 M 2026-02-23T10:51:10.022 INFO:teuthology.orchestra.run.trial198.stdout: ceph-osd-crimson x86_64 2:20.3.0-5439.ge371433f.el9 ceph 87 M 2026-02-23T10:51:10.022 INFO:teuthology.orchestra.run.trial198.stdout: ledmon-libs x86_64 1.1.0-3.el9 baseos 40 k 2026-02-23T10:51:10.022 INFO:teuthology.orchestra.run.trial198.stdout: libconfig x86_64 1.7.2-9.el9 baseos 72 k 2026-02-23T10:51:10.022 INFO:teuthology.orchestra.run.trial198.stdout: libstoragemgmt x86_64 1.10.1-1.el9 appstream 246 k 2026-02-23T10:51:10.022 INFO:teuthology.orchestra.run.trial198.stdout: lua x86_64 5.4.4-4.el9 appstream 188 k 2026-02-23T10:51:10.022 INFO:teuthology.orchestra.run.trial198.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2026-02-23T10:51:10.022 INFO:teuthology.orchestra.run.trial198.stdout: luarocks noarch 3.9.2-5.el9 epel 151 k 2026-02-23T10:51:10.022 INFO:teuthology.orchestra.run.trial198.stdout: python3-autocommand noarch 2.2.2-8.el9 epel 29 k 2026-02-23T10:51:10.022 INFO:teuthology.orchestra.run.trial198.stdout: python3-backports-tarfile noarch 1.2.0-1.el9 epel 60 k 2026-02-23T10:51:10.022 INFO:teuthology.orchestra.run.trial198.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2026-02-23T10:51:10.022 INFO:teuthology.orchestra.run.trial198.stdout: python3-cffi x86_64 1.14.5-5.el9 baseos 253 k 2026-02-23T10:51:10.022 INFO:teuthology.orchestra.run.trial198.stdout: python3-cheroot noarch 10.0.1-4.el9 epel 173 k 2026-02-23T10:51:10.022 INFO:teuthology.orchestra.run.trial198.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2026-02-23T10:51:10.022 INFO:teuthology.orchestra.run.trial198.stdout: python3-cryptography x86_64 36.0.1-5.el9 baseos 1.2 M 2026-02-23T10:51:10.022 INFO:teuthology.orchestra.run.trial198.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2026-02-23T10:51:10.023 INFO:teuthology.orchestra.run.trial198.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2026-02-23T10:51:10.023 INFO:teuthology.orchestra.run.trial198.stdout: python3-jaraco-collections 2026-02-23T10:51:10.023 INFO:teuthology.orchestra.run.trial198.stdout: noarch 3.0.0-8.el9 epel 23 k 2026-02-23T10:51:10.023 INFO:teuthology.orchestra.run.trial198.stdout: python3-jaraco-context noarch 6.0.1-3.el9 epel 20 k 2026-02-23T10:51:10.023 INFO:teuthology.orchestra.run.trial198.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2026-02-23T10:51:10.023 INFO:teuthology.orchestra.run.trial198.stdout: python3-jaraco-text noarch 4.0.0-2.el9 epel 26 k 2026-02-23T10:51:10.023 INFO:teuthology.orchestra.run.trial198.stdout: python3-libstoragemgmt x86_64 1.10.1-1.el9 appstream 177 k 2026-02-23T10:51:10.023 INFO:teuthology.orchestra.run.trial198.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2026-02-23T10:51:10.023 INFO:teuthology.orchestra.run.trial198.stdout: python3-packaging noarch 20.9-5.el9 appstream 77 k 2026-02-23T10:51:10.023 INFO:teuthology.orchestra.run.trial198.stdout: python3-ply noarch 3.11-14.el9 baseos 106 k 2026-02-23T10:51:10.023 INFO:teuthology.orchestra.run.trial198.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2026-02-23T10:51:10.023 INFO:teuthology.orchestra.run.trial198.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2026-02-23T10:51:10.023 INFO:teuthology.orchestra.run.trial198.stdout: python3-pycparser noarch 2.20-6.el9 baseos 135 k 2026-02-23T10:51:10.023 INFO:teuthology.orchestra.run.trial198.stdout: python3-pyparsing noarch 2.4.7-9.el9 baseos 150 k 2026-02-23T10:51:10.023 INFO:teuthology.orchestra.run.trial198.stdout: python3-requests noarch 2.25.1-10.el9 baseos 126 k 2026-02-23T10:51:10.023 INFO:teuthology.orchestra.run.trial198.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2026-02-23T10:51:10.023 INFO:teuthology.orchestra.run.trial198.stdout: python3-toml noarch 0.10.2-6.el9 appstream 42 k 2026-02-23T10:51:10.023 INFO:teuthology.orchestra.run.trial198.stdout: python3-urllib3 noarch 1.26.5-7.el9 baseos 218 k 2026-02-23T10:51:10.023 INFO:teuthology.orchestra.run.trial198.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2026-02-23T10:51:10.023 INFO:teuthology.orchestra.run.trial198.stdout: unzip x86_64 6.0-59.el9 baseos 182 k 2026-02-23T10:51:10.024 INFO:teuthology.orchestra.run.trial198.stdout: zip x86_64 3.0-35.el9 baseos 266 k 2026-02-23T10:51:10.024 INFO:teuthology.orchestra.run.trial198.stdout: 2026-02-23T10:51:10.024 INFO:teuthology.orchestra.run.trial198.stdout:Transaction Summary 2026-02-23T10:51:10.024 INFO:teuthology.orchestra.run.trial198.stdout:================================================================================ 2026-02-23T10:51:10.024 INFO:teuthology.orchestra.run.trial198.stdout:Install 42 Packages 2026-02-23T10:51:10.024 INFO:teuthology.orchestra.run.trial198.stdout: 2026-02-23T10:51:10.024 INFO:teuthology.orchestra.run.trial198.stdout:Total download size: 114 M 2026-02-23T10:51:10.024 INFO:teuthology.orchestra.run.trial198.stdout:Installed size: 961 M 2026-02-23T10:51:10.024 INFO:teuthology.orchestra.run.trial198.stdout:Downloading Packages: 2026-02-23T10:51:10.780 INFO:teuthology.orchestra.run.trial198.stdout:(1/42): ceph-20.3.0-5439.ge371433f.el9.x86_64.r 40 kB/s | 6.5 kB 00:00 2026-02-23T10:51:11.041 INFO:teuthology.orchestra.run.trial198.stdout:(2/42): ceph-mgr-20.3.0-5439.ge371433f.el9.x86_ 2.2 MB/s | 937 kB 00:00 2026-02-23T10:51:11.093 INFO:teuthology.orchestra.run.trial198.stdout:(3/42): ceph-osd-20.3.0-5439.ge371433f.el9.x86_ 3.1 MB/s | 164 kB 00:00 2026-02-23T10:51:11.333 INFO:teuthology.orchestra.run.trial198.stdout:(4/42): ceph-mds-20.3.0-5439.ge371433f.el9.x86_ 2.8 MB/s | 2.0 MB 00:00 2026-02-23T10:51:11.716 INFO:teuthology.orchestra.run.trial198.stdout:(5/42): ceph-mon-20.3.0-5439.ge371433f.el9.x86_ 5.2 MB/s | 4.9 MB 00:00 2026-02-23T10:51:11.775 INFO:teuthology.orchestra.run.trial198.stdout:(6/42): ceph-mgr-modules-core-20.3.0-5439.ge371 4.9 MB/s | 298 kB 00:00 2026-02-23T10:51:12.178 INFO:teuthology.orchestra.run.trial198.stdout:(7/42): ledmon-libs-1.1.0-3.el9.x86_64.rpm 100 kB/s | 40 kB 00:00 2026-02-23T10:51:12.339 INFO:teuthology.orchestra.run.trial198.stdout:(8/42): libconfig-1.7.2-9.el9.x86_64.rpm 450 kB/s | 72 kB 00:00 2026-02-23T10:51:12.370 INFO:teuthology.orchestra.run.trial198.stdout:(9/42): ceph-osd-classic-20.3.0-5439.ge371433f. 12 MB/s | 15 MB 00:01 2026-02-23T10:51:12.656 INFO:teuthology.orchestra.run.trial198.stdout:(10/42): python3-cffi-1.14.5-5.el9.x86_64.rpm 798 kB/s | 253 kB 00:00 2026-02-23T10:51:12.817 INFO:teuthology.orchestra.run.trial198.stdout:(11/42): python3-ply-3.11-14.el9.noarch.rpm 662 kB/s | 106 kB 00:00 2026-02-23T10:51:12.978 INFO:teuthology.orchestra.run.trial198.stdout:(12/42): python3-pycparser-2.20-6.el9.noarch.rp 840 kB/s | 135 kB 00:00 2026-02-23T10:51:13.217 INFO:teuthology.orchestra.run.trial198.stdout:(13/42): python3-pyparsing-2.4.7-9.el9.noarch.r 631 kB/s | 150 kB 00:00 2026-02-23T10:51:13.379 INFO:teuthology.orchestra.run.trial198.stdout:(14/42): python3-requests-2.25.1-10.el9.noarch. 783 kB/s | 126 kB 00:00 2026-02-23T10:51:13.695 INFO:teuthology.orchestra.run.trial198.stdout:(15/42): python3-urllib3-1.26.5-7.el9.noarch.rp 689 kB/s | 218 kB 00:00 2026-02-23T10:51:13.934 INFO:teuthology.orchestra.run.trial198.stdout:(16/42): unzip-6.0-59.el9.x86_64.rpm 762 kB/s | 182 kB 00:00 2026-02-23T10:51:14.017 INFO:teuthology.orchestra.run.trial198.stdout:[MIRROR] zip-3.0-35.el9.x86_64.rpm: Status code: 503 for https://ftp2.osuosl.org/pub/centos-stream/9-stream/BaseOS/x86_64/os/Packages/zip-3.0-35.el9.x86_64.rpm (IP: 140.211.166.134) 2026-02-23T10:51:14.325 INFO:teuthology.orchestra.run.trial198.stdout:(17/42): python3-cryptography-36.0.1-5.el9.x86_ 652 kB/s | 1.2 MB 00:01 2026-02-23T10:51:14.351 INFO:teuthology.orchestra.run.trial198.stdout:[MIRROR] zip-3.0-35.el9.x86_64.rpm: Status code: 503 for https://ftp2.osuosl.org/pub/centos-stream/9-stream/BaseOS/x86_64/os/Packages/zip-3.0-35.el9.x86_64.rpm (IP: 140.211.166.134) 2026-02-23T10:51:14.408 INFO:teuthology.orchestra.run.trial198.stdout:[MIRROR] libstoragemgmt-1.10.1-1.el9.x86_64.rpm: Status code: 503 for https://ftp2.osuosl.org/pub/centos-stream/9-stream/AppStream/x86_64/os/Packages/libstoragemgmt-1.10.1-1.el9.x86_64.rpm (IP: 140.211.166.134) 2026-02-23T10:51:14.411 INFO:teuthology.orchestra.run.trial198.stdout:(18/42): zip-3.0-35.el9.x86_64.rpm 557 kB/s | 266 kB 00:00 2026-02-23T10:51:14.728 INFO:teuthology.orchestra.run.trial198.stdout:[MIRROR] libstoragemgmt-1.10.1-1.el9.x86_64.rpm: Status code: 503 for https://ftp2.osuosl.org/pub/centos-stream/9-stream/AppStream/x86_64/os/Packages/libstoragemgmt-1.10.1-1.el9.x86_64.rpm (IP: 140.211.166.134) 2026-02-23T10:51:14.731 INFO:teuthology.orchestra.run.trial198.stdout:[MIRROR] lua-5.4.4-4.el9.x86_64.rpm: Status code: 503 for https://ftp2.osuosl.org/pub/centos-stream/9-stream/AppStream/x86_64/os/Packages/lua-5.4.4-4.el9.x86_64.rpm (IP: 140.211.166.134) 2026-02-23T10:51:14.864 INFO:teuthology.orchestra.run.trial198.stdout:(19/42): libstoragemgmt-1.10.1-1.el9.x86_64.rpm 458 kB/s | 246 kB 00:00 2026-02-23T10:51:15.129 INFO:teuthology.orchestra.run.trial198.stdout:(20/42): ceph-osd-crimson-20.3.0-5439.ge371433f 23 MB/s | 87 MB 00:03 2026-02-23T10:51:15.130 INFO:teuthology.orchestra.run.trial198.stdout:[MIRROR] lua-5.4.4-4.el9.x86_64.rpm: Status code: 503 for https://ftp2.osuosl.org/pub/centos-stream/9-stream/AppStream/x86_64/os/Packages/lua-5.4.4-4.el9.x86_64.rpm (IP: 140.211.166.134) 2026-02-23T10:51:15.173 INFO:teuthology.orchestra.run.trial198.stdout:(21/42): lua-5.4.4-4.el9.x86_64.rpm 247 kB/s | 188 kB 00:00 2026-02-23T10:51:15.294 INFO:teuthology.orchestra.run.trial198.stdout:[MIRROR] python3-libstoragemgmt-1.10.1-1.el9.x86_64.rpm: Status code: 503 for https://ftp2.osuosl.org/pub/centos-stream/9-stream/AppStream/x86_64/os/Packages/python3-libstoragemgmt-1.10.1-1.el9.x86_64.rpm (IP: 140.211.166.134) 2026-02-23T10:51:15.573 INFO:teuthology.orchestra.run.trial198.stdout:(22/42): python3-toml-0.10.2-6.el9.noarch.rpm 104 kB/s | 42 kB 00:00 2026-02-23T10:51:15.583 INFO:teuthology.orchestra.run.trial198.stdout:(23/42): lua-devel-5.4.4-4.el9.x86_64.rpm 2.1 MB/s | 22 kB 00:00 2026-02-23T10:51:15.609 INFO:teuthology.orchestra.run.trial198.stdout:(24/42): python3-packaging-20.9-5.el9.noarch.rp 161 kB/s | 77 kB 00:00 2026-02-23T10:51:15.726 INFO:teuthology.orchestra.run.trial198.stdout:(25/42): python3-autocommand-2.2.2-8.el9.noarch 253 kB/s | 29 kB 00:00 2026-02-23T10:51:15.806 INFO:teuthology.orchestra.run.trial198.stdout:(26/42): python3-backports-tarfile-1.2.0-1.el9. 750 kB/s | 60 kB 00:00 2026-02-23T10:51:15.817 INFO:teuthology.orchestra.run.trial198.stdout:(27/42): luarocks-3.9.2-5.el9.noarch.rpm 649 kB/s | 151 kB 00:00 2026-02-23T10:51:15.853 INFO:teuthology.orchestra.run.trial198.stdout:(28/42): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 927 kB/s | 43 kB 00:00 2026-02-23T10:51:15.889 INFO:teuthology.orchestra.run.trial198.stdout:(29/42): python3-cheroot-10.0.1-4.el9.noarch.rp 2.3 MB/s | 173 kB 00:00 2026-02-23T10:51:15.926 INFO:teuthology.orchestra.run.trial198.stdout:(30/42): python3-jaraco-8.2.1-3.el9.noarch.rpm 290 kB/s | 11 kB 00:00 2026-02-23T10:51:15.933 INFO:teuthology.orchestra.run.trial198.stdout:(31/42): python3-libstoragemgmt-1.10.1-1.el9.x8 165 kB/s | 177 kB 00:01 2026-02-23T10:51:15.964 INFO:teuthology.orchestra.run.trial198.stdout:(32/42): python3-jaraco-classes-3.2.1-5.el9.noa 470 kB/s | 18 kB 00:00 2026-02-23T10:51:15.967 INFO:teuthology.orchestra.run.trial198.stdout:(33/42): python3-cherrypy-18.6.1-2.el9.noarch.r 3.1 MB/s | 358 kB 00:00 2026-02-23T10:51:16.001 INFO:teuthology.orchestra.run.trial198.stdout:(34/42): python3-jaraco-context-6.0.1-3.el9.noa 529 kB/s | 20 kB 00:00 2026-02-23T10:51:16.003 INFO:teuthology.orchestra.run.trial198.stdout:(35/42): python3-jaraco-functools-3.5.0-2.el9.n 541 kB/s | 19 kB 00:00 2026-02-23T10:51:16.039 INFO:teuthology.orchestra.run.trial198.stdout:(36/42): python3-jaraco-text-4.0.0-2.el9.noarch 696 kB/s | 26 kB 00:00 2026-02-23T10:51:16.043 INFO:teuthology.orchestra.run.trial198.stdout:(37/42): python3-more-itertools-8.12.0-2.el9.no 1.9 MB/s | 79 kB 00:00 2026-02-23T10:51:16.046 INFO:teuthology.orchestra.run.trial198.stdout:(38/42): python3-jaraco-collections-3.0.0-8.el9 204 kB/s | 23 kB 00:00 2026-02-23T10:51:16.076 INFO:teuthology.orchestra.run.trial198.stdout:(39/42): python3-portend-3.1.0-2.el9.noarch.rpm 441 kB/s | 16 kB 00:00 2026-02-23T10:51:16.083 INFO:teuthology.orchestra.run.trial198.stdout:(40/42): python3-pyOpenSSL-21.0.0-1.el9.noarch. 2.2 MB/s | 90 kB 00:00 2026-02-23T10:51:16.109 INFO:teuthology.orchestra.run.trial198.stdout:(41/42): python3-tempora-5.0.0-2.el9.noarch.rpm 572 kB/s | 36 kB 00:00 2026-02-23T10:51:16.114 INFO:teuthology.orchestra.run.trial198.stdout:(42/42): python3-zc-lockfile-2.0-10.el9.noarch. 529 kB/s | 20 kB 00:00 2026-02-23T10:51:16.116 INFO:teuthology.orchestra.run.trial198.stdout:-------------------------------------------------------------------------------- 2026-02-23T10:51:16.117 INFO:teuthology.orchestra.run.trial198.stdout:Total 19 MB/s | 114 MB 00:06 2026-02-23T10:51:16.264 INFO:teuthology.orchestra.run.trial198.stdout:Running transaction check 2026-02-23T10:51:16.283 INFO:teuthology.orchestra.run.trial198.stdout:Transaction check succeeded. 2026-02-23T10:51:16.283 INFO:teuthology.orchestra.run.trial198.stdout:Running transaction test 2026-02-23T10:51:16.676 INFO:teuthology.orchestra.run.trial198.stdout:Transaction test succeeded. 2026-02-23T10:51:16.676 INFO:teuthology.orchestra.run.trial198.stdout:Running transaction 2026-02-23T10:51:17.109 INFO:teuthology.orchestra.run.trial198.stdout: Preparing : 1/1 2026-02-23T10:51:17.111 INFO:teuthology.orchestra.run.trial198.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/42 2026-02-23T10:51:17.115 INFO:teuthology.orchestra.run.trial198.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/42 2026-02-23T10:51:17.120 INFO:teuthology.orchestra.run.trial198.stdout: Installing : lua-5.4.4-4.el9.x86_64 3/42 2026-02-23T10:51:17.122 INFO:teuthology.orchestra.run.trial198.stdout: Installing : unzip-6.0-59.el9.x86_64 4/42 2026-02-23T10:51:17.128 INFO:teuthology.orchestra.run.trial198.stdout: Installing : ledmon-libs-1.1.0-3.el9.x86_64 5/42 2026-02-23T10:51:17.152 INFO:teuthology.orchestra.run.trial198.stdout: Installing : zip-3.0-35.el9.x86_64 6/42 2026-02-23T10:51:17.156 INFO:teuthology.orchestra.run.trial198.stdout: Installing : luarocks-3.9.2-5.el9.noarch 7/42 2026-02-23T10:51:17.160 INFO:teuthology.orchestra.run.trial198.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 8/42 2026-02-23T10:51:17.165 INFO:teuthology.orchestra.run.trial198.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 9/42 2026-02-23T10:51:17.171 INFO:teuthology.orchestra.run.trial198.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 10/42 2026-02-23T10:51:17.175 INFO:teuthology.orchestra.run.trial198.stdout: Installing : python3-backports-tarfile-1.2.0-1.el9.noarch 11/42 2026-02-23T10:51:17.181 INFO:teuthology.orchestra.run.trial198.stdout: Installing : python3-jaraco-context-6.0.1-3.el9.noarch 12/42 2026-02-23T10:51:17.186 INFO:teuthology.orchestra.run.trial198.stdout: Installing : python3-autocommand-2.2.2-8.el9.noarch 13/42 2026-02-23T10:51:17.189 INFO:teuthology.orchestra.run.trial198.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 14/42 2026-02-23T10:51:17.195 INFO:teuthology.orchestra.run.trial198.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 15/42 2026-02-23T10:51:17.199 INFO:teuthology.orchestra.run.trial198.stdout: Installing : python3-jaraco-text-4.0.0-2.el9.noarch 16/42 2026-02-23T10:51:17.206 INFO:teuthology.orchestra.run.trial198.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 17/42 2026-02-23T10:51:17.210 INFO:teuthology.orchestra.run.trial198.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 18/42 2026-02-23T10:51:17.232 INFO:teuthology.orchestra.run.trial198.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 19/42 2026-02-23T10:51:17.244 INFO:teuthology.orchestra.run.trial198.stdout: Installing : python3-urllib3-1.26.5-7.el9.noarch 20/42 2026-02-23T10:51:17.250 INFO:teuthology.orchestra.run.trial198.stdout: Installing : python3-requests-2.25.1-10.el9.noarch 21/42 2026-02-23T10:51:17.259 INFO:teuthology.orchestra.run.trial198.stdout: Installing : python3-pyparsing-2.4.7-9.el9.noarch 22/42 2026-02-23T10:51:17.266 INFO:teuthology.orchestra.run.trial198.stdout: Installing : python3-packaging-20.9-5.el9.noarch 23/42 2026-02-23T10:51:17.279 INFO:teuthology.orchestra.run.trial198.stdout: Installing : python3-ply-3.11-14.el9.noarch 24/42 2026-02-23T10:51:17.295 INFO:teuthology.orchestra.run.trial198.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 25/42 2026-02-23T10:51:17.370 INFO:teuthology.orchestra.run.trial198.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 26/42 2026-02-23T10:51:17.378 INFO:teuthology.orchestra.run.trial198.stdout: Installing : python3-cryptography-36.0.1-5.el9.x86_64 27/42 2026-02-23T10:51:17.401 INFO:teuthology.orchestra.run.trial198.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 28/42 2026-02-23T10:51:17.431 INFO:teuthology.orchestra.run.trial198.stdout: Installing : python3-cheroot-10.0.1-4.el9.noarch 29/42 2026-02-23T10:51:17.436 INFO:teuthology.orchestra.run.trial198.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 30/42 2026-02-23T10:51:17.489 INFO:teuthology.orchestra.run.trial198.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 31/42 2026-02-23T10:51:17.529 INFO:teuthology.orchestra.run.trial198.stdout: Installing : ceph-mgr-modules-core-2:20.3.0-5439.ge371433f.el9. 32/42 2026-02-23T10:51:17.530 INFO:teuthology.orchestra.run.trial198.stdout: Installing : ceph-mgr-2:20.3.0-5439.ge371433f.el9.x86_64 33/42 2026-02-23T10:51:17.554 INFO:teuthology.orchestra.run.trial198.stdout: Running scriptlet: ceph-mgr-2:20.3.0-5439.ge371433f.el9.x86_64 33/42 2026-02-23T10:51:17.554 INFO:teuthology.orchestra.run.trial198.stdout:Glob pattern passed to enable, but globs are not supported for this. 2026-02-23T10:51:17.554 INFO:teuthology.orchestra.run.trial198.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2026-02-23T10:51:17.554 INFO:teuthology.orchestra.run.trial198.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2026-02-23T10:51:17.554 INFO:teuthology.orchestra.run.trial198.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2026-02-23T10:51:17.555 INFO:teuthology.orchestra.run.trial198.stdout: 2026-02-23T10:51:17.559 INFO:teuthology.orchestra.run.trial198.stdout: Installing : libconfig-1.7.2-9.el9.x86_64 34/42 2026-02-23T10:51:17.575 INFO:teuthology.orchestra.run.trial198.stdout: Running scriptlet: libstoragemgmt-1.10.1-1.el9.x86_64 35/42 2026-02-23T10:51:17.575 INFO:teuthology.orchestra.run.trial198.stdout:Creating group 'libstoragemgmt' with GID 991. 2026-02-23T10:51:17.575 INFO:teuthology.orchestra.run.trial198.stdout:Creating user 'libstoragemgmt' (daemon account for libstoragemgmt) with UID 991 and GID 991. 2026-02-23T10:51:17.575 INFO:teuthology.orchestra.run.trial198.stdout: 2026-02-23T10:51:17.582 INFO:teuthology.orchestra.run.trial198.stdout: Installing : libstoragemgmt-1.10.1-1.el9.x86_64 35/42 2026-02-23T10:51:17.606 INFO:teuthology.orchestra.run.trial198.stdout: Running scriptlet: libstoragemgmt-1.10.1-1.el9.x86_64 35/42 2026-02-23T10:51:17.606 INFO:teuthology.orchestra.run.trial198.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/libstoragemgmt.service → /usr/lib/systemd/system/libstoragemgmt.service. 2026-02-23T10:51:17.606 INFO:teuthology.orchestra.run.trial198.stdout: 2026-02-23T10:51:18.218 INFO:teuthology.orchestra.run.trial198.stdout: Installing : python3-libstoragemgmt-1.10.1-1.el9.x86_64 36/42 2026-02-23T10:51:23.624 INFO:teuthology.orchestra.run.trial198.stdout: Installing : ceph-osd-classic-2:20.3.0-5439.ge371433f.el9.x86_6 37/42 2026-02-23T10:51:23.634 INFO:teuthology.orchestra.run.trial198.stdout: Installing : ceph-osd-crimson-2:20.3.0-5439.ge371433f.el9.x86_6 38/42 2026-02-23T10:51:23.635 INFO:teuthology.orchestra.run.trial198.stdout: Installing : ceph-osd-2:20.3.0-5439.ge371433f.el9.x86_64 39/42 2026-02-23T10:51:23.660 INFO:teuthology.orchestra.run.trial198.stdout: Running scriptlet: ceph-osd-2:20.3.0-5439.ge371433f.el9.x86_64 39/42 2026-02-23T10:51:23.660 INFO:teuthology.orchestra.run.trial198.stdout:Glob pattern passed to enable, but globs are not supported for this. 2026-02-23T10:51:23.660 INFO:teuthology.orchestra.run.trial198.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2026-02-23T10:51:23.660 INFO:teuthology.orchestra.run.trial198.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2026-02-23T10:51:23.660 INFO:teuthology.orchestra.run.trial198.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2026-02-23T10:51:23.660 INFO:teuthology.orchestra.run.trial198.stdout: 2026-02-23T10:51:23.870 INFO:teuthology.orchestra.run.trial198.stdout: Installing : ceph-mon-2:20.3.0-5439.ge371433f.el9.x86_64 40/42 2026-02-23T10:51:23.891 INFO:teuthology.orchestra.run.trial198.stdout: Running scriptlet: ceph-mon-2:20.3.0-5439.ge371433f.el9.x86_64 40/42 2026-02-23T10:51:23.891 INFO:teuthology.orchestra.run.trial198.stdout:Glob pattern passed to enable, but globs are not supported for this. 2026-02-23T10:51:23.891 INFO:teuthology.orchestra.run.trial198.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2026-02-23T10:51:23.891 INFO:teuthology.orchestra.run.trial198.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2026-02-23T10:51:23.891 INFO:teuthology.orchestra.run.trial198.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2026-02-23T10:51:23.891 INFO:teuthology.orchestra.run.trial198.stdout: 2026-02-23T10:51:23.976 INFO:teuthology.orchestra.run.trial198.stdout: Installing : ceph-mds-2:20.3.0-5439.ge371433f.el9.x86_64 41/42 2026-02-23T10:51:23.996 INFO:teuthology.orchestra.run.trial198.stdout: Running scriptlet: ceph-mds-2:20.3.0-5439.ge371433f.el9.x86_64 41/42 2026-02-23T10:51:23.996 INFO:teuthology.orchestra.run.trial198.stdout:Glob pattern passed to enable, but globs are not supported for this. 2026-02-23T10:51:23.996 INFO:teuthology.orchestra.run.trial198.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2026-02-23T10:51:23.996 INFO:teuthology.orchestra.run.trial198.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2026-02-23T10:51:23.996 INFO:teuthology.orchestra.run.trial198.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2026-02-23T10:51:23.997 INFO:teuthology.orchestra.run.trial198.stdout: 2026-02-23T10:51:23.997 INFO:teuthology.orchestra.run.trial198.stdout: Installing : ceph-2:20.3.0-5439.ge371433f.el9.x86_64 42/42 2026-02-23T10:51:24.004 INFO:teuthology.orchestra.run.trial198.stdout: Running scriptlet: ceph-osd-classic-2:20.3.0-5439.ge371433f.el9.x86_6 42/42 2026-02-23T10:51:24.029 INFO:teuthology.orchestra.run.trial198.stdout: Running scriptlet: ceph-osd-crimson-2:20.3.0-5439.ge371433f.el9.x86_6 42/42 2026-02-23T10:51:25.002 INFO:teuthology.orchestra.run.trial198.stdout: Running scriptlet: ceph-2:20.3.0-5439.ge371433f.el9.x86_64 42/42 2026-02-23T10:51:25.002 INFO:teuthology.orchestra.run.trial198.stdout: Verifying : ceph-2:20.3.0-5439.ge371433f.el9.x86_64 1/42 2026-02-23T10:51:25.002 INFO:teuthology.orchestra.run.trial198.stdout: Verifying : ceph-mds-2:20.3.0-5439.ge371433f.el9.x86_64 2/42 2026-02-23T10:51:25.003 INFO:teuthology.orchestra.run.trial198.stdout: Verifying : ceph-mgr-2:20.3.0-5439.ge371433f.el9.x86_64 3/42 2026-02-23T10:51:25.003 INFO:teuthology.orchestra.run.trial198.stdout: Verifying : ceph-mon-2:20.3.0-5439.ge371433f.el9.x86_64 4/42 2026-02-23T10:51:25.003 INFO:teuthology.orchestra.run.trial198.stdout: Verifying : ceph-osd-2:20.3.0-5439.ge371433f.el9.x86_64 5/42 2026-02-23T10:51:25.003 INFO:teuthology.orchestra.run.trial198.stdout: Verifying : ceph-osd-classic-2:20.3.0-5439.ge371433f.el9.x86_6 6/42 2026-02-23T10:51:25.003 INFO:teuthology.orchestra.run.trial198.stdout: Verifying : ceph-osd-crimson-2:20.3.0-5439.ge371433f.el9.x86_6 7/42 2026-02-23T10:51:25.003 INFO:teuthology.orchestra.run.trial198.stdout: Verifying : ceph-mgr-modules-core-2:20.3.0-5439.ge371433f.el9. 8/42 2026-02-23T10:51:25.003 INFO:teuthology.orchestra.run.trial198.stdout: Verifying : ledmon-libs-1.1.0-3.el9.x86_64 9/42 2026-02-23T10:51:25.003 INFO:teuthology.orchestra.run.trial198.stdout: Verifying : libconfig-1.7.2-9.el9.x86_64 10/42 2026-02-23T10:51:25.003 INFO:teuthology.orchestra.run.trial198.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 11/42 2026-02-23T10:51:25.003 INFO:teuthology.orchestra.run.trial198.stdout: Verifying : python3-cryptography-36.0.1-5.el9.x86_64 12/42 2026-02-23T10:51:25.003 INFO:teuthology.orchestra.run.trial198.stdout: Verifying : python3-ply-3.11-14.el9.noarch 13/42 2026-02-23T10:51:25.003 INFO:teuthology.orchestra.run.trial198.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 14/42 2026-02-23T10:51:25.004 INFO:teuthology.orchestra.run.trial198.stdout: Verifying : python3-pyparsing-2.4.7-9.el9.noarch 15/42 2026-02-23T10:51:25.004 INFO:teuthology.orchestra.run.trial198.stdout: Verifying : python3-requests-2.25.1-10.el9.noarch 16/42 2026-02-23T10:51:25.004 INFO:teuthology.orchestra.run.trial198.stdout: Verifying : python3-urllib3-1.26.5-7.el9.noarch 17/42 2026-02-23T10:51:25.004 INFO:teuthology.orchestra.run.trial198.stdout: Verifying : unzip-6.0-59.el9.x86_64 18/42 2026-02-23T10:51:25.004 INFO:teuthology.orchestra.run.trial198.stdout: Verifying : zip-3.0-35.el9.x86_64 19/42 2026-02-23T10:51:25.004 INFO:teuthology.orchestra.run.trial198.stdout: Verifying : libstoragemgmt-1.10.1-1.el9.x86_64 20/42 2026-02-23T10:51:25.004 INFO:teuthology.orchestra.run.trial198.stdout: Verifying : lua-5.4.4-4.el9.x86_64 21/42 2026-02-23T10:51:25.004 INFO:teuthology.orchestra.run.trial198.stdout: Verifying : python3-libstoragemgmt-1.10.1-1.el9.x86_64 22/42 2026-02-23T10:51:25.004 INFO:teuthology.orchestra.run.trial198.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 23/42 2026-02-23T10:51:25.004 INFO:teuthology.orchestra.run.trial198.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 24/42 2026-02-23T10:51:25.004 INFO:teuthology.orchestra.run.trial198.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 25/42 2026-02-23T10:51:25.004 INFO:teuthology.orchestra.run.trial198.stdout: Verifying : luarocks-3.9.2-5.el9.noarch 26/42 2026-02-23T10:51:25.005 INFO:teuthology.orchestra.run.trial198.stdout: Verifying : python3-autocommand-2.2.2-8.el9.noarch 27/42 2026-02-23T10:51:25.005 INFO:teuthology.orchestra.run.trial198.stdout: Verifying : python3-backports-tarfile-1.2.0-1.el9.noarch 28/42 2026-02-23T10:51:25.005 INFO:teuthology.orchestra.run.trial198.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 29/42 2026-02-23T10:51:25.005 INFO:teuthology.orchestra.run.trial198.stdout: Verifying : python3-cheroot-10.0.1-4.el9.noarch 30/42 2026-02-23T10:51:25.005 INFO:teuthology.orchestra.run.trial198.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 31/42 2026-02-23T10:51:25.005 INFO:teuthology.orchestra.run.trial198.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 32/42 2026-02-23T10:51:25.005 INFO:teuthology.orchestra.run.trial198.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 33/42 2026-02-23T10:51:25.005 INFO:teuthology.orchestra.run.trial198.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 34/42 2026-02-23T10:51:25.005 INFO:teuthology.orchestra.run.trial198.stdout: Verifying : python3-jaraco-context-6.0.1-3.el9.noarch 35/42 2026-02-23T10:51:25.005 INFO:teuthology.orchestra.run.trial198.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 36/42 2026-02-23T10:51:25.005 INFO:teuthology.orchestra.run.trial198.stdout: Verifying : python3-jaraco-text-4.0.0-2.el9.noarch 37/42 2026-02-23T10:51:25.006 INFO:teuthology.orchestra.run.trial198.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 38/42 2026-02-23T10:51:25.006 INFO:teuthology.orchestra.run.trial198.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 39/42 2026-02-23T10:51:25.006 INFO:teuthology.orchestra.run.trial198.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 40/42 2026-02-23T10:51:25.006 INFO:teuthology.orchestra.run.trial198.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 41/42 2026-02-23T10:51:25.059 INFO:teuthology.orchestra.run.trial198.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 42/42 2026-02-23T10:51:25.059 INFO:teuthology.orchestra.run.trial198.stdout: 2026-02-23T10:51:25.060 INFO:teuthology.orchestra.run.trial198.stdout:Installed: 2026-02-23T10:51:25.060 INFO:teuthology.orchestra.run.trial198.stdout: ceph-2:20.3.0-5439.ge371433f.el9.x86_64 2026-02-23T10:51:25.060 INFO:teuthology.orchestra.run.trial198.stdout: ceph-mds-2:20.3.0-5439.ge371433f.el9.x86_64 2026-02-23T10:51:25.060 INFO:teuthology.orchestra.run.trial198.stdout: ceph-mgr-2:20.3.0-5439.ge371433f.el9.x86_64 2026-02-23T10:51:25.060 INFO:teuthology.orchestra.run.trial198.stdout: ceph-mgr-modules-core-2:20.3.0-5439.ge371433f.el9.noarch 2026-02-23T10:51:25.060 INFO:teuthology.orchestra.run.trial198.stdout: ceph-mon-2:20.3.0-5439.ge371433f.el9.x86_64 2026-02-23T10:51:25.060 INFO:teuthology.orchestra.run.trial198.stdout: ceph-osd-2:20.3.0-5439.ge371433f.el9.x86_64 2026-02-23T10:51:25.060 INFO:teuthology.orchestra.run.trial198.stdout: ceph-osd-classic-2:20.3.0-5439.ge371433f.el9.x86_64 2026-02-23T10:51:25.060 INFO:teuthology.orchestra.run.trial198.stdout: ceph-osd-crimson-2:20.3.0-5439.ge371433f.el9.x86_64 2026-02-23T10:51:25.060 INFO:teuthology.orchestra.run.trial198.stdout: ledmon-libs-1.1.0-3.el9.x86_64 2026-02-23T10:51:25.061 INFO:teuthology.orchestra.run.trial198.stdout: libconfig-1.7.2-9.el9.x86_64 2026-02-23T10:51:25.061 INFO:teuthology.orchestra.run.trial198.stdout: libstoragemgmt-1.10.1-1.el9.x86_64 2026-02-23T10:51:25.061 INFO:teuthology.orchestra.run.trial198.stdout: lua-5.4.4-4.el9.x86_64 2026-02-23T10:51:25.061 INFO:teuthology.orchestra.run.trial198.stdout: lua-devel-5.4.4-4.el9.x86_64 2026-02-23T10:51:25.061 INFO:teuthology.orchestra.run.trial198.stdout: luarocks-3.9.2-5.el9.noarch 2026-02-23T10:51:25.061 INFO:teuthology.orchestra.run.trial198.stdout: python3-autocommand-2.2.2-8.el9.noarch 2026-02-23T10:51:25.061 INFO:teuthology.orchestra.run.trial198.stdout: python3-backports-tarfile-1.2.0-1.el9.noarch 2026-02-23T10:51:25.061 INFO:teuthology.orchestra.run.trial198.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2026-02-23T10:51:25.061 INFO:teuthology.orchestra.run.trial198.stdout: python3-cffi-1.14.5-5.el9.x86_64 2026-02-23T10:51:25.061 INFO:teuthology.orchestra.run.trial198.stdout: python3-cheroot-10.0.1-4.el9.noarch 2026-02-23T10:51:25.061 INFO:teuthology.orchestra.run.trial198.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2026-02-23T10:51:25.061 INFO:teuthology.orchestra.run.trial198.stdout: python3-cryptography-36.0.1-5.el9.x86_64 2026-02-23T10:51:25.061 INFO:teuthology.orchestra.run.trial198.stdout: python3-jaraco-8.2.1-3.el9.noarch 2026-02-23T10:51:25.062 INFO:teuthology.orchestra.run.trial198.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2026-02-23T10:51:25.062 INFO:teuthology.orchestra.run.trial198.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2026-02-23T10:51:25.062 INFO:teuthology.orchestra.run.trial198.stdout: python3-jaraco-context-6.0.1-3.el9.noarch 2026-02-23T10:51:25.062 INFO:teuthology.orchestra.run.trial198.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2026-02-23T10:51:25.062 INFO:teuthology.orchestra.run.trial198.stdout: python3-jaraco-text-4.0.0-2.el9.noarch 2026-02-23T10:51:25.062 INFO:teuthology.orchestra.run.trial198.stdout: python3-libstoragemgmt-1.10.1-1.el9.x86_64 2026-02-23T10:51:25.062 INFO:teuthology.orchestra.run.trial198.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2026-02-23T10:51:25.062 INFO:teuthology.orchestra.run.trial198.stdout: python3-packaging-20.9-5.el9.noarch 2026-02-23T10:51:25.062 INFO:teuthology.orchestra.run.trial198.stdout: python3-ply-3.11-14.el9.noarch 2026-02-23T10:51:25.062 INFO:teuthology.orchestra.run.trial198.stdout: python3-portend-3.1.0-2.el9.noarch 2026-02-23T10:51:25.062 INFO:teuthology.orchestra.run.trial198.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2026-02-23T10:51:25.062 INFO:teuthology.orchestra.run.trial198.stdout: python3-pycparser-2.20-6.el9.noarch 2026-02-23T10:51:25.062 INFO:teuthology.orchestra.run.trial198.stdout: python3-pyparsing-2.4.7-9.el9.noarch 2026-02-23T10:51:25.063 INFO:teuthology.orchestra.run.trial198.stdout: python3-requests-2.25.1-10.el9.noarch 2026-02-23T10:51:25.063 INFO:teuthology.orchestra.run.trial198.stdout: python3-tempora-5.0.0-2.el9.noarch 2026-02-23T10:51:25.063 INFO:teuthology.orchestra.run.trial198.stdout: python3-toml-0.10.2-6.el9.noarch 2026-02-23T10:51:25.063 INFO:teuthology.orchestra.run.trial198.stdout: python3-urllib3-1.26.5-7.el9.noarch 2026-02-23T10:51:25.063 INFO:teuthology.orchestra.run.trial198.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2026-02-23T10:51:25.063 INFO:teuthology.orchestra.run.trial198.stdout: unzip-6.0-59.el9.x86_64 2026-02-23T10:51:25.063 INFO:teuthology.orchestra.run.trial198.stdout: zip-3.0-35.el9.x86_64 2026-02-23T10:51:25.063 INFO:teuthology.orchestra.run.trial198.stdout: 2026-02-23T10:51:25.063 INFO:teuthology.orchestra.run.trial198.stdout:Complete! 2026-02-23T10:51:25.105 DEBUG:teuthology.orchestra.run.trial198:> sudo yum -y install ceph-base 2026-02-23T10:51:25.416 INFO:teuthology.orchestra.run.trial198.stdout:Last metadata expiration check: 0:00:53 ago on Mon 23 Feb 2026 10:50:32 AM UTC. 2026-02-23T10:51:25.479 INFO:teuthology.orchestra.run.trial198.stdout:Package ceph-base-2:20.3.0-5439.ge371433f.el9.x86_64 is already installed. 2026-02-23T10:51:25.497 INFO:teuthology.orchestra.run.trial198.stdout:Dependencies resolved. 2026-02-23T10:51:25.497 INFO:teuthology.orchestra.run.trial198.stdout:Nothing to do. 2026-02-23T10:51:25.497 INFO:teuthology.orchestra.run.trial198.stdout:Complete! 2026-02-23T10:51:25.518 DEBUG:teuthology.orchestra.run.trial198:> sudo yum -y install cephadm 2026-02-23T10:51:25.824 INFO:teuthology.orchestra.run.trial198.stdout:Last metadata expiration check: 0:00:53 ago on Mon 23 Feb 2026 10:50:32 AM UTC. 2026-02-23T10:51:25.905 INFO:teuthology.orchestra.run.trial198.stdout:Dependencies resolved. 2026-02-23T10:51:25.905 INFO:teuthology.orchestra.run.trial198.stdout:================================================================================ 2026-02-23T10:51:25.905 INFO:teuthology.orchestra.run.trial198.stdout: Package Arch Version Repository Size 2026-02-23T10:51:25.906 INFO:teuthology.orchestra.run.trial198.stdout:================================================================================ 2026-02-23T10:51:25.906 INFO:teuthology.orchestra.run.trial198.stdout:Installing: 2026-02-23T10:51:25.906 INFO:teuthology.orchestra.run.trial198.stdout: cephadm noarch 2:20.3.0-5439.ge371433f.el9 ceph-noarch 1.0 M 2026-02-23T10:51:25.906 INFO:teuthology.orchestra.run.trial198.stdout: 2026-02-23T10:51:25.906 INFO:teuthology.orchestra.run.trial198.stdout:Transaction Summary 2026-02-23T10:51:25.906 INFO:teuthology.orchestra.run.trial198.stdout:================================================================================ 2026-02-23T10:51:25.906 INFO:teuthology.orchestra.run.trial198.stdout:Install 1 Package 2026-02-23T10:51:25.906 INFO:teuthology.orchestra.run.trial198.stdout: 2026-02-23T10:51:25.907 INFO:teuthology.orchestra.run.trial198.stdout:Total download size: 1.0 M 2026-02-23T10:51:25.907 INFO:teuthology.orchestra.run.trial198.stdout:Installed size: 1.0 M 2026-02-23T10:51:25.907 INFO:teuthology.orchestra.run.trial198.stdout:Downloading Packages: 2026-02-23T10:51:26.967 INFO:teuthology.orchestra.run.trial198.stdout:cephadm-20.3.0-5439.ge371433f.el9.noarch.rpm 983 kB/s | 1.0 MB 00:01 2026-02-23T10:51:26.968 INFO:teuthology.orchestra.run.trial198.stdout:-------------------------------------------------------------------------------- 2026-02-23T10:51:26.968 INFO:teuthology.orchestra.run.trial198.stdout:Total 982 kB/s | 1.0 MB 00:01 2026-02-23T10:51:26.968 INFO:teuthology.orchestra.run.trial198.stdout:Running transaction check 2026-02-23T10:51:26.971 INFO:teuthology.orchestra.run.trial198.stdout:Transaction check succeeded. 2026-02-23T10:51:26.971 INFO:teuthology.orchestra.run.trial198.stdout:Running transaction test 2026-02-23T10:51:26.975 INFO:teuthology.orchestra.run.trial198.stdout:Transaction test succeeded. 2026-02-23T10:51:26.975 INFO:teuthology.orchestra.run.trial198.stdout:Running transaction 2026-02-23T10:51:26.993 INFO:teuthology.orchestra.run.trial198.stdout: Preparing : 1/1 2026-02-23T10:51:27.055 INFO:teuthology.orchestra.run.trial198.stdout: Running scriptlet: cephadm-2:20.3.0-5439.ge371433f.el9.noarch 1/1 2026-02-23T10:51:27.069 INFO:teuthology.orchestra.run.trial198.stdout: Installing : cephadm-2:20.3.0-5439.ge371433f.el9.noarch 1/1 2026-02-23T10:51:27.137 INFO:teuthology.orchestra.run.trial198.stdout: Running scriptlet: cephadm-2:20.3.0-5439.ge371433f.el9.noarch 1/1 2026-02-23T10:51:27.192 INFO:teuthology.orchestra.run.trial198.stdout: Verifying : cephadm-2:20.3.0-5439.ge371433f.el9.noarch 1/1 2026-02-23T10:51:27.192 INFO:teuthology.orchestra.run.trial198.stdout: 2026-02-23T10:51:27.192 INFO:teuthology.orchestra.run.trial198.stdout:Installed: 2026-02-23T10:51:27.193 INFO:teuthology.orchestra.run.trial198.stdout: cephadm-2:20.3.0-5439.ge371433f.el9.noarch 2026-02-23T10:51:27.193 INFO:teuthology.orchestra.run.trial198.stdout: 2026-02-23T10:51:27.193 INFO:teuthology.orchestra.run.trial198.stdout:Complete! 2026-02-23T10:51:27.219 DEBUG:teuthology.orchestra.run.trial198:> sudo yum -y install ceph-immutable-object-cache 2026-02-23T10:51:27.529 INFO:teuthology.orchestra.run.trial198.stdout:Last metadata expiration check: 0:00:55 ago on Mon 23 Feb 2026 10:50:32 AM UTC. 2026-02-23T10:51:27.611 INFO:teuthology.orchestra.run.trial198.stdout:Dependencies resolved. 2026-02-23T10:51:27.611 INFO:teuthology.orchestra.run.trial198.stdout:================================================================================ 2026-02-23T10:51:27.611 INFO:teuthology.orchestra.run.trial198.stdout: Package Arch Version Repo Size 2026-02-23T10:51:27.611 INFO:teuthology.orchestra.run.trial198.stdout:================================================================================ 2026-02-23T10:51:27.612 INFO:teuthology.orchestra.run.trial198.stdout:Installing: 2026-02-23T10:51:27.612 INFO:teuthology.orchestra.run.trial198.stdout: ceph-immutable-object-cache x86_64 2:20.3.0-5439.ge371433f.el9 ceph 149 k 2026-02-23T10:51:27.612 INFO:teuthology.orchestra.run.trial198.stdout: 2026-02-23T10:51:27.612 INFO:teuthology.orchestra.run.trial198.stdout:Transaction Summary 2026-02-23T10:51:27.612 INFO:teuthology.orchestra.run.trial198.stdout:================================================================================ 2026-02-23T10:51:27.612 INFO:teuthology.orchestra.run.trial198.stdout:Install 1 Package 2026-02-23T10:51:27.612 INFO:teuthology.orchestra.run.trial198.stdout: 2026-02-23T10:51:27.612 INFO:teuthology.orchestra.run.trial198.stdout:Total download size: 149 k 2026-02-23T10:51:27.612 INFO:teuthology.orchestra.run.trial198.stdout:Installed size: 455 k 2026-02-23T10:51:27.612 INFO:teuthology.orchestra.run.trial198.stdout:Downloading Packages: 2026-02-23T10:51:28.004 INFO:teuthology.orchestra.run.trial198.stdout:ceph-immutable-object-cache-20.3.0-5439.ge37143 379 kB/s | 149 kB 00:00 2026-02-23T10:51:28.005 INFO:teuthology.orchestra.run.trial198.stdout:-------------------------------------------------------------------------------- 2026-02-23T10:51:28.005 INFO:teuthology.orchestra.run.trial198.stdout:Total 378 kB/s | 149 kB 00:00 2026-02-23T10:51:28.005 INFO:teuthology.orchestra.run.trial198.stdout:Running transaction check 2026-02-23T10:51:28.010 INFO:teuthology.orchestra.run.trial198.stdout:Transaction check succeeded. 2026-02-23T10:51:28.010 INFO:teuthology.orchestra.run.trial198.stdout:Running transaction test 2026-02-23T10:51:28.036 INFO:teuthology.orchestra.run.trial198.stdout:Transaction test succeeded. 2026-02-23T10:51:28.036 INFO:teuthology.orchestra.run.trial198.stdout:Running transaction 2026-02-23T10:51:28.085 INFO:teuthology.orchestra.run.trial198.stdout: Preparing : 1/1 2026-02-23T10:51:28.087 INFO:teuthology.orchestra.run.trial198.stdout: Installing : ceph-immutable-object-cache-2:20.3.0-5439.ge371433f. 1/1 2026-02-23T10:51:28.109 INFO:teuthology.orchestra.run.trial198.stdout: Running scriptlet: ceph-immutable-object-cache-2:20.3.0-5439.ge371433f. 1/1 2026-02-23T10:51:28.109 INFO:teuthology.orchestra.run.trial198.stdout:Glob pattern passed to enable, but globs are not supported for this. 2026-02-23T10:51:28.109 INFO:teuthology.orchestra.run.trial198.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2026-02-23T10:51:28.109 INFO:teuthology.orchestra.run.trial198.stdout: 2026-02-23T10:51:28.387 INFO:teuthology.orchestra.run.trial198.stdout: Verifying : ceph-immutable-object-cache-2:20.3.0-5439.ge371433f. 1/1 2026-02-23T10:51:28.387 INFO:teuthology.orchestra.run.trial198.stdout: 2026-02-23T10:51:28.387 INFO:teuthology.orchestra.run.trial198.stdout:Installed: 2026-02-23T10:51:28.387 INFO:teuthology.orchestra.run.trial198.stdout: ceph-immutable-object-cache-2:20.3.0-5439.ge371433f.el9.x86_64 2026-02-23T10:51:28.387 INFO:teuthology.orchestra.run.trial198.stdout: 2026-02-23T10:51:28.387 INFO:teuthology.orchestra.run.trial198.stdout:Complete! 2026-02-23T10:51:28.415 DEBUG:teuthology.orchestra.run.trial198:> sudo yum -y install ceph-mgr 2026-02-23T10:51:28.724 INFO:teuthology.orchestra.run.trial198.stdout:Last metadata expiration check: 0:00:56 ago on Mon 23 Feb 2026 10:50:32 AM UTC. 2026-02-23T10:51:28.788 INFO:teuthology.orchestra.run.trial198.stdout:Package ceph-mgr-2:20.3.0-5439.ge371433f.el9.x86_64 is already installed. 2026-02-23T10:51:28.805 INFO:teuthology.orchestra.run.trial198.stdout:Dependencies resolved. 2026-02-23T10:51:28.806 INFO:teuthology.orchestra.run.trial198.stdout:Nothing to do. 2026-02-23T10:51:28.806 INFO:teuthology.orchestra.run.trial198.stdout:Complete! 2026-02-23T10:51:28.827 DEBUG:teuthology.orchestra.run.trial198:> sudo yum -y install ceph-mgr-dashboard 2026-02-23T10:51:29.137 INFO:teuthology.orchestra.run.trial198.stdout:Last metadata expiration check: 0:00:57 ago on Mon 23 Feb 2026 10:50:32 AM UTC. 2026-02-23T10:51:29.220 INFO:teuthology.orchestra.run.trial198.stdout:Dependencies resolved. 2026-02-23T10:51:29.221 INFO:teuthology.orchestra.run.trial198.stdout:================================================================================ 2026-02-23T10:51:29.222 INFO:teuthology.orchestra.run.trial198.stdout: Package Arch Version Repository Size 2026-02-23T10:51:29.222 INFO:teuthology.orchestra.run.trial198.stdout:================================================================================ 2026-02-23T10:51:29.222 INFO:teuthology.orchestra.run.trial198.stdout:Installing: 2026-02-23T10:51:29.222 INFO:teuthology.orchestra.run.trial198.stdout: ceph-mgr-dashboard noarch 2:20.3.0-5439.ge371433f.el9 ceph-noarch 51 M 2026-02-23T10:51:29.222 INFO:teuthology.orchestra.run.trial198.stdout:Installing dependencies: 2026-02-23T10:51:29.222 INFO:teuthology.orchestra.run.trial198.stdout: abseil-cpp x86_64 20211102.0-4.el9 epel 551 k 2026-02-23T10:51:29.222 INFO:teuthology.orchestra.run.trial198.stdout: ceph-grafana-dashboards noarch 2:20.3.0-5439.ge371433f.el9 ceph-noarch 57 k 2026-02-23T10:51:29.222 INFO:teuthology.orchestra.run.trial198.stdout: ceph-prometheus-alerts noarch 2:20.3.0-5439.ge371433f.el9 ceph-noarch 17 k 2026-02-23T10:51:29.222 INFO:teuthology.orchestra.run.trial198.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2026-02-23T10:51:29.222 INFO:teuthology.orchestra.run.trial198.stdout: protobuf-compiler x86_64 3.14.0-17.el9 crb 862 k 2026-02-23T10:51:29.223 INFO:teuthology.orchestra.run.trial198.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2026-02-23T10:51:29.223 INFO:teuthology.orchestra.run.trial198.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2026-02-23T10:51:29.223 INFO:teuthology.orchestra.run.trial198.stdout: python3-jmespath noarch 1.0.1-1.el9 appstream 48 k 2026-02-23T10:51:29.223 INFO:teuthology.orchestra.run.trial198.stdout: python3-protobuf noarch 3.14.0-17.el9 appstream 267 k 2026-02-23T10:51:29.223 INFO:teuthology.orchestra.run.trial198.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2026-02-23T10:51:29.223 INFO:teuthology.orchestra.run.trial198.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2026-02-23T10:51:29.223 INFO:teuthology.orchestra.run.trial198.stdout: python3-xmltodict noarch 0.12.0-15.el9 epel 22 k 2026-02-23T10:51:29.223 INFO:teuthology.orchestra.run.trial198.stdout: 2026-02-23T10:51:29.223 INFO:teuthology.orchestra.run.trial198.stdout:Transaction Summary 2026-02-23T10:51:29.223 INFO:teuthology.orchestra.run.trial198.stdout:================================================================================ 2026-02-23T10:51:29.223 INFO:teuthology.orchestra.run.trial198.stdout:Install 13 Packages 2026-02-23T10:51:29.223 INFO:teuthology.orchestra.run.trial198.stdout: 2026-02-23T10:51:29.224 INFO:teuthology.orchestra.run.trial198.stdout:Total download size: 55 M 2026-02-23T10:51:29.224 INFO:teuthology.orchestra.run.trial198.stdout:Installed size: 262 M 2026-02-23T10:51:29.224 INFO:teuthology.orchestra.run.trial198.stdout:Downloading Packages: 2026-02-23T10:51:29.803 INFO:teuthology.orchestra.run.trial198.stdout:(1/13): ceph-prometheus-alerts-20.3.0-5439.ge37 55 kB/s | 17 kB 00:00 2026-02-23T10:51:29.837 INFO:teuthology.orchestra.run.trial198.stdout:(2/13): python3-jmespath-1.0.1-1.el9.noarch.rpm 1.4 MB/s | 48 kB 00:00 2026-02-23T10:51:29.864 INFO:teuthology.orchestra.run.trial198.stdout:(3/13): ceph-grafana-dashboards-20.3.0-5439.ge3 152 kB/s | 57 kB 00:00 2026-02-23T10:51:29.867 INFO:teuthology.orchestra.run.trial198.stdout:(4/13): python3-protobuf-3.14.0-17.el9.noarch.r 8.7 MB/s | 267 kB 00:00 2026-02-23T10:51:29.912 INFO:teuthology.orchestra.run.trial198.stdout:(5/13): abseil-cpp-20211102.0-4.el9.x86_64.rpm 12 MB/s | 551 kB 00:00 2026-02-23T10:51:29.916 INFO:teuthology.orchestra.run.trial198.stdout:(6/13): grpc-data-1.46.7-10.el9.noarch.rpm 5.7 MB/s | 19 kB 00:00 2026-02-23T10:51:29.928 INFO:teuthology.orchestra.run.trial198.stdout:(7/13): python3-grpcio-1.46.7-10.el9.x86_64.rpm 174 MB/s | 2.0 MB 00:00 2026-02-23T10:51:29.931 INFO:teuthology.orchestra.run.trial198.stdout:(8/13): python3-grpcio-tools-1.46.7-10.el9.x86_ 38 MB/s | 144 kB 00:00 2026-02-23T10:51:29.935 INFO:teuthology.orchestra.run.trial198.stdout:(9/13): python3-repoze-lru-0.7-16.el9.noarch.rp 9.1 MB/s | 31 kB 00:00 2026-02-23T10:51:29.939 INFO:teuthology.orchestra.run.trial198.stdout:(10/13): python3-routes-2.5.1-5.el9.noarch.rpm 48 MB/s | 188 kB 00:00 2026-02-23T10:51:29.942 INFO:teuthology.orchestra.run.trial198.stdout:(11/13): python3-xmltodict-0.12.0-15.el9.noarch 6.4 MB/s | 22 kB 00:00 2026-02-23T10:51:30.091 INFO:teuthology.orchestra.run.trial198.stdout:(12/13): protobuf-compiler-3.14.0-17.el9.x86_64 3.7 MB/s | 862 kB 00:00 2026-02-23T10:51:48.558 INFO:teuthology.orchestra.run.trial198.stdout:(13/13): ceph-mgr-dashboard-20.3.0-5439.ge37143 2.7 MB/s | 51 MB 00:19 2026-02-23T10:51:48.560 INFO:teuthology.orchestra.run.trial198.stdout:-------------------------------------------------------------------------------- 2026-02-23T10:51:48.560 INFO:teuthology.orchestra.run.trial198.stdout:Total 2.8 MB/s | 55 MB 00:19 2026-02-23T10:51:48.585 INFO:teuthology.orchestra.run.trial198.stdout:Running transaction check 2026-02-23T10:51:48.595 INFO:teuthology.orchestra.run.trial198.stdout:Transaction check succeeded. 2026-02-23T10:51:48.595 INFO:teuthology.orchestra.run.trial198.stdout:Running transaction test 2026-02-23T10:51:48.808 INFO:teuthology.orchestra.run.trial198.stdout:Transaction test succeeded. 2026-02-23T10:51:48.809 INFO:teuthology.orchestra.run.trial198.stdout:Running transaction 2026-02-23T10:51:49.037 INFO:teuthology.orchestra.run.trial198.stdout: Preparing : 1/1 2026-02-23T10:51:49.042 INFO:teuthology.orchestra.run.trial198.stdout: Installing : python3-xmltodict-0.12.0-15.el9.noarch 1/13 2026-02-23T10:51:49.054 INFO:teuthology.orchestra.run.trial198.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 2/13 2026-02-23T10:51:49.056 INFO:teuthology.orchestra.run.trial198.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 3/13 2026-02-23T10:51:49.084 INFO:teuthology.orchestra.run.trial198.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 4/13 2026-02-23T10:51:49.127 INFO:teuthology.orchestra.run.trial198.stdout: Installing : abseil-cpp-20211102.0-4.el9.x86_64 5/13 2026-02-23T10:51:49.136 INFO:teuthology.orchestra.run.trial198.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 6/13 2026-02-23T10:51:49.163 INFO:teuthology.orchestra.run.trial198.stdout: Installing : protobuf-compiler-3.14.0-17.el9.x86_64 7/13 2026-02-23T10:51:49.173 INFO:teuthology.orchestra.run.trial198.stdout: Installing : python3-protobuf-3.14.0-17.el9.noarch 8/13 2026-02-23T10:51:49.180 INFO:teuthology.orchestra.run.trial198.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 9/13 2026-02-23T10:51:49.183 INFO:teuthology.orchestra.run.trial198.stdout: Installing : python3-jmespath-1.0.1-1.el9.noarch 10/13 2026-02-23T10:51:49.192 INFO:teuthology.orchestra.run.trial198.stdout: Installing : ceph-prometheus-alerts-2:20.3.0-5439.ge371433f.el9 11/13 2026-02-23T10:51:52.072 INFO:teuthology.orchestra.run.trial198.stdout: Installing : ceph-grafana-dashboards-2:20.3.0-5439.ge371433f.el 12/13 2026-02-23T10:51:52.079 INFO:teuthology.orchestra.run.trial198.stdout: Installing : ceph-mgr-dashboard-2:20.3.0-5439.ge371433f.el9.noa 13/13 2026-02-23T10:51:52.360 INFO:teuthology.orchestra.run.trial198.stdout: Running scriptlet: ceph-mgr-dashboard-2:20.3.0-5439.ge371433f.el9.noa 13/13 2026-02-23T10:51:52.360 INFO:teuthology.orchestra.run.trial198.stdout: Verifying : ceph-grafana-dashboards-2:20.3.0-5439.ge371433f.el 1/13 2026-02-23T10:51:52.360 INFO:teuthology.orchestra.run.trial198.stdout: Verifying : ceph-mgr-dashboard-2:20.3.0-5439.ge371433f.el9.noa 2/13 2026-02-23T10:51:52.360 INFO:teuthology.orchestra.run.trial198.stdout: Verifying : ceph-prometheus-alerts-2:20.3.0-5439.ge371433f.el9 3/13 2026-02-23T10:51:52.360 INFO:teuthology.orchestra.run.trial198.stdout: Verifying : python3-jmespath-1.0.1-1.el9.noarch 4/13 2026-02-23T10:51:52.360 INFO:teuthology.orchestra.run.trial198.stdout: Verifying : python3-protobuf-3.14.0-17.el9.noarch 5/13 2026-02-23T10:51:52.360 INFO:teuthology.orchestra.run.trial198.stdout: Verifying : protobuf-compiler-3.14.0-17.el9.x86_64 6/13 2026-02-23T10:51:52.360 INFO:teuthology.orchestra.run.trial198.stdout: Verifying : abseil-cpp-20211102.0-4.el9.x86_64 7/13 2026-02-23T10:51:52.360 INFO:teuthology.orchestra.run.trial198.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 8/13 2026-02-23T10:51:52.360 INFO:teuthology.orchestra.run.trial198.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 9/13 2026-02-23T10:51:52.360 INFO:teuthology.orchestra.run.trial198.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 10/13 2026-02-23T10:51:52.360 INFO:teuthology.orchestra.run.trial198.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 11/13 2026-02-23T10:51:52.360 INFO:teuthology.orchestra.run.trial198.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 12/13 2026-02-23T10:51:52.416 INFO:teuthology.orchestra.run.trial198.stdout: Verifying : python3-xmltodict-0.12.0-15.el9.noarch 13/13 2026-02-23T10:51:52.416 INFO:teuthology.orchestra.run.trial198.stdout: 2026-02-23T10:51:52.416 INFO:teuthology.orchestra.run.trial198.stdout:Installed: 2026-02-23T10:51:52.416 INFO:teuthology.orchestra.run.trial198.stdout: abseil-cpp-20211102.0-4.el9.x86_64 2026-02-23T10:51:52.416 INFO:teuthology.orchestra.run.trial198.stdout: ceph-grafana-dashboards-2:20.3.0-5439.ge371433f.el9.noarch 2026-02-23T10:51:52.416 INFO:teuthology.orchestra.run.trial198.stdout: ceph-mgr-dashboard-2:20.3.0-5439.ge371433f.el9.noarch 2026-02-23T10:51:52.416 INFO:teuthology.orchestra.run.trial198.stdout: ceph-prometheus-alerts-2:20.3.0-5439.ge371433f.el9.noarch 2026-02-23T10:51:52.416 INFO:teuthology.orchestra.run.trial198.stdout: grpc-data-1.46.7-10.el9.noarch 2026-02-23T10:51:52.416 INFO:teuthology.orchestra.run.trial198.stdout: protobuf-compiler-3.14.0-17.el9.x86_64 2026-02-23T10:51:52.417 INFO:teuthology.orchestra.run.trial198.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2026-02-23T10:51:52.417 INFO:teuthology.orchestra.run.trial198.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2026-02-23T10:51:52.417 INFO:teuthology.orchestra.run.trial198.stdout: python3-jmespath-1.0.1-1.el9.noarch 2026-02-23T10:51:52.417 INFO:teuthology.orchestra.run.trial198.stdout: python3-protobuf-3.14.0-17.el9.noarch 2026-02-23T10:51:52.417 INFO:teuthology.orchestra.run.trial198.stdout: python3-repoze-lru-0.7-16.el9.noarch 2026-02-23T10:51:52.417 INFO:teuthology.orchestra.run.trial198.stdout: python3-routes-2.5.1-5.el9.noarch 2026-02-23T10:51:52.417 INFO:teuthology.orchestra.run.trial198.stdout: python3-xmltodict-0.12.0-15.el9.noarch 2026-02-23T10:51:52.417 INFO:teuthology.orchestra.run.trial198.stdout: 2026-02-23T10:51:52.417 INFO:teuthology.orchestra.run.trial198.stdout:Complete! 2026-02-23T10:51:52.454 DEBUG:teuthology.orchestra.run.trial198:> sudo yum -y install ceph-mgr-diskprediction-local 2026-02-23T10:51:52.769 INFO:teuthology.orchestra.run.trial198.stdout:Last metadata expiration check: 0:01:20 ago on Mon 23 Feb 2026 10:50:32 AM UTC. 2026-02-23T10:51:52.852 INFO:teuthology.orchestra.run.trial198.stdout:Dependencies resolved. 2026-02-23T10:51:52.853 INFO:teuthology.orchestra.run.trial198.stdout:======================================================================================= 2026-02-23T10:51:52.853 INFO:teuthology.orchestra.run.trial198.stdout: Package Arch Version Repository Size 2026-02-23T10:51:52.853 INFO:teuthology.orchestra.run.trial198.stdout:======================================================================================= 2026-02-23T10:51:52.853 INFO:teuthology.orchestra.run.trial198.stdout:Installing: 2026-02-23T10:51:52.853 INFO:teuthology.orchestra.run.trial198.stdout: ceph-mgr-diskprediction-local noarch 2:20.3.0-5439.ge371433f.el9 ceph-noarch 7.4 M 2026-02-23T10:51:52.853 INFO:teuthology.orchestra.run.trial198.stdout:Installing dependencies: 2026-02-23T10:51:52.853 INFO:teuthology.orchestra.run.trial198.stdout: flexiblas x86_64 3.0.4-9.el9 appstream 30 k 2026-02-23T10:51:52.853 INFO:teuthology.orchestra.run.trial198.stdout: flexiblas-netlib x86_64 3.0.4-9.el9 appstream 3.0 M 2026-02-23T10:51:52.854 INFO:teuthology.orchestra.run.trial198.stdout: flexiblas-openblas-openmp x86_64 3.0.4-9.el9 appstream 15 k 2026-02-23T10:51:52.854 INFO:teuthology.orchestra.run.trial198.stdout: libgfortran x86_64 11.5.0-14.el9 baseos 794 k 2026-02-23T10:51:52.854 INFO:teuthology.orchestra.run.trial198.stdout: libquadmath x86_64 11.5.0-14.el9 baseos 184 k 2026-02-23T10:51:52.854 INFO:teuthology.orchestra.run.trial198.stdout: openblas x86_64 0.3.29-1.el9 appstream 42 k 2026-02-23T10:51:52.854 INFO:teuthology.orchestra.run.trial198.stdout: openblas-openmp x86_64 0.3.29-1.el9 appstream 5.3 M 2026-02-23T10:51:52.854 INFO:teuthology.orchestra.run.trial198.stdout: python3-devel x86_64 3.9.25-3.el9 appstream 244 k 2026-02-23T10:51:52.854 INFO:teuthology.orchestra.run.trial198.stdout: python3-numpy x86_64 1:1.23.5-2.el9 appstream 6.1 M 2026-02-23T10:51:52.854 INFO:teuthology.orchestra.run.trial198.stdout: python3-numpy-f2py x86_64 1:1.23.5-2.el9 appstream 442 k 2026-02-23T10:51:52.854 INFO:teuthology.orchestra.run.trial198.stdout: python3-scipy x86_64 1.9.3-2.el9 appstream 19 M 2026-02-23T10:51:52.854 INFO:teuthology.orchestra.run.trial198.stdout: 2026-02-23T10:51:52.854 INFO:teuthology.orchestra.run.trial198.stdout:Transaction Summary 2026-02-23T10:51:52.854 INFO:teuthology.orchestra.run.trial198.stdout:======================================================================================= 2026-02-23T10:51:52.855 INFO:teuthology.orchestra.run.trial198.stdout:Install 12 Packages 2026-02-23T10:51:52.855 INFO:teuthology.orchestra.run.trial198.stdout: 2026-02-23T10:51:52.855 INFO:teuthology.orchestra.run.trial198.stdout:Total download size: 43 M 2026-02-23T10:51:52.855 INFO:teuthology.orchestra.run.trial198.stdout:Installed size: 234 M 2026-02-23T10:51:52.855 INFO:teuthology.orchestra.run.trial198.stdout:Downloading Packages: 2026-02-23T10:51:53.139 INFO:teuthology.orchestra.run.trial198.stdout:(1/12): libquadmath-11.5.0-14.el9.x86_64.rpm 1.2 MB/s | 184 kB 00:00 2026-02-23T10:51:53.191 INFO:teuthology.orchestra.run.trial198.stdout:(2/12): libgfortran-11.5.0-14.el9.x86_64.rpm 3.9 MB/s | 794 kB 00:00 2026-02-23T10:51:53.708 INFO:teuthology.orchestra.run.trial198.stdout:(3/12): flexiblas-3.0.4-9.el9.x86_64.rpm 52 kB/s | 30 kB 00:00 2026-02-23T10:51:54.189 INFO:teuthology.orchestra.run.trial198.stdout:(4/12): flexiblas-openblas-openmp-3.0.4-9.el9.x 31 kB/s | 15 kB 00:00 2026-02-23T10:51:54.234 INFO:teuthology.orchestra.run.trial198.stdout:(5/12): flexiblas-netlib-3.0.4-9.el9.x86_64.rpm 2.9 MB/s | 3.0 MB 00:01 2026-02-23T10:51:54.770 INFO:teuthology.orchestra.run.trial198.stdout:(6/12): openblas-0.3.29-1.el9.x86_64.rpm 73 kB/s | 42 kB 00:00 2026-02-23T10:51:55.346 INFO:teuthology.orchestra.run.trial198.stdout:(7/12): openblas-openmp-0.3.29-1.el9.x86_64.rpm 4.8 MB/s | 5.3 MB 00:01 2026-02-23T10:51:55.358 INFO:teuthology.orchestra.run.trial198.stdout:(8/12): ceph-mgr-diskprediction-local-20.3.0-54 3.1 MB/s | 7.4 MB 00:02 2026-02-23T10:51:55.532 INFO:teuthology.orchestra.run.trial198.stdout:(9/12): python3-devel-3.9.25-3.el9.x86_64.rpm 321 kB/s | 244 kB 00:00 2026-02-23T10:51:56.197 INFO:teuthology.orchestra.run.trial198.stdout:(10/12): python3-numpy-f2py-1.23.5-2.el9.x86_64 527 kB/s | 442 kB 00:00 2026-02-23T10:51:56.476 INFO:teuthology.orchestra.run.trial198.stdout:(11/12): python3-numpy-1.23.5-2.el9.x86_64.rpm 5.4 MB/s | 6.1 MB 00:01 2026-02-23T10:51:56.941 INFO:teuthology.orchestra.run.trial198.stdout:(12/12): python3-scipy-1.9.3-2.el9.x86_64.rpm 14 MB/s | 19 MB 00:01 2026-02-23T10:51:56.942 INFO:teuthology.orchestra.run.trial198.stdout:-------------------------------------------------------------------------------- 2026-02-23T10:51:56.942 INFO:teuthology.orchestra.run.trial198.stdout:Total 10 MB/s | 43 MB 00:04 2026-02-23T10:51:57.103 INFO:teuthology.orchestra.run.trial198.stdout:Running transaction check 2026-02-23T10:51:57.117 INFO:teuthology.orchestra.run.trial198.stdout:Transaction check succeeded. 2026-02-23T10:51:57.117 INFO:teuthology.orchestra.run.trial198.stdout:Running transaction test 2026-02-23T10:51:57.319 INFO:teuthology.orchestra.run.trial198.stdout:Transaction test succeeded. 2026-02-23T10:51:57.319 INFO:teuthology.orchestra.run.trial198.stdout:Running transaction 2026-02-23T10:51:57.536 INFO:teuthology.orchestra.run.trial198.stdout: Preparing : 1/1 2026-02-23T10:51:57.539 INFO:teuthology.orchestra.run.trial198.stdout: Installing : flexiblas-3.0.4-9.el9.x86_64 1/12 2026-02-23T10:51:57.545 INFO:teuthology.orchestra.run.trial198.stdout: Installing : libquadmath-11.5.0-14.el9.x86_64 2/12 2026-02-23T10:51:57.579 INFO:teuthology.orchestra.run.trial198.stdout: Installing : libgfortran-11.5.0-14.el9.x86_64 3/12 2026-02-23T10:51:57.584 INFO:teuthology.orchestra.run.trial198.stdout: Installing : python3-devel-3.9.25-3.el9.x86_64 4/12 2026-02-23T10:51:57.643 INFO:teuthology.orchestra.run.trial198.stdout: Installing : openblas-0.3.29-1.el9.x86_64 5/12 2026-02-23T10:51:57.646 INFO:teuthology.orchestra.run.trial198.stdout: Installing : openblas-openmp-0.3.29-1.el9.x86_64 6/12 2026-02-23T10:51:57.667 INFO:teuthology.orchestra.run.trial198.stdout: Installing : flexiblas-openblas-openmp-3.0.4-9.el9.x86_64 7/12 2026-02-23T10:51:58.000 INFO:teuthology.orchestra.run.trial198.stdout: Installing : flexiblas-netlib-3.0.4-9.el9.x86_64 8/12 2026-02-23T10:51:58.075 INFO:teuthology.orchestra.run.trial198.stdout: Installing : python3-numpy-1:1.23.5-2.el9.x86_64 9/12 2026-02-23T10:51:58.761 INFO:teuthology.orchestra.run.trial198.stdout: Installing : python3-numpy-f2py-1:1.23.5-2.el9.x86_64 10/12 2026-02-23T10:51:59.219 INFO:teuthology.orchestra.run.trial198.stdout: Installing : python3-scipy-1.9.3-2.el9.x86_64 11/12 2026-02-23T10:51:59.220 INFO:teuthology.orchestra.run.trial198.stdout: Installing : ceph-mgr-diskprediction-local-2:20.3.0-5439.ge3714 12/12 2026-02-23T10:51:59.490 INFO:teuthology.orchestra.run.trial198.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:20.3.0-5439.ge3714 12/12 2026-02-23T10:51:59.491 INFO:teuthology.orchestra.run.trial198.stdout: Verifying : ceph-mgr-diskprediction-local-2:20.3.0-5439.ge3714 1/12 2026-02-23T10:51:59.491 INFO:teuthology.orchestra.run.trial198.stdout: Verifying : libgfortran-11.5.0-14.el9.x86_64 2/12 2026-02-23T10:51:59.491 INFO:teuthology.orchestra.run.trial198.stdout: Verifying : libquadmath-11.5.0-14.el9.x86_64 3/12 2026-02-23T10:51:59.491 INFO:teuthology.orchestra.run.trial198.stdout: Verifying : flexiblas-3.0.4-9.el9.x86_64 4/12 2026-02-23T10:51:59.491 INFO:teuthology.orchestra.run.trial198.stdout: Verifying : flexiblas-netlib-3.0.4-9.el9.x86_64 5/12 2026-02-23T10:51:59.491 INFO:teuthology.orchestra.run.trial198.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-9.el9.x86_64 6/12 2026-02-23T10:51:59.491 INFO:teuthology.orchestra.run.trial198.stdout: Verifying : openblas-0.3.29-1.el9.x86_64 7/12 2026-02-23T10:51:59.491 INFO:teuthology.orchestra.run.trial198.stdout: Verifying : openblas-openmp-0.3.29-1.el9.x86_64 8/12 2026-02-23T10:51:59.491 INFO:teuthology.orchestra.run.trial198.stdout: Verifying : python3-devel-3.9.25-3.el9.x86_64 9/12 2026-02-23T10:51:59.491 INFO:teuthology.orchestra.run.trial198.stdout: Verifying : python3-numpy-1:1.23.5-2.el9.x86_64 10/12 2026-02-23T10:51:59.492 INFO:teuthology.orchestra.run.trial198.stdout: Verifying : python3-numpy-f2py-1:1.23.5-2.el9.x86_64 11/12 2026-02-23T10:51:59.556 INFO:teuthology.orchestra.run.trial198.stdout: Verifying : python3-scipy-1.9.3-2.el9.x86_64 12/12 2026-02-23T10:51:59.556 INFO:teuthology.orchestra.run.trial198.stdout: 2026-02-23T10:51:59.556 INFO:teuthology.orchestra.run.trial198.stdout:Installed: 2026-02-23T10:51:59.556 INFO:teuthology.orchestra.run.trial198.stdout: ceph-mgr-diskprediction-local-2:20.3.0-5439.ge371433f.el9.noarch 2026-02-23T10:51:59.556 INFO:teuthology.orchestra.run.trial198.stdout: flexiblas-3.0.4-9.el9.x86_64 2026-02-23T10:51:59.556 INFO:teuthology.orchestra.run.trial198.stdout: flexiblas-netlib-3.0.4-9.el9.x86_64 2026-02-23T10:51:59.556 INFO:teuthology.orchestra.run.trial198.stdout: flexiblas-openblas-openmp-3.0.4-9.el9.x86_64 2026-02-23T10:51:59.556 INFO:teuthology.orchestra.run.trial198.stdout: libgfortran-11.5.0-14.el9.x86_64 2026-02-23T10:51:59.556 INFO:teuthology.orchestra.run.trial198.stdout: libquadmath-11.5.0-14.el9.x86_64 2026-02-23T10:51:59.557 INFO:teuthology.orchestra.run.trial198.stdout: openblas-0.3.29-1.el9.x86_64 2026-02-23T10:51:59.557 INFO:teuthology.orchestra.run.trial198.stdout: openblas-openmp-0.3.29-1.el9.x86_64 2026-02-23T10:51:59.557 INFO:teuthology.orchestra.run.trial198.stdout: python3-devel-3.9.25-3.el9.x86_64 2026-02-23T10:51:59.557 INFO:teuthology.orchestra.run.trial198.stdout: python3-numpy-1:1.23.5-2.el9.x86_64 2026-02-23T10:51:59.557 INFO:teuthology.orchestra.run.trial198.stdout: python3-numpy-f2py-1:1.23.5-2.el9.x86_64 2026-02-23T10:51:59.557 INFO:teuthology.orchestra.run.trial198.stdout: python3-scipy-1.9.3-2.el9.x86_64 2026-02-23T10:51:59.557 INFO:teuthology.orchestra.run.trial198.stdout: 2026-02-23T10:51:59.557 INFO:teuthology.orchestra.run.trial198.stdout:Complete! 2026-02-23T10:51:59.594 DEBUG:teuthology.orchestra.run.trial198:> sudo yum -y install ceph-mgr-rook 2026-02-23T10:51:59.906 INFO:teuthology.orchestra.run.trial198.stdout:Last metadata expiration check: 0:01:27 ago on Mon 23 Feb 2026 10:50:32 AM UTC. 2026-02-23T10:51:59.989 INFO:teuthology.orchestra.run.trial198.stdout:Dependencies resolved. 2026-02-23T10:51:59.990 INFO:teuthology.orchestra.run.trial198.stdout:================================================================================ 2026-02-23T10:51:59.991 INFO:teuthology.orchestra.run.trial198.stdout: Package Arch Version Repository Size 2026-02-23T10:51:59.991 INFO:teuthology.orchestra.run.trial198.stdout:================================================================================ 2026-02-23T10:51:59.991 INFO:teuthology.orchestra.run.trial198.stdout:Installing: 2026-02-23T10:51:59.991 INFO:teuthology.orchestra.run.trial198.stdout: ceph-mgr-rook noarch 2:20.3.0-5439.ge371433f.el9 ceph-noarch 50 k 2026-02-23T10:51:59.991 INFO:teuthology.orchestra.run.trial198.stdout:Installing dependencies: 2026-02-23T10:51:59.991 INFO:teuthology.orchestra.run.trial198.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2026-02-23T10:51:59.991 INFO:teuthology.orchestra.run.trial198.stdout: python3-certifi noarch 2023.05.07-4.el9 epel 14 k 2026-02-23T10:51:59.991 INFO:teuthology.orchestra.run.trial198.stdout: python3-google-auth noarch 1:2.45.0-1.el9 epel 254 k 2026-02-23T10:51:59.991 INFO:teuthology.orchestra.run.trial198.stdout: python3-kubernetes noarch 1:26.1.0-3.el9 epel 1.0 M 2026-02-23T10:51:59.991 INFO:teuthology.orchestra.run.trial198.stdout: python3-pyasn1 noarch 0.4.8-6.el9 appstream 159 k 2026-02-23T10:51:59.991 INFO:teuthology.orchestra.run.trial198.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 appstream 279 k 2026-02-23T10:51:59.992 INFO:teuthology.orchestra.run.trial198.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 appstream 54 k 2026-02-23T10:51:59.992 INFO:teuthology.orchestra.run.trial198.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2026-02-23T10:51:59.992 INFO:teuthology.orchestra.run.trial198.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2026-02-23T10:51:59.992 INFO:teuthology.orchestra.run.trial198.stdout: 2026-02-23T10:51:59.992 INFO:teuthology.orchestra.run.trial198.stdout:Transaction Summary 2026-02-23T10:51:59.992 INFO:teuthology.orchestra.run.trial198.stdout:================================================================================ 2026-02-23T10:51:59.992 INFO:teuthology.orchestra.run.trial198.stdout:Install 10 Packages 2026-02-23T10:51:59.992 INFO:teuthology.orchestra.run.trial198.stdout: 2026-02-23T10:51:59.992 INFO:teuthology.orchestra.run.trial198.stdout:Total download size: 2.0 M 2026-02-23T10:51:59.992 INFO:teuthology.orchestra.run.trial198.stdout:Installed size: 25 M 2026-02-23T10:51:59.992 INFO:teuthology.orchestra.run.trial198.stdout:Downloading Packages: 2026-02-23T10:52:00.566 INFO:teuthology.orchestra.run.trial198.stdout:(1/10): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.1 MB/s | 159 kB 00:00 2026-02-23T10:52:00.585 INFO:teuthology.orchestra.run.trial198.stdout:(2/10): python3-pyasn1-modules-0.4.8-6.el9.noar 1.6 MB/s | 279 kB 00:00 2026-02-23T10:52:00.596 INFO:teuthology.orchestra.run.trial198.stdout:(3/10): python3-requests-oauthlib-1.3.0-12.el9. 1.7 MB/s | 54 kB 00:00 2026-02-23T10:52:00.766 INFO:teuthology.orchestra.run.trial198.stdout:(4/10): python3-certifi-2023.05.07-4.el9.noarch 83 kB/s | 14 kB 00:00 2026-02-23T10:52:00.775 INFO:teuthology.orchestra.run.trial198.stdout:(5/10): python3-cachetools-4.2.4-1.el9.noarch.r 169 kB/s | 32 kB 00:00 2026-02-23T10:52:00.822 INFO:teuthology.orchestra.run.trial198.stdout:(6/10): ceph-mgr-rook-20.3.0-5439.ge371433f.el9 125 kB/s | 50 kB 00:00 2026-02-23T10:52:00.950 INFO:teuthology.orchestra.run.trial198.stdout:(7/10): python3-google-auth-2.45.0-1.el9.noarch 1.3 MB/s | 254 kB 00:00 2026-02-23T10:52:01.002 INFO:teuthology.orchestra.run.trial198.stdout:(8/10): python3-websocket-client-1.2.3-2.el9.no 1.7 MB/s | 90 kB 00:00 2026-02-23T10:52:01.025 INFO:teuthology.orchestra.run.trial198.stdout:(9/10): python3-kubernetes-26.1.0-3.el9.noarch. 4.1 MB/s | 1.0 MB 00:00 2026-02-23T10:52:01.041 INFO:teuthology.orchestra.run.trial198.stdout:(10/10): python3-rsa-4.9-2.el9.noarch.rpm 271 kB/s | 59 kB 00:00 2026-02-23T10:52:01.042 INFO:teuthology.orchestra.run.trial198.stdout:-------------------------------------------------------------------------------- 2026-02-23T10:52:01.042 INFO:teuthology.orchestra.run.trial198.stdout:Total 1.9 MB/s | 2.0 MB 00:01 2026-02-23T10:52:01.069 INFO:teuthology.orchestra.run.trial198.stdout:Running transaction check 2026-02-23T10:52:01.078 INFO:teuthology.orchestra.run.trial198.stdout:Transaction check succeeded. 2026-02-23T10:52:01.078 INFO:teuthology.orchestra.run.trial198.stdout:Running transaction test 2026-02-23T10:52:01.118 INFO:teuthology.orchestra.run.trial198.stdout:Transaction test succeeded. 2026-02-23T10:52:01.118 INFO:teuthology.orchestra.run.trial198.stdout:Running transaction 2026-02-23T10:52:01.192 INFO:teuthology.orchestra.run.trial198.stdout: Preparing : 1/1 2026-02-23T10:52:01.242 INFO:teuthology.orchestra.run.trial198.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/10 2026-02-23T10:52:01.254 INFO:teuthology.orchestra.run.trial198.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/10 2026-02-23T10:52:01.268 INFO:teuthology.orchestra.run.trial198.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/10 2026-02-23T10:52:01.273 INFO:teuthology.orchestra.run.trial198.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/10 2026-02-23T10:52:01.281 INFO:teuthology.orchestra.run.trial198.stdout: Installing : python3-certifi-2023.05.07-4.el9.noarch 5/10 2026-02-23T10:52:01.320 INFO:teuthology.orchestra.run.trial198.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/10 2026-02-23T10:52:01.332 INFO:teuthology.orchestra.run.trial198.stdout: Installing : python3-google-auth-1:2.45.0-1.el9.noarch 7/10 2026-02-23T10:52:01.649 INFO:teuthology.orchestra.run.trial198.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 8/10 2026-02-23T10:52:01.664 INFO:teuthology.orchestra.run.trial198.stdout: Installing : python3-kubernetes-1:26.1.0-3.el9.noarch 9/10 2026-02-23T10:52:01.665 INFO:teuthology.orchestra.run.trial198.stdout: Installing : ceph-mgr-rook-2:20.3.0-5439.ge371433f.el9.noarch 10/10 2026-02-23T10:52:01.774 INFO:teuthology.orchestra.run.trial198.stdout: Running scriptlet: ceph-mgr-rook-2:20.3.0-5439.ge371433f.el9.noarch 10/10 2026-02-23T10:52:01.774 INFO:teuthology.orchestra.run.trial198.stdout: Verifying : ceph-mgr-rook-2:20.3.0-5439.ge371433f.el9.noarch 1/10 2026-02-23T10:52:01.774 INFO:teuthology.orchestra.run.trial198.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 2/10 2026-02-23T10:52:01.774 INFO:teuthology.orchestra.run.trial198.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 3/10 2026-02-23T10:52:01.774 INFO:teuthology.orchestra.run.trial198.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 4/10 2026-02-23T10:52:01.774 INFO:teuthology.orchestra.run.trial198.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 5/10 2026-02-23T10:52:01.775 INFO:teuthology.orchestra.run.trial198.stdout: Verifying : python3-certifi-2023.05.07-4.el9.noarch 6/10 2026-02-23T10:52:01.775 INFO:teuthology.orchestra.run.trial198.stdout: Verifying : python3-google-auth-1:2.45.0-1.el9.noarch 7/10 2026-02-23T10:52:01.775 INFO:teuthology.orchestra.run.trial198.stdout: Verifying : python3-kubernetes-1:26.1.0-3.el9.noarch 8/10 2026-02-23T10:52:01.775 INFO:teuthology.orchestra.run.trial198.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 9/10 2026-02-23T10:52:01.831 INFO:teuthology.orchestra.run.trial198.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 10/10 2026-02-23T10:52:01.831 INFO:teuthology.orchestra.run.trial198.stdout: 2026-02-23T10:52:01.831 INFO:teuthology.orchestra.run.trial198.stdout:Installed: 2026-02-23T10:52:01.832 INFO:teuthology.orchestra.run.trial198.stdout: ceph-mgr-rook-2:20.3.0-5439.ge371433f.el9.noarch 2026-02-23T10:52:01.832 INFO:teuthology.orchestra.run.trial198.stdout: python3-cachetools-4.2.4-1.el9.noarch 2026-02-23T10:52:01.832 INFO:teuthology.orchestra.run.trial198.stdout: python3-certifi-2023.05.07-4.el9.noarch 2026-02-23T10:52:01.832 INFO:teuthology.orchestra.run.trial198.stdout: python3-google-auth-1:2.45.0-1.el9.noarch 2026-02-23T10:52:01.832 INFO:teuthology.orchestra.run.trial198.stdout: python3-kubernetes-1:26.1.0-3.el9.noarch 2026-02-23T10:52:01.832 INFO:teuthology.orchestra.run.trial198.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2026-02-23T10:52:01.832 INFO:teuthology.orchestra.run.trial198.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2026-02-23T10:52:01.832 INFO:teuthology.orchestra.run.trial198.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2026-02-23T10:52:01.832 INFO:teuthology.orchestra.run.trial198.stdout: python3-rsa-4.9-2.el9.noarch 2026-02-23T10:52:01.832 INFO:teuthology.orchestra.run.trial198.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2026-02-23T10:52:01.832 INFO:teuthology.orchestra.run.trial198.stdout: 2026-02-23T10:52:01.832 INFO:teuthology.orchestra.run.trial198.stdout:Complete! 2026-02-23T10:52:01.861 DEBUG:teuthology.orchestra.run.trial198:> sudo yum -y install ceph-mgr-cephadm 2026-02-23T10:52:02.174 INFO:teuthology.orchestra.run.trial198.stdout:Last metadata expiration check: 0:01:30 ago on Mon 23 Feb 2026 10:50:32 AM UTC. 2026-02-23T10:52:02.256 INFO:teuthology.orchestra.run.trial198.stdout:Dependencies resolved. 2026-02-23T10:52:02.256 INFO:teuthology.orchestra.run.trial198.stdout:================================================================================ 2026-02-23T10:52:02.256 INFO:teuthology.orchestra.run.trial198.stdout: Package Arch Version Repository Size 2026-02-23T10:52:02.256 INFO:teuthology.orchestra.run.trial198.stdout:================================================================================ 2026-02-23T10:52:02.256 INFO:teuthology.orchestra.run.trial198.stdout:Installing: 2026-02-23T10:52:02.256 INFO:teuthology.orchestra.run.trial198.stdout: ceph-mgr-cephadm noarch 2:20.3.0-5439.ge371433f.el9 ceph-noarch 184 k 2026-02-23T10:52:02.256 INFO:teuthology.orchestra.run.trial198.stdout:Installing dependencies: 2026-02-23T10:52:02.256 INFO:teuthology.orchestra.run.trial198.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2026-02-23T10:52:02.256 INFO:teuthology.orchestra.run.trial198.stdout: python3-babel noarch 2.9.1-2.el9 appstream 6.0 M 2026-02-23T10:52:02.256 INFO:teuthology.orchestra.run.trial198.stdout: python3-jinja2 noarch 2.11.3-8.el9 appstream 249 k 2026-02-23T10:52:02.256 INFO:teuthology.orchestra.run.trial198.stdout: python3-markupsafe x86_64 1.1.1-12.el9 appstream 35 k 2026-02-23T10:52:02.256 INFO:teuthology.orchestra.run.trial198.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2026-02-23T10:52:02.257 INFO:teuthology.orchestra.run.trial198.stdout: python3-typing-extensions noarch 4.15.0-1.el9 epel 86 k 2026-02-23T10:52:02.257 INFO:teuthology.orchestra.run.trial198.stdout: 2026-02-23T10:52:02.257 INFO:teuthology.orchestra.run.trial198.stdout:Transaction Summary 2026-02-23T10:52:02.257 INFO:teuthology.orchestra.run.trial198.stdout:================================================================================ 2026-02-23T10:52:02.257 INFO:teuthology.orchestra.run.trial198.stdout:Install 7 Packages 2026-02-23T10:52:02.257 INFO:teuthology.orchestra.run.trial198.stdout: 2026-02-23T10:52:02.257 INFO:teuthology.orchestra.run.trial198.stdout:Total download size: 7.1 M 2026-02-23T10:52:02.257 INFO:teuthology.orchestra.run.trial198.stdout:Installed size: 34 M 2026-02-23T10:52:02.257 INFO:teuthology.orchestra.run.trial198.stdout:Downloading Packages: 2026-02-23T10:52:03.110 INFO:teuthology.orchestra.run.trial198.stdout:(1/7): ceph-mgr-cephadm-20.3.0-5439.ge371433f.e 434 kB/s | 184 kB 00:00 2026-02-23T10:52:03.453 INFO:teuthology.orchestra.run.trial198.stdout:(2/7): python3-jinja2-2.11.3-8.el9.noarch.rpm 324 kB/s | 249 kB 00:00 2026-02-23T10:52:03.585 INFO:teuthology.orchestra.run.trial198.stdout:(3/7): python3-asyncssh-2.13.2-5.el9.noarch.rpm 4.1 MB/s | 548 kB 00:00 2026-02-23T10:52:03.604 INFO:teuthology.orchestra.run.trial198.stdout:(4/7): python3-natsort-7.1.1-5.el9.noarch.rpm 3.0 MB/s | 58 kB 00:00 2026-02-23T10:52:03.623 INFO:teuthology.orchestra.run.trial198.stdout:(5/7): python3-typing-extensions-4.15.0-1.el9.n 4.6 MB/s | 86 kB 00:00 2026-02-23T10:52:03.678 INFO:teuthology.orchestra.run.trial198.stdout:(6/7): python3-markupsafe-1.1.1-12.el9.x86_64.r 61 kB/s | 35 kB 00:00 2026-02-23T10:52:03.819 INFO:teuthology.orchestra.run.trial198.stdout:(7/7): python3-babel-2.9.1-2.el9.noarch.rpm 5.3 MB/s | 6.0 MB 00:01 2026-02-23T10:52:03.820 INFO:teuthology.orchestra.run.trial198.stdout:-------------------------------------------------------------------------------- 2026-02-23T10:52:03.820 INFO:teuthology.orchestra.run.trial198.stdout:Total 4.5 MB/s | 7.1 MB 00:01 2026-02-23T10:52:03.858 INFO:teuthology.orchestra.run.trial198.stdout:Running transaction check 2026-02-23T10:52:03.865 INFO:teuthology.orchestra.run.trial198.stdout:Transaction check succeeded. 2026-02-23T10:52:03.865 INFO:teuthology.orchestra.run.trial198.stdout:Running transaction test 2026-02-23T10:52:03.939 INFO:teuthology.orchestra.run.trial198.stdout:Transaction test succeeded. 2026-02-23T10:52:03.939 INFO:teuthology.orchestra.run.trial198.stdout:Running transaction 2026-02-23T10:52:04.028 INFO:teuthology.orchestra.run.trial198.stdout: Preparing : 1/1 2026-02-23T10:52:04.076 INFO:teuthology.orchestra.run.trial198.stdout: Installing : python3-typing-extensions-4.15.0-1.el9.noarch 1/7 2026-02-23T10:52:04.086 INFO:teuthology.orchestra.run.trial198.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 2/7 2026-02-23T10:52:04.091 INFO:teuthology.orchestra.run.trial198.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/7 2026-02-23T10:52:04.308 INFO:teuthology.orchestra.run.trial198.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 4/7 2026-02-23T10:52:04.331 INFO:teuthology.orchestra.run.trial198.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 5/7 2026-02-23T10:52:04.355 INFO:teuthology.orchestra.run.trial198.stdout: Installing : python3-jinja2-2.11.3-8.el9.noarch 6/7 2026-02-23T10:52:04.357 INFO:teuthology.orchestra.run.trial198.stdout: Installing : ceph-mgr-cephadm-2:20.3.0-5439.ge371433f.el9.noarch 7/7 2026-02-23T10:52:04.467 INFO:teuthology.orchestra.run.trial198.stdout: Running scriptlet: ceph-mgr-cephadm-2:20.3.0-5439.ge371433f.el9.noarch 7/7 2026-02-23T10:52:04.467 INFO:teuthology.orchestra.run.trial198.stdout: Verifying : ceph-mgr-cephadm-2:20.3.0-5439.ge371433f.el9.noarch 1/7 2026-02-23T10:52:04.467 INFO:teuthology.orchestra.run.trial198.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/7 2026-02-23T10:52:04.467 INFO:teuthology.orchestra.run.trial198.stdout: Verifying : python3-jinja2-2.11.3-8.el9.noarch 3/7 2026-02-23T10:52:04.467 INFO:teuthology.orchestra.run.trial198.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 4/7 2026-02-23T10:52:04.467 INFO:teuthology.orchestra.run.trial198.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 5/7 2026-02-23T10:52:04.467 INFO:teuthology.orchestra.run.trial198.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 6/7 2026-02-23T10:52:04.525 INFO:teuthology.orchestra.run.trial198.stdout: Verifying : python3-typing-extensions-4.15.0-1.el9.noarch 7/7 2026-02-23T10:52:04.525 INFO:teuthology.orchestra.run.trial198.stdout: 2026-02-23T10:52:04.525 INFO:teuthology.orchestra.run.trial198.stdout:Installed: 2026-02-23T10:52:04.525 INFO:teuthology.orchestra.run.trial198.stdout: ceph-mgr-cephadm-2:20.3.0-5439.ge371433f.el9.noarch 2026-02-23T10:52:04.525 INFO:teuthology.orchestra.run.trial198.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2026-02-23T10:52:04.525 INFO:teuthology.orchestra.run.trial198.stdout: python3-babel-2.9.1-2.el9.noarch 2026-02-23T10:52:04.525 INFO:teuthology.orchestra.run.trial198.stdout: python3-jinja2-2.11.3-8.el9.noarch 2026-02-23T10:52:04.525 INFO:teuthology.orchestra.run.trial198.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2026-02-23T10:52:04.525 INFO:teuthology.orchestra.run.trial198.stdout: python3-natsort-7.1.1-5.el9.noarch 2026-02-23T10:52:04.525 INFO:teuthology.orchestra.run.trial198.stdout: python3-typing-extensions-4.15.0-1.el9.noarch 2026-02-23T10:52:04.525 INFO:teuthology.orchestra.run.trial198.stdout: 2026-02-23T10:52:04.525 INFO:teuthology.orchestra.run.trial198.stdout:Complete! 2026-02-23T10:52:04.555 DEBUG:teuthology.orchestra.run.trial198:> sudo yum -y install ceph-osd 2026-02-23T10:52:04.857 INFO:teuthology.orchestra.run.trial198.stdout:Last metadata expiration check: 0:01:32 ago on Mon 23 Feb 2026 10:50:32 AM UTC. 2026-02-23T10:52:04.922 INFO:teuthology.orchestra.run.trial198.stdout:Package ceph-osd-2:20.3.0-5439.ge371433f.el9.x86_64 is already installed. 2026-02-23T10:52:04.941 INFO:teuthology.orchestra.run.trial198.stdout:Dependencies resolved. 2026-02-23T10:52:04.942 INFO:teuthology.orchestra.run.trial198.stdout:Nothing to do. 2026-02-23T10:52:04.942 INFO:teuthology.orchestra.run.trial198.stdout:Complete! 2026-02-23T10:52:04.963 DEBUG:teuthology.orchestra.run.trial198:> sudo yum -y install ceph-osd-classic 2026-02-23T10:52:05.268 INFO:teuthology.orchestra.run.trial198.stdout:Last metadata expiration check: 0:01:33 ago on Mon 23 Feb 2026 10:50:32 AM UTC. 2026-02-23T10:52:05.335 INFO:teuthology.orchestra.run.trial198.stdout:Package ceph-osd-classic-2:20.3.0-5439.ge371433f.el9.x86_64 is already installed. 2026-02-23T10:52:05.353 INFO:teuthology.orchestra.run.trial198.stdout:Dependencies resolved. 2026-02-23T10:52:05.353 INFO:teuthology.orchestra.run.trial198.stdout:Nothing to do. 2026-02-23T10:52:05.354 INFO:teuthology.orchestra.run.trial198.stdout:Complete! 2026-02-23T10:52:05.375 DEBUG:teuthology.orchestra.run.trial198:> sudo yum -y install ceph-fuse 2026-02-23T10:52:05.683 INFO:teuthology.orchestra.run.trial198.stdout:Last metadata expiration check: 0:01:33 ago on Mon 23 Feb 2026 10:50:32 AM UTC. 2026-02-23T10:52:05.766 INFO:teuthology.orchestra.run.trial198.stdout:Dependencies resolved. 2026-02-23T10:52:05.766 INFO:teuthology.orchestra.run.trial198.stdout:================================================================================ 2026-02-23T10:52:05.767 INFO:teuthology.orchestra.run.trial198.stdout: Package Arch Version Repo Size 2026-02-23T10:52:05.767 INFO:teuthology.orchestra.run.trial198.stdout:================================================================================ 2026-02-23T10:52:05.767 INFO:teuthology.orchestra.run.trial198.stdout:Installing: 2026-02-23T10:52:05.767 INFO:teuthology.orchestra.run.trial198.stdout: ceph-fuse x86_64 2:20.3.0-5439.ge371433f.el9 ceph 914 k 2026-02-23T10:52:05.767 INFO:teuthology.orchestra.run.trial198.stdout: 2026-02-23T10:52:05.767 INFO:teuthology.orchestra.run.trial198.stdout:Transaction Summary 2026-02-23T10:52:05.767 INFO:teuthology.orchestra.run.trial198.stdout:================================================================================ 2026-02-23T10:52:05.767 INFO:teuthology.orchestra.run.trial198.stdout:Install 1 Package 2026-02-23T10:52:05.767 INFO:teuthology.orchestra.run.trial198.stdout: 2026-02-23T10:52:05.767 INFO:teuthology.orchestra.run.trial198.stdout:Total download size: 914 k 2026-02-23T10:52:05.768 INFO:teuthology.orchestra.run.trial198.stdout:Installed size: 2.9 M 2026-02-23T10:52:05.768 INFO:teuthology.orchestra.run.trial198.stdout:Downloading Packages: 2026-02-23T10:52:05.982 INFO:teuthology.orchestra.run.trial198.stdout:ceph-fuse-20.3.0-5439.ge371433f.el9.x86_64.rpm 4.2 MB/s | 914 kB 00:00 2026-02-23T10:52:05.982 INFO:teuthology.orchestra.run.trial198.stdout:-------------------------------------------------------------------------------- 2026-02-23T10:52:05.982 INFO:teuthology.orchestra.run.trial198.stdout:Total 4.1 MB/s | 914 kB 00:00 2026-02-23T10:52:05.982 INFO:teuthology.orchestra.run.trial198.stdout:Running transaction check 2026-02-23T10:52:05.987 INFO:teuthology.orchestra.run.trial198.stdout:Transaction check succeeded. 2026-02-23T10:52:05.987 INFO:teuthology.orchestra.run.trial198.stdout:Running transaction test 2026-02-23T10:52:06.020 INFO:teuthology.orchestra.run.trial198.stdout:Transaction test succeeded. 2026-02-23T10:52:06.020 INFO:teuthology.orchestra.run.trial198.stdout:Running transaction 2026-02-23T10:52:06.110 INFO:teuthology.orchestra.run.trial198.stdout: Preparing : 1/1 2026-02-23T10:52:06.127 INFO:teuthology.orchestra.run.trial198.stdout: Installing : ceph-fuse-2:20.3.0-5439.ge371433f.el9.x86_64 1/1 2026-02-23T10:52:06.339 INFO:teuthology.orchestra.run.trial198.stdout: Running scriptlet: ceph-fuse-2:20.3.0-5439.ge371433f.el9.x86_64 1/1 2026-02-23T10:52:06.396 INFO:teuthology.orchestra.run.trial198.stdout: Verifying : ceph-fuse-2:20.3.0-5439.ge371433f.el9.x86_64 1/1 2026-02-23T10:52:06.397 INFO:teuthology.orchestra.run.trial198.stdout: 2026-02-23T10:52:06.397 INFO:teuthology.orchestra.run.trial198.stdout:Installed: 2026-02-23T10:52:06.397 INFO:teuthology.orchestra.run.trial198.stdout: ceph-fuse-2:20.3.0-5439.ge371433f.el9.x86_64 2026-02-23T10:52:06.397 INFO:teuthology.orchestra.run.trial198.stdout: 2026-02-23T10:52:06.397 INFO:teuthology.orchestra.run.trial198.stdout:Complete! 2026-02-23T10:52:06.425 DEBUG:teuthology.orchestra.run.trial198:> sudo yum -y install ceph-volume 2026-02-23T10:52:06.740 INFO:teuthology.orchestra.run.trial198.stdout:Last metadata expiration check: 0:01:34 ago on Mon 23 Feb 2026 10:50:32 AM UTC. 2026-02-23T10:52:06.822 INFO:teuthology.orchestra.run.trial198.stdout:Dependencies resolved. 2026-02-23T10:52:06.823 INFO:teuthology.orchestra.run.trial198.stdout:================================================================================ 2026-02-23T10:52:06.823 INFO:teuthology.orchestra.run.trial198.stdout: Package Arch Version Repository Size 2026-02-23T10:52:06.823 INFO:teuthology.orchestra.run.trial198.stdout:================================================================================ 2026-02-23T10:52:06.823 INFO:teuthology.orchestra.run.trial198.stdout:Installing: 2026-02-23T10:52:06.823 INFO:teuthology.orchestra.run.trial198.stdout: ceph-volume noarch 2:20.3.0-5439.ge371433f.el9 ceph-noarch 301 k 2026-02-23T10:52:06.824 INFO:teuthology.orchestra.run.trial198.stdout:Installing dependencies: 2026-02-23T10:52:06.824 INFO:teuthology.orchestra.run.trial198.stdout: cryptsetup x86_64 2.8.1-3.el9 baseos 351 k 2026-02-23T10:52:06.824 INFO:teuthology.orchestra.run.trial198.stdout: 2026-02-23T10:52:06.824 INFO:teuthology.orchestra.run.trial198.stdout:Transaction Summary 2026-02-23T10:52:06.824 INFO:teuthology.orchestra.run.trial198.stdout:================================================================================ 2026-02-23T10:52:06.824 INFO:teuthology.orchestra.run.trial198.stdout:Install 2 Packages 2026-02-23T10:52:06.824 INFO:teuthology.orchestra.run.trial198.stdout: 2026-02-23T10:52:06.824 INFO:teuthology.orchestra.run.trial198.stdout:Total download size: 652 k 2026-02-23T10:52:06.824 INFO:teuthology.orchestra.run.trial198.stdout:Installed size: 2.2 M 2026-02-23T10:52:06.824 INFO:teuthology.orchestra.run.trial198.stdout:Downloading Packages: 2026-02-23T10:52:06.948 INFO:teuthology.orchestra.run.trial198.stdout:(1/2): cryptsetup-2.8.1-3.el9.x86_64.rpm 6.0 MB/s | 351 kB 00:00 2026-02-23T10:52:07.332 INFO:teuthology.orchestra.run.trial198.stdout:(2/2): ceph-volume-20.3.0-5439.ge371433f.el9.no 683 kB/s | 301 kB 00:00 2026-02-23T10:52:07.333 INFO:teuthology.orchestra.run.trial198.stdout:-------------------------------------------------------------------------------- 2026-02-23T10:52:07.333 INFO:teuthology.orchestra.run.trial198.stdout:Total 1.2 MB/s | 652 kB 00:00 2026-02-23T10:52:07.343 INFO:teuthology.orchestra.run.trial198.stdout:Running transaction check 2026-02-23T10:52:07.350 INFO:teuthology.orchestra.run.trial198.stdout:Transaction check succeeded. 2026-02-23T10:52:07.350 INFO:teuthology.orchestra.run.trial198.stdout:Running transaction test 2026-02-23T10:52:07.394 INFO:teuthology.orchestra.run.trial198.stdout:Transaction test succeeded. 2026-02-23T10:52:07.394 INFO:teuthology.orchestra.run.trial198.stdout:Running transaction 2026-02-23T10:52:07.475 INFO:teuthology.orchestra.run.trial198.stdout: Preparing : 1/1 2026-02-23T10:52:07.538 INFO:teuthology.orchestra.run.trial198.stdout: Installing : cryptsetup-2.8.1-3.el9.x86_64 1/2 2026-02-23T10:52:07.540 INFO:teuthology.orchestra.run.trial198.stdout: Installing : ceph-volume-2:20.3.0-5439.ge371433f.el9.noarch 2/2 2026-02-23T10:52:07.555 INFO:teuthology.orchestra.run.trial198.stdout: Running scriptlet: ceph-volume-2:20.3.0-5439.ge371433f.el9.noarch 2/2 2026-02-23T10:52:07.555 INFO:teuthology.orchestra.run.trial198.stdout:Glob pattern passed to enable, but globs are not supported for this. 2026-02-23T10:52:07.555 INFO:teuthology.orchestra.run.trial198.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2026-02-23T10:52:07.555 INFO:teuthology.orchestra.run.trial198.stdout: 2026-02-23T10:52:07.775 INFO:teuthology.orchestra.run.trial198.stdout: Verifying : ceph-volume-2:20.3.0-5439.ge371433f.el9.noarch 1/2 2026-02-23T10:52:07.834 INFO:teuthology.orchestra.run.trial198.stdout: Verifying : cryptsetup-2.8.1-3.el9.x86_64 2/2 2026-02-23T10:52:07.834 INFO:teuthology.orchestra.run.trial198.stdout: 2026-02-23T10:52:07.834 INFO:teuthology.orchestra.run.trial198.stdout:Installed: 2026-02-23T10:52:07.835 INFO:teuthology.orchestra.run.trial198.stdout: ceph-volume-2:20.3.0-5439.ge371433f.el9.noarch cryptsetup-2.8.1-3.el9.x86_64 2026-02-23T10:52:07.835 INFO:teuthology.orchestra.run.trial198.stdout: 2026-02-23T10:52:07.835 INFO:teuthology.orchestra.run.trial198.stdout:Complete! 2026-02-23T10:52:07.864 DEBUG:teuthology.orchestra.run.trial198:> sudo yum -y install librados-devel 2026-02-23T10:52:08.177 INFO:teuthology.orchestra.run.trial198.stdout:Last metadata expiration check: 0:01:36 ago on Mon 23 Feb 2026 10:50:32 AM UTC. 2026-02-23T10:52:08.261 INFO:teuthology.orchestra.run.trial198.stdout:Dependencies resolved. 2026-02-23T10:52:08.262 INFO:teuthology.orchestra.run.trial198.stdout:================================================================================ 2026-02-23T10:52:08.262 INFO:teuthology.orchestra.run.trial198.stdout: Package Arch Version Repo Size 2026-02-23T10:52:08.262 INFO:teuthology.orchestra.run.trial198.stdout:================================================================================ 2026-02-23T10:52:08.262 INFO:teuthology.orchestra.run.trial198.stdout:Installing: 2026-02-23T10:52:08.262 INFO:teuthology.orchestra.run.trial198.stdout: librados-devel x86_64 2:20.3.0-5439.ge371433f.el9 ceph 124 k 2026-02-23T10:52:08.262 INFO:teuthology.orchestra.run.trial198.stdout: 2026-02-23T10:52:08.262 INFO:teuthology.orchestra.run.trial198.stdout:Transaction Summary 2026-02-23T10:52:08.262 INFO:teuthology.orchestra.run.trial198.stdout:================================================================================ 2026-02-23T10:52:08.262 INFO:teuthology.orchestra.run.trial198.stdout:Install 1 Package 2026-02-23T10:52:08.262 INFO:teuthology.orchestra.run.trial198.stdout: 2026-02-23T10:52:08.262 INFO:teuthology.orchestra.run.trial198.stdout:Total download size: 124 k 2026-02-23T10:52:08.263 INFO:teuthology.orchestra.run.trial198.stdout:Installed size: 434 k 2026-02-23T10:52:08.263 INFO:teuthology.orchestra.run.trial198.stdout:Downloading Packages: 2026-02-23T10:52:08.347 INFO:teuthology.orchestra.run.trial198.stdout:librados-devel-20.3.0-5439.ge371433f.el9.x86_64 1.4 MB/s | 124 kB 00:00 2026-02-23T10:52:08.347 INFO:teuthology.orchestra.run.trial198.stdout:-------------------------------------------------------------------------------- 2026-02-23T10:52:08.347 INFO:teuthology.orchestra.run.trial198.stdout:Total 1.4 MB/s | 124 kB 00:00 2026-02-23T10:52:08.347 INFO:teuthology.orchestra.run.trial198.stdout:Running transaction check 2026-02-23T10:52:08.350 INFO:teuthology.orchestra.run.trial198.stdout:Transaction check succeeded. 2026-02-23T10:52:08.350 INFO:teuthology.orchestra.run.trial198.stdout:Running transaction test 2026-02-23T10:52:08.381 INFO:teuthology.orchestra.run.trial198.stdout:Transaction test succeeded. 2026-02-23T10:52:08.381 INFO:teuthology.orchestra.run.trial198.stdout:Running transaction 2026-02-23T10:52:08.434 INFO:teuthology.orchestra.run.trial198.stdout: Preparing : 1/1 2026-02-23T10:52:08.450 INFO:teuthology.orchestra.run.trial198.stdout: Installing : librados-devel-2:20.3.0-5439.ge371433f.el9.x86_64 1/1 2026-02-23T10:52:08.531 INFO:teuthology.orchestra.run.trial198.stdout: Running scriptlet: librados-devel-2:20.3.0-5439.ge371433f.el9.x86_64 1/1 2026-02-23T10:52:08.589 INFO:teuthology.orchestra.run.trial198.stdout: Verifying : librados-devel-2:20.3.0-5439.ge371433f.el9.x86_64 1/1 2026-02-23T10:52:08.589 INFO:teuthology.orchestra.run.trial198.stdout: 2026-02-23T10:52:08.590 INFO:teuthology.orchestra.run.trial198.stdout:Installed: 2026-02-23T10:52:08.590 INFO:teuthology.orchestra.run.trial198.stdout: librados-devel-2:20.3.0-5439.ge371433f.el9.x86_64 2026-02-23T10:52:08.590 INFO:teuthology.orchestra.run.trial198.stdout: 2026-02-23T10:52:08.590 INFO:teuthology.orchestra.run.trial198.stdout:Complete! 2026-02-23T10:52:08.620 DEBUG:teuthology.orchestra.run.trial198:> sudo yum -y install libcephfs2 2026-02-23T10:52:08.930 INFO:teuthology.orchestra.run.trial198.stdout:Last metadata expiration check: 0:01:36 ago on Mon 23 Feb 2026 10:50:32 AM UTC. 2026-02-23T10:52:08.994 INFO:teuthology.orchestra.run.trial198.stdout:Package libcephfs2-2:20.3.0-5439.ge371433f.el9.x86_64 is already installed. 2026-02-23T10:52:09.012 INFO:teuthology.orchestra.run.trial198.stdout:Dependencies resolved. 2026-02-23T10:52:09.013 INFO:teuthology.orchestra.run.trial198.stdout:Nothing to do. 2026-02-23T10:52:09.013 INFO:teuthology.orchestra.run.trial198.stdout:Complete! 2026-02-23T10:52:09.034 DEBUG:teuthology.orchestra.run.trial198:> sudo yum -y install libcephfs-devel 2026-02-23T10:52:09.339 INFO:teuthology.orchestra.run.trial198.stdout:Last metadata expiration check: 0:01:37 ago on Mon 23 Feb 2026 10:50:32 AM UTC. 2026-02-23T10:52:09.422 INFO:teuthology.orchestra.run.trial198.stdout:Dependencies resolved. 2026-02-23T10:52:09.422 INFO:teuthology.orchestra.run.trial198.stdout:================================================================================ 2026-02-23T10:52:09.422 INFO:teuthology.orchestra.run.trial198.stdout: Package Arch Version Repo Size 2026-02-23T10:52:09.422 INFO:teuthology.orchestra.run.trial198.stdout:================================================================================ 2026-02-23T10:52:09.422 INFO:teuthology.orchestra.run.trial198.stdout:Installing: 2026-02-23T10:52:09.422 INFO:teuthology.orchestra.run.trial198.stdout: libcephfs-devel x86_64 2:20.3.0-5439.ge371433f.el9 ceph 37 k 2026-02-23T10:52:09.422 INFO:teuthology.orchestra.run.trial198.stdout:Installing dependencies: 2026-02-23T10:52:09.422 INFO:teuthology.orchestra.run.trial198.stdout: libcephfs-proxy2 x86_64 2:20.3.0-5439.ge371433f.el9 ceph 26 k 2026-02-23T10:52:09.422 INFO:teuthology.orchestra.run.trial198.stdout: 2026-02-23T10:52:09.422 INFO:teuthology.orchestra.run.trial198.stdout:Transaction Summary 2026-02-23T10:52:09.423 INFO:teuthology.orchestra.run.trial198.stdout:================================================================================ 2026-02-23T10:52:09.423 INFO:teuthology.orchestra.run.trial198.stdout:Install 2 Packages 2026-02-23T10:52:09.423 INFO:teuthology.orchestra.run.trial198.stdout: 2026-02-23T10:52:09.423 INFO:teuthology.orchestra.run.trial198.stdout:Total download size: 64 k 2026-02-23T10:52:09.423 INFO:teuthology.orchestra.run.trial198.stdout:Installed size: 231 k 2026-02-23T10:52:09.423 INFO:teuthology.orchestra.run.trial198.stdout:Downloading Packages: 2026-02-23T10:52:09.487 INFO:teuthology.orchestra.run.trial198.stdout:(1/2): libcephfs-devel-20.3.0-5439.ge371433f.el 579 kB/s | 37 kB 00:00 2026-02-23T10:52:09.498 INFO:teuthology.orchestra.run.trial198.stdout:(2/2): libcephfs-proxy2-20.3.0-5439.ge371433f.e 351 kB/s | 26 kB 00:00 2026-02-23T10:52:09.498 INFO:teuthology.orchestra.run.trial198.stdout:-------------------------------------------------------------------------------- 2026-02-23T10:52:09.499 INFO:teuthology.orchestra.run.trial198.stdout:Total 835 kB/s | 64 kB 00:00 2026-02-23T10:52:09.499 INFO:teuthology.orchestra.run.trial198.stdout:Running transaction check 2026-02-23T10:52:09.502 INFO:teuthology.orchestra.run.trial198.stdout:Transaction check succeeded. 2026-02-23T10:52:09.502 INFO:teuthology.orchestra.run.trial198.stdout:Running transaction test 2026-02-23T10:52:09.533 INFO:teuthology.orchestra.run.trial198.stdout:Transaction test succeeded. 2026-02-23T10:52:09.533 INFO:teuthology.orchestra.run.trial198.stdout:Running transaction 2026-02-23T10:52:09.582 INFO:teuthology.orchestra.run.trial198.stdout: Preparing : 1/1 2026-02-23T10:52:09.583 INFO:teuthology.orchestra.run.trial198.stdout: Installing : libcephfs-proxy2-2:20.3.0-5439.ge371433f.el9.x86_64 1/2 2026-02-23T10:52:09.598 INFO:teuthology.orchestra.run.trial198.stdout: Running scriptlet: libcephfs-proxy2-2:20.3.0-5439.ge371433f.el9.x86_64 1/2 2026-02-23T10:52:09.615 INFO:teuthology.orchestra.run.trial198.stdout: Installing : libcephfs-devel-2:20.3.0-5439.ge371433f.el9.x86_64 2/2 2026-02-23T10:52:09.672 INFO:teuthology.orchestra.run.trial198.stdout: Running scriptlet: libcephfs-devel-2:20.3.0-5439.ge371433f.el9.x86_64 2/2 2026-02-23T10:52:09.673 INFO:teuthology.orchestra.run.trial198.stdout: Verifying : libcephfs-devel-2:20.3.0-5439.ge371433f.el9.x86_64 1/2 2026-02-23T10:52:09.730 INFO:teuthology.orchestra.run.trial198.stdout: Verifying : libcephfs-proxy2-2:20.3.0-5439.ge371433f.el9.x86_64 2/2 2026-02-23T10:52:09.730 INFO:teuthology.orchestra.run.trial198.stdout: 2026-02-23T10:52:09.730 INFO:teuthology.orchestra.run.trial198.stdout:Installed: 2026-02-23T10:52:09.730 INFO:teuthology.orchestra.run.trial198.stdout: libcephfs-devel-2:20.3.0-5439.ge371433f.el9.x86_64 2026-02-23T10:52:09.730 INFO:teuthology.orchestra.run.trial198.stdout: libcephfs-proxy2-2:20.3.0-5439.ge371433f.el9.x86_64 2026-02-23T10:52:09.730 INFO:teuthology.orchestra.run.trial198.stdout: 2026-02-23T10:52:09.730 INFO:teuthology.orchestra.run.trial198.stdout:Complete! 2026-02-23T10:52:09.758 DEBUG:teuthology.orchestra.run.trial198:> sudo yum -y install librados2 2026-02-23T10:52:10.074 INFO:teuthology.orchestra.run.trial198.stdout:Last metadata expiration check: 0:01:38 ago on Mon 23 Feb 2026 10:50:32 AM UTC. 2026-02-23T10:52:10.138 INFO:teuthology.orchestra.run.trial198.stdout:Package librados2-2:20.3.0-5439.ge371433f.el9.x86_64 is already installed. 2026-02-23T10:52:10.157 INFO:teuthology.orchestra.run.trial198.stdout:Dependencies resolved. 2026-02-23T10:52:10.158 INFO:teuthology.orchestra.run.trial198.stdout:Nothing to do. 2026-02-23T10:52:10.158 INFO:teuthology.orchestra.run.trial198.stdout:Complete! 2026-02-23T10:52:10.180 DEBUG:teuthology.orchestra.run.trial198:> sudo yum -y install librbd1 2026-02-23T10:52:10.494 INFO:teuthology.orchestra.run.trial198.stdout:Last metadata expiration check: 0:01:38 ago on Mon 23 Feb 2026 10:50:32 AM UTC. 2026-02-23T10:52:10.558 INFO:teuthology.orchestra.run.trial198.stdout:Package librbd1-2:20.3.0-5439.ge371433f.el9.x86_64 is already installed. 2026-02-23T10:52:10.576 INFO:teuthology.orchestra.run.trial198.stdout:Dependencies resolved. 2026-02-23T10:52:10.577 INFO:teuthology.orchestra.run.trial198.stdout:Nothing to do. 2026-02-23T10:52:10.577 INFO:teuthology.orchestra.run.trial198.stdout:Complete! 2026-02-23T10:52:10.599 DEBUG:teuthology.orchestra.run.trial198:> sudo yum -y install python3-rados 2026-02-23T10:52:10.905 INFO:teuthology.orchestra.run.trial198.stdout:Last metadata expiration check: 0:01:38 ago on Mon 23 Feb 2026 10:50:32 AM UTC. 2026-02-23T10:52:10.972 INFO:teuthology.orchestra.run.trial198.stdout:Package python3-rados-2:20.3.0-5439.ge371433f.el9.x86_64 is already installed. 2026-02-23T10:52:10.991 INFO:teuthology.orchestra.run.trial198.stdout:Dependencies resolved. 2026-02-23T10:52:10.991 INFO:teuthology.orchestra.run.trial198.stdout:Nothing to do. 2026-02-23T10:52:10.991 INFO:teuthology.orchestra.run.trial198.stdout:Complete! 2026-02-23T10:52:11.012 DEBUG:teuthology.orchestra.run.trial198:> sudo yum -y install python3-rgw 2026-02-23T10:52:11.358 INFO:teuthology.orchestra.run.trial198.stdout:Last metadata expiration check: 0:01:39 ago on Mon 23 Feb 2026 10:50:32 AM UTC. 2026-02-23T10:52:11.422 INFO:teuthology.orchestra.run.trial198.stdout:Package python3-rgw-2:20.3.0-5439.ge371433f.el9.x86_64 is already installed. 2026-02-23T10:52:11.440 INFO:teuthology.orchestra.run.trial198.stdout:Dependencies resolved. 2026-02-23T10:52:11.441 INFO:teuthology.orchestra.run.trial198.stdout:Nothing to do. 2026-02-23T10:52:11.441 INFO:teuthology.orchestra.run.trial198.stdout:Complete! 2026-02-23T10:52:11.463 DEBUG:teuthology.orchestra.run.trial198:> sudo yum -y install python3-cephfs 2026-02-23T10:52:11.772 INFO:teuthology.orchestra.run.trial198.stdout:Last metadata expiration check: 0:01:39 ago on Mon 23 Feb 2026 10:50:32 AM UTC. 2026-02-23T10:52:11.838 INFO:teuthology.orchestra.run.trial198.stdout:Package python3-cephfs-2:20.3.0-5439.ge371433f.el9.x86_64 is already installed. 2026-02-23T10:52:11.856 INFO:teuthology.orchestra.run.trial198.stdout:Dependencies resolved. 2026-02-23T10:52:11.857 INFO:teuthology.orchestra.run.trial198.stdout:Nothing to do. 2026-02-23T10:52:11.857 INFO:teuthology.orchestra.run.trial198.stdout:Complete! 2026-02-23T10:52:11.878 DEBUG:teuthology.orchestra.run.trial198:> sudo yum -y install python3-rbd 2026-02-23T10:52:12.190 INFO:teuthology.orchestra.run.trial198.stdout:Last metadata expiration check: 0:01:40 ago on Mon 23 Feb 2026 10:50:32 AM UTC. 2026-02-23T10:52:12.254 INFO:teuthology.orchestra.run.trial198.stdout:Package python3-rbd-2:20.3.0-5439.ge371433f.el9.x86_64 is already installed. 2026-02-23T10:52:12.273 INFO:teuthology.orchestra.run.trial198.stdout:Dependencies resolved. 2026-02-23T10:52:12.273 INFO:teuthology.orchestra.run.trial198.stdout:Nothing to do. 2026-02-23T10:52:12.274 INFO:teuthology.orchestra.run.trial198.stdout:Complete! 2026-02-23T10:52:12.296 DEBUG:teuthology.orchestra.run.trial198:> sudo yum -y install rbd-fuse 2026-02-23T10:52:12.604 INFO:teuthology.orchestra.run.trial198.stdout:Last metadata expiration check: 0:01:40 ago on Mon 23 Feb 2026 10:50:32 AM UTC. 2026-02-23T10:52:12.687 INFO:teuthology.orchestra.run.trial198.stdout:Dependencies resolved. 2026-02-23T10:52:12.687 INFO:teuthology.orchestra.run.trial198.stdout:================================================================================ 2026-02-23T10:52:12.687 INFO:teuthology.orchestra.run.trial198.stdout: Package Architecture Version Repository Size 2026-02-23T10:52:12.687 INFO:teuthology.orchestra.run.trial198.stdout:================================================================================ 2026-02-23T10:52:12.688 INFO:teuthology.orchestra.run.trial198.stdout:Installing: 2026-02-23T10:52:12.688 INFO:teuthology.orchestra.run.trial198.stdout: rbd-fuse x86_64 2:20.3.0-5439.ge371433f.el9 ceph 86 k 2026-02-23T10:52:12.688 INFO:teuthology.orchestra.run.trial198.stdout: 2026-02-23T10:52:12.688 INFO:teuthology.orchestra.run.trial198.stdout:Transaction Summary 2026-02-23T10:52:12.688 INFO:teuthology.orchestra.run.trial198.stdout:================================================================================ 2026-02-23T10:52:12.688 INFO:teuthology.orchestra.run.trial198.stdout:Install 1 Package 2026-02-23T10:52:12.688 INFO:teuthology.orchestra.run.trial198.stdout: 2026-02-23T10:52:12.688 INFO:teuthology.orchestra.run.trial198.stdout:Total download size: 86 k 2026-02-23T10:52:12.688 INFO:teuthology.orchestra.run.trial198.stdout:Installed size: 236 k 2026-02-23T10:52:12.688 INFO:teuthology.orchestra.run.trial198.stdout:Downloading Packages: 2026-02-23T10:52:13.086 INFO:teuthology.orchestra.run.trial198.stdout:rbd-fuse-20.3.0-5439.ge371433f.el9.x86_64.rpm 217 kB/s | 86 kB 00:00 2026-02-23T10:52:13.087 INFO:teuthology.orchestra.run.trial198.stdout:-------------------------------------------------------------------------------- 2026-02-23T10:52:13.087 INFO:teuthology.orchestra.run.trial198.stdout:Total 217 kB/s | 86 kB 00:00 2026-02-23T10:52:13.087 INFO:teuthology.orchestra.run.trial198.stdout:Running transaction check 2026-02-23T10:52:13.091 INFO:teuthology.orchestra.run.trial198.stdout:Transaction check succeeded. 2026-02-23T10:52:13.091 INFO:teuthology.orchestra.run.trial198.stdout:Running transaction test 2026-02-23T10:52:13.122 INFO:teuthology.orchestra.run.trial198.stdout:Transaction test succeeded. 2026-02-23T10:52:13.122 INFO:teuthology.orchestra.run.trial198.stdout:Running transaction 2026-02-23T10:52:13.173 INFO:teuthology.orchestra.run.trial198.stdout: Preparing : 1/1 2026-02-23T10:52:13.190 INFO:teuthology.orchestra.run.trial198.stdout: Installing : rbd-fuse-2:20.3.0-5439.ge371433f.el9.x86_64 1/1 2026-02-23T10:52:13.271 INFO:teuthology.orchestra.run.trial198.stdout: Running scriptlet: rbd-fuse-2:20.3.0-5439.ge371433f.el9.x86_64 1/1 2026-02-23T10:52:13.330 INFO:teuthology.orchestra.run.trial198.stdout: Verifying : rbd-fuse-2:20.3.0-5439.ge371433f.el9.x86_64 1/1 2026-02-23T10:52:13.330 INFO:teuthology.orchestra.run.trial198.stdout: 2026-02-23T10:52:13.330 INFO:teuthology.orchestra.run.trial198.stdout:Installed: 2026-02-23T10:52:13.330 INFO:teuthology.orchestra.run.trial198.stdout: rbd-fuse-2:20.3.0-5439.ge371433f.el9.x86_64 2026-02-23T10:52:13.330 INFO:teuthology.orchestra.run.trial198.stdout: 2026-02-23T10:52:13.330 INFO:teuthology.orchestra.run.trial198.stdout:Complete! 2026-02-23T10:52:13.359 DEBUG:teuthology.orchestra.run.trial198:> sudo yum -y install rbd-mirror 2026-02-23T10:52:13.666 INFO:teuthology.orchestra.run.trial198.stdout:Last metadata expiration check: 0:01:41 ago on Mon 23 Feb 2026 10:50:32 AM UTC. 2026-02-23T10:52:13.748 INFO:teuthology.orchestra.run.trial198.stdout:Dependencies resolved. 2026-02-23T10:52:13.749 INFO:teuthology.orchestra.run.trial198.stdout:================================================================================ 2026-02-23T10:52:13.749 INFO:teuthology.orchestra.run.trial198.stdout: Package Arch Version Repo Size 2026-02-23T10:52:13.749 INFO:teuthology.orchestra.run.trial198.stdout:================================================================================ 2026-02-23T10:52:13.749 INFO:teuthology.orchestra.run.trial198.stdout:Installing: 2026-02-23T10:52:13.749 INFO:teuthology.orchestra.run.trial198.stdout: rbd-mirror x86_64 2:20.3.0-5439.ge371433f.el9 ceph 2.6 M 2026-02-23T10:52:13.749 INFO:teuthology.orchestra.run.trial198.stdout: 2026-02-23T10:52:13.749 INFO:teuthology.orchestra.run.trial198.stdout:Transaction Summary 2026-02-23T10:52:13.749 INFO:teuthology.orchestra.run.trial198.stdout:================================================================================ 2026-02-23T10:52:13.749 INFO:teuthology.orchestra.run.trial198.stdout:Install 1 Package 2026-02-23T10:52:13.749 INFO:teuthology.orchestra.run.trial198.stdout: 2026-02-23T10:52:13.749 INFO:teuthology.orchestra.run.trial198.stdout:Total download size: 2.6 M 2026-02-23T10:52:13.749 INFO:teuthology.orchestra.run.trial198.stdout:Installed size: 11 M 2026-02-23T10:52:13.749 INFO:teuthology.orchestra.run.trial198.stdout:Downloading Packages: 2026-02-23T10:52:14.756 INFO:teuthology.orchestra.run.trial198.stdout:rbd-mirror-20.3.0-5439.ge371433f.el9.x86_64.rpm 2.6 MB/s | 2.6 MB 00:01 2026-02-23T10:52:14.757 INFO:teuthology.orchestra.run.trial198.stdout:-------------------------------------------------------------------------------- 2026-02-23T10:52:14.757 INFO:teuthology.orchestra.run.trial198.stdout:Total 2.6 MB/s | 2.6 MB 00:01 2026-02-23T10:52:14.757 INFO:teuthology.orchestra.run.trial198.stdout:Running transaction check 2026-02-23T10:52:14.762 INFO:teuthology.orchestra.run.trial198.stdout:Transaction check succeeded. 2026-02-23T10:52:14.762 INFO:teuthology.orchestra.run.trial198.stdout:Running transaction test 2026-02-23T10:52:14.800 INFO:teuthology.orchestra.run.trial198.stdout:Transaction test succeeded. 2026-02-23T10:52:14.800 INFO:teuthology.orchestra.run.trial198.stdout:Running transaction 2026-02-23T10:52:14.961 INFO:teuthology.orchestra.run.trial198.stdout: Preparing : 1/1 2026-02-23T10:52:14.963 INFO:teuthology.orchestra.run.trial198.stdout: Installing : rbd-mirror-2:20.3.0-5439.ge371433f.el9.x86_64 1/1 2026-02-23T10:52:14.985 INFO:teuthology.orchestra.run.trial198.stdout: Running scriptlet: rbd-mirror-2:20.3.0-5439.ge371433f.el9.x86_64 1/1 2026-02-23T10:52:14.985 INFO:teuthology.orchestra.run.trial198.stdout:Glob pattern passed to enable, but globs are not supported for this. 2026-02-23T10:52:14.985 INFO:teuthology.orchestra.run.trial198.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2026-02-23T10:52:14.985 INFO:teuthology.orchestra.run.trial198.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2026-02-23T10:52:14.985 INFO:teuthology.orchestra.run.trial198.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2026-02-23T10:52:14.985 INFO:teuthology.orchestra.run.trial198.stdout: 2026-02-23T10:52:15.307 INFO:teuthology.orchestra.run.trial198.stdout: Verifying : rbd-mirror-2:20.3.0-5439.ge371433f.el9.x86_64 1/1 2026-02-23T10:52:15.307 INFO:teuthology.orchestra.run.trial198.stdout: 2026-02-23T10:52:15.307 INFO:teuthology.orchestra.run.trial198.stdout:Installed: 2026-02-23T10:52:15.307 INFO:teuthology.orchestra.run.trial198.stdout: rbd-mirror-2:20.3.0-5439.ge371433f.el9.x86_64 2026-02-23T10:52:15.307 INFO:teuthology.orchestra.run.trial198.stdout: 2026-02-23T10:52:15.307 INFO:teuthology.orchestra.run.trial198.stdout:Complete! 2026-02-23T10:52:15.335 DEBUG:teuthology.orchestra.run.trial198:> sudo yum -y install rbd-nbd 2026-02-23T10:52:15.647 INFO:teuthology.orchestra.run.trial198.stdout:Last metadata expiration check: 0:01:43 ago on Mon 23 Feb 2026 10:50:32 AM UTC. 2026-02-23T10:52:15.732 INFO:teuthology.orchestra.run.trial198.stdout:Dependencies resolved. 2026-02-23T10:52:15.733 INFO:teuthology.orchestra.run.trial198.stdout:================================================================================ 2026-02-23T10:52:15.733 INFO:teuthology.orchestra.run.trial198.stdout: Package Architecture Version Repository Size 2026-02-23T10:52:15.733 INFO:teuthology.orchestra.run.trial198.stdout:================================================================================ 2026-02-23T10:52:15.733 INFO:teuthology.orchestra.run.trial198.stdout:Installing: 2026-02-23T10:52:15.733 INFO:teuthology.orchestra.run.trial198.stdout: rbd-nbd x86_64 2:20.3.0-5439.ge371433f.el9 ceph 166 k 2026-02-23T10:52:15.733 INFO:teuthology.orchestra.run.trial198.stdout: 2026-02-23T10:52:15.733 INFO:teuthology.orchestra.run.trial198.stdout:Transaction Summary 2026-02-23T10:52:15.733 INFO:teuthology.orchestra.run.trial198.stdout:================================================================================ 2026-02-23T10:52:15.733 INFO:teuthology.orchestra.run.trial198.stdout:Install 1 Package 2026-02-23T10:52:15.734 INFO:teuthology.orchestra.run.trial198.stdout: 2026-02-23T10:52:15.734 INFO:teuthology.orchestra.run.trial198.stdout:Total download size: 166 k 2026-02-23T10:52:15.734 INFO:teuthology.orchestra.run.trial198.stdout:Installed size: 496 k 2026-02-23T10:52:15.734 INFO:teuthology.orchestra.run.trial198.stdout:Downloading Packages: 2026-02-23T10:52:16.203 INFO:teuthology.orchestra.run.trial198.stdout:rbd-nbd-20.3.0-5439.ge371433f.el9.x86_64.rpm 355 kB/s | 166 kB 00:00 2026-02-23T10:52:16.204 INFO:teuthology.orchestra.run.trial198.stdout:-------------------------------------------------------------------------------- 2026-02-23T10:52:16.204 INFO:teuthology.orchestra.run.trial198.stdout:Total 354 kB/s | 166 kB 00:00 2026-02-23T10:52:16.204 INFO:teuthology.orchestra.run.trial198.stdout:Running transaction check 2026-02-23T10:52:16.208 INFO:teuthology.orchestra.run.trial198.stdout:Transaction check succeeded. 2026-02-23T10:52:16.208 INFO:teuthology.orchestra.run.trial198.stdout:Running transaction test 2026-02-23T10:52:16.240 INFO:teuthology.orchestra.run.trial198.stdout:Transaction test succeeded. 2026-02-23T10:52:16.240 INFO:teuthology.orchestra.run.trial198.stdout:Running transaction 2026-02-23T10:52:16.295 INFO:teuthology.orchestra.run.trial198.stdout: Preparing : 1/1 2026-02-23T10:52:16.312 INFO:teuthology.orchestra.run.trial198.stdout: Installing : rbd-nbd-2:20.3.0-5439.ge371433f.el9.x86_64 1/1 2026-02-23T10:52:16.393 INFO:teuthology.orchestra.run.trial198.stdout: Running scriptlet: rbd-nbd-2:20.3.0-5439.ge371433f.el9.x86_64 1/1 2026-02-23T10:52:16.451 INFO:teuthology.orchestra.run.trial198.stdout: Verifying : rbd-nbd-2:20.3.0-5439.ge371433f.el9.x86_64 1/1 2026-02-23T10:52:16.451 INFO:teuthology.orchestra.run.trial198.stdout: 2026-02-23T10:52:16.451 INFO:teuthology.orchestra.run.trial198.stdout:Installed: 2026-02-23T10:52:16.451 INFO:teuthology.orchestra.run.trial198.stdout: rbd-nbd-2:20.3.0-5439.ge371433f.el9.x86_64 2026-02-23T10:52:16.451 INFO:teuthology.orchestra.run.trial198.stdout: 2026-02-23T10:52:16.451 INFO:teuthology.orchestra.run.trial198.stdout:Complete! 2026-02-23T10:52:16.480 DEBUG:teuthology.orchestra.run.trial198:> sudo yum -y install ceph-osd-crimson 2026-02-23T10:52:16.792 INFO:teuthology.orchestra.run.trial198.stdout:Last metadata expiration check: 0:01:44 ago on Mon 23 Feb 2026 10:50:32 AM UTC. 2026-02-23T10:52:16.858 INFO:teuthology.orchestra.run.trial198.stdout:Package ceph-osd-crimson-2:20.3.0-5439.ge371433f.el9.x86_64 is already installed. 2026-02-23T10:52:16.877 INFO:teuthology.orchestra.run.trial198.stdout:Dependencies resolved. 2026-02-23T10:52:16.877 INFO:teuthology.orchestra.run.trial198.stdout:Nothing to do. 2026-02-23T10:52:16.877 INFO:teuthology.orchestra.run.trial198.stdout:Complete! 2026-02-23T10:52:16.900 DEBUG:teuthology.orchestra.run.trial198:> sudo yum -y install ceph-osd-crimson-debuginfo 2026-02-23T10:52:17.219 INFO:teuthology.orchestra.run.trial198.stdout:Last metadata expiration check: 0:01:45 ago on Mon 23 Feb 2026 10:50:32 AM UTC. 2026-02-23T10:52:17.305 INFO:teuthology.orchestra.run.trial198.stdout:Dependencies resolved. 2026-02-23T10:52:17.305 INFO:teuthology.orchestra.run.trial198.stdout:================================================================================ 2026-02-23T10:52:17.305 INFO:teuthology.orchestra.run.trial198.stdout: Package Arch Version Repo Size 2026-02-23T10:52:17.305 INFO:teuthology.orchestra.run.trial198.stdout:================================================================================ 2026-02-23T10:52:17.306 INFO:teuthology.orchestra.run.trial198.stdout:Installing: 2026-02-23T10:52:17.306 INFO:teuthology.orchestra.run.trial198.stdout: ceph-osd-crimson-debuginfo x86_64 2:20.3.0-5439.ge371433f.el9 ceph 429 M 2026-02-23T10:52:17.306 INFO:teuthology.orchestra.run.trial198.stdout: 2026-02-23T10:52:17.306 INFO:teuthology.orchestra.run.trial198.stdout:Transaction Summary 2026-02-23T10:52:17.306 INFO:teuthology.orchestra.run.trial198.stdout:================================================================================ 2026-02-23T10:52:17.306 INFO:teuthology.orchestra.run.trial198.stdout:Install 1 Package 2026-02-23T10:52:17.306 INFO:teuthology.orchestra.run.trial198.stdout: 2026-02-23T10:52:17.306 INFO:teuthology.orchestra.run.trial198.stdout:Total download size: 429 M 2026-02-23T10:52:17.306 INFO:teuthology.orchestra.run.trial198.stdout:Installed size: 2.6 G 2026-02-23T10:52:17.306 INFO:teuthology.orchestra.run.trial198.stdout:Downloading Packages: 2026-02-23T10:53:36.664 INFO:teuthology.orchestra.run.trial198.stdout:ceph-osd-crimson-debuginfo-20.3.0-5439.ge371433 5.4 MB/s | 429 MB 01:19 2026-02-23T10:53:36.664 INFO:teuthology.orchestra.run.trial198.stdout:-------------------------------------------------------------------------------- 2026-02-23T10:53:36.665 INFO:teuthology.orchestra.run.trial198.stdout:Total 5.4 MB/s | 429 MB 01:19 2026-02-23T10:53:36.665 INFO:teuthology.orchestra.run.trial198.stdout:Running transaction check 2026-02-23T10:53:36.667 INFO:teuthology.orchestra.run.trial198.stdout:Transaction check succeeded. 2026-02-23T10:53:36.667 INFO:teuthology.orchestra.run.trial198.stdout:Running transaction test 2026-02-23T10:53:37.940 INFO:teuthology.orchestra.run.trial198.stdout:Transaction test succeeded. 2026-02-23T10:53:37.940 INFO:teuthology.orchestra.run.trial198.stdout:Running transaction 2026-02-23T10:54:01.661 INFO:teuthology.orchestra.run.trial198.stdout: Preparing : 1/1 2026-02-23T10:54:01.677 INFO:teuthology.orchestra.run.trial198.stdout: Installing : ceph-osd-crimson-debuginfo-2:20.3.0-5439.ge371433f.e 1/1 2026-02-23T10:54:03.585 INFO:teuthology.orchestra.run.trial198.stdout: Running scriptlet: ceph-osd-crimson-debuginfo-2:20.3.0-5439.ge371433f.e 1/1 2026-02-23T10:54:03.642 INFO:teuthology.orchestra.run.trial198.stdout: Verifying : ceph-osd-crimson-debuginfo-2:20.3.0-5439.ge371433f.e 1/1 2026-02-23T10:54:03.642 INFO:teuthology.orchestra.run.trial198.stdout: 2026-02-23T10:54:03.642 INFO:teuthology.orchestra.run.trial198.stdout:Installed: 2026-02-23T10:54:03.642 INFO:teuthology.orchestra.run.trial198.stdout: ceph-osd-crimson-debuginfo-2:20.3.0-5439.ge371433f.el9.x86_64 2026-02-23T10:54:03.643 INFO:teuthology.orchestra.run.trial198.stdout: 2026-02-23T10:54:03.643 INFO:teuthology.orchestra.run.trial198.stdout:Complete! 2026-02-23T10:54:03.708 DEBUG:teuthology.parallel:result is None 2026-02-23T10:54:03.708 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=debug&distros=centos%2F9%2Fx86_64&sha1=e371433fcd1390a0b82cfb18effc422c1f5d8c1d 2026-02-23T10:54:03.876 DEBUG:teuthology.orchestra.run.trial198:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2026-02-23T10:54:03.897 INFO:teuthology.orchestra.run.trial198.stdout:20.3.0-5439.ge371433f.el9 2026-02-23T10:54:03.898 INFO:teuthology.packaging:The installed version of ceph is 20.3.0-5439.ge371433f.el9 2026-02-23T10:54:03.898 INFO:teuthology.task.install:The correct ceph version 20.3.0-5439.ge371433f is installed. 2026-02-23T10:54:03.901 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2026-02-23T10:54:03.901 DEBUG:teuthology.orchestra.run.trial198:> set -ex 2026-02-23T10:54:03.901 DEBUG:teuthology.orchestra.run.trial198:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2026-02-23T10:54:03.963 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2026-02-23T10:54:03.963 DEBUG:teuthology.orchestra.run.trial198:> set -ex 2026-02-23T10:54:03.963 DEBUG:teuthology.orchestra.run.trial198:> sudo dd of=/usr/bin/daemon-helper 2026-02-23T10:54:04.027 DEBUG:teuthology.orchestra.run.trial198:> sudo chmod a=rx -- /usr/bin/daemon-helper 2026-02-23T10:54:04.090 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2026-02-23T10:54:04.090 DEBUG:teuthology.orchestra.run.trial198:> set -ex 2026-02-23T10:54:04.090 DEBUG:teuthology.orchestra.run.trial198:> sudo dd of=/usr/bin/adjust-ulimits 2026-02-23T10:54:04.153 DEBUG:teuthology.orchestra.run.trial198:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2026-02-23T10:54:04.216 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2026-02-23T10:54:04.216 DEBUG:teuthology.orchestra.run.trial198:> set -ex 2026-02-23T10:54:04.216 DEBUG:teuthology.orchestra.run.trial198:> sudo dd of=/usr/bin/stdin-killer 2026-02-23T10:54:04.282 DEBUG:teuthology.orchestra.run.trial198:> sudo chmod a=rx -- /usr/bin/stdin-killer 2026-02-23T10:54:04.346 INFO:teuthology.run_tasks:Running task pexec... 2026-02-23T10:54:04.361 INFO:teuthology.task.pexec:Executing custom commands... 2026-02-23T10:54:04.361 DEBUG:teuthology.orchestra.run.trial198:> TESTDIR=/home/ubuntu/cephtest bash -s 2026-02-23T10:54:04.389 INFO:teuthology.task.pexec:Running commands on host ubuntu@trial198.front.sepia.ceph.com 2026-02-23T10:54:04.389 INFO:teuthology.task.pexec:sudo update-alternatives --set ceph-osd /usr/bin/ceph-osd-crimson 2026-02-23T10:54:04.440 DEBUG:teuthology.parallel:result is None 2026-02-23T10:54:04.441 INFO:teuthology.run_tasks:Running task install... 2026-02-23T10:54:04.454 DEBUG:teuthology.task.install:project ceph 2026-02-23T10:54:04.455 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'debug', 'sha1': 'e371433fcd1390a0b82cfb18effc422c1f5d8c1d'}} 2026-02-23T10:54:04.455 DEBUG:teuthology.task.install:config {'flavor': 'debug', 'sha1': 'e371433fcd1390a0b82cfb18effc422c1f5d8c1d'} 2026-02-23T10:54:04.455 INFO:teuthology.task.install:Using flavor: debug 2026-02-23T10:54:04.463 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-osd', 'ceph-osd-classic', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-osd', 'ceph-osd-classic', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2026-02-23T10:54:04.463 INFO:teuthology.task.install:extra packages: [] 2026-02-23T10:54:04.464 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'debug', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': 'e371433fcd1390a0b82cfb18effc422c1f5d8c1d', 'tag': None, 'wait_for_package': False} 2026-02-23T10:54:04.464 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=debug&distros=centos%2F9%2Fx86_64&sha1=e371433fcd1390a0b82cfb18effc422c1f5d8c1d 2026-02-23T10:54:04.652 INFO:teuthology.task.install.rpm:Pulling from https://3.chacra.ceph.com/r/ceph/wip-shraddhaag-enable-cephadm-tests/e371433fcd1390a0b82cfb18effc422c1f5d8c1d/centos/9/flavors/debug/ 2026-02-23T10:54:04.652 INFO:teuthology.task.install.rpm:Package version is 20.3.0-5439.ge371433f 2026-02-23T10:54:04.910 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://3.chacra.ceph.com/r/ceph/wip-shraddhaag-enable-cephadm-tests/e371433fcd1390a0b82cfb18effc422c1f5d8c1d/centos/9/flavors/debug/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://3.chacra.ceph.com/r/ceph/wip-shraddhaag-enable-cephadm-tests/e371433fcd1390a0b82cfb18effc422c1f5d8c1d/centos/9/flavors/debug/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://3.chacra.ceph.com/r/ceph/wip-shraddhaag-enable-cephadm-tests/e371433fcd1390a0b82cfb18effc422c1f5d8c1d/centos/9/flavors/debug/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2026-02-23T10:54:04.910 DEBUG:teuthology.orchestra.run.trial198:> set -ex 2026-02-23T10:54:04.910 DEBUG:teuthology.orchestra.run.trial198:> sudo dd of=/etc/yum.repos.d/ceph.repo 2026-02-23T10:54:04.935 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-osd, ceph-osd-classic, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2026-02-23T10:54:04.935 DEBUG:teuthology.orchestra.run.trial198:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/e371433fcd1390a0b82cfb18effc422c1f5d8c1d/;g' /etc/yum.repos.d/ceph.repo ; fi 2026-02-23T10:54:05.000 DEBUG:teuthology.orchestra.run.trial198:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2026-02-23T10:54:05.064 DEBUG:teuthology.orchestra.run.trial198:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2026-02-23T10:54:05.121 INFO:teuthology.orchestra.run.trial198.stdout:check_obsoletes = 1 2026-02-23T10:54:05.131 DEBUG:teuthology.orchestra.run.trial198:> sudo yum clean all 2026-02-23T10:54:05.290 INFO:teuthology.orchestra.run.trial198.stdout:56 files removed 2026-02-23T10:54:05.306 DEBUG:teuthology.orchestra.run.trial198:> sudo yum -y install ceph-radosgw 2026-02-23T10:54:05.798 INFO:teuthology.orchestra.run.trial198.stdout:ceph packages for x86_64 268 kB/s | 92 kB 00:00 2026-02-23T10:54:06.092 INFO:teuthology.orchestra.run.trial198.stdout:ceph noarch packages 89 kB/s | 24 kB 00:00 2026-02-23T10:54:06.330 INFO:teuthology.orchestra.run.trial198.stdout:ceph source packages 9.5 kB/s | 2.1 kB 00:00 2026-02-23T10:54:07.041 INFO:teuthology.orchestra.run.trial198.stdout:CentOS Stream 9 - BaseOS 13 MB/s | 8.9 MB 00:00 2026-02-23T10:54:08.175 INFO:teuthology.orchestra.run.trial198.stdout:CentOS Stream 9 - AppStream 45 MB/s | 27 MB 00:00 2026-02-23T10:54:11.637 INFO:teuthology.orchestra.run.trial198.stdout:CentOS Stream 9 - CRB 7.2 MB/s | 7.8 MB 00:01 2026-02-23T10:54:12.728 INFO:teuthology.orchestra.run.trial198.stdout:CentOS Stream 9 - Extras packages 54 kB/s | 20 kB 00:00 2026-02-23T10:54:13.665 INFO:teuthology.orchestra.run.trial198.stdout:Extra Packages for Enterprise Linux 23 MB/s | 20 MB 00:00 2026-02-23T10:54:16.850 INFO:teuthology.orchestra.run.trial198.stdout:lab-extras 2.7 MB/s | 50 kB 00:00 2026-02-23T10:54:18.029 INFO:teuthology.orchestra.run.trial198.stdout:Package ceph-radosgw-2:20.3.0-5439.ge371433f.el9.x86_64 is already installed. 2026-02-23T10:54:18.047 INFO:teuthology.orchestra.run.trial198.stdout:Dependencies resolved. 2026-02-23T10:54:18.048 INFO:teuthology.orchestra.run.trial198.stdout:Nothing to do. 2026-02-23T10:54:18.048 INFO:teuthology.orchestra.run.trial198.stdout:Complete! 2026-02-23T10:54:18.072 DEBUG:teuthology.orchestra.run.trial198:> sudo yum -y install ceph-test 2026-02-23T10:54:18.385 INFO:teuthology.orchestra.run.trial198.stdout:Last metadata expiration check: 0:00:02 ago on Mon 23 Feb 2026 10:54:16 AM UTC. 2026-02-23T10:54:18.450 INFO:teuthology.orchestra.run.trial198.stdout:Package ceph-test-2:20.3.0-5439.ge371433f.el9.x86_64 is already installed. 2026-02-23T10:54:18.468 INFO:teuthology.orchestra.run.trial198.stdout:Dependencies resolved. 2026-02-23T10:54:18.469 INFO:teuthology.orchestra.run.trial198.stdout:Nothing to do. 2026-02-23T10:54:18.469 INFO:teuthology.orchestra.run.trial198.stdout:Complete! 2026-02-23T10:54:18.490 DEBUG:teuthology.orchestra.run.trial198:> sudo yum -y install ceph 2026-02-23T10:54:18.797 INFO:teuthology.orchestra.run.trial198.stdout:Last metadata expiration check: 0:00:02 ago on Mon 23 Feb 2026 10:54:16 AM UTC. 2026-02-23T10:54:18.861 INFO:teuthology.orchestra.run.trial198.stdout:Package ceph-2:20.3.0-5439.ge371433f.el9.x86_64 is already installed. 2026-02-23T10:54:18.879 INFO:teuthology.orchestra.run.trial198.stdout:Dependencies resolved. 2026-02-23T10:54:18.880 INFO:teuthology.orchestra.run.trial198.stdout:Nothing to do. 2026-02-23T10:54:18.880 INFO:teuthology.orchestra.run.trial198.stdout:Complete! 2026-02-23T10:54:18.901 DEBUG:teuthology.orchestra.run.trial198:> sudo yum -y install ceph-base 2026-02-23T10:54:19.209 INFO:teuthology.orchestra.run.trial198.stdout:Last metadata expiration check: 0:00:03 ago on Mon 23 Feb 2026 10:54:16 AM UTC. 2026-02-23T10:54:19.273 INFO:teuthology.orchestra.run.trial198.stdout:Package ceph-base-2:20.3.0-5439.ge371433f.el9.x86_64 is already installed. 2026-02-23T10:54:19.292 INFO:teuthology.orchestra.run.trial198.stdout:Dependencies resolved. 2026-02-23T10:54:19.292 INFO:teuthology.orchestra.run.trial198.stdout:Nothing to do. 2026-02-23T10:54:19.292 INFO:teuthology.orchestra.run.trial198.stdout:Complete! 2026-02-23T10:54:19.313 DEBUG:teuthology.orchestra.run.trial198:> sudo yum -y install cephadm 2026-02-23T10:54:19.617 INFO:teuthology.orchestra.run.trial198.stdout:Last metadata expiration check: 0:00:03 ago on Mon 23 Feb 2026 10:54:16 AM UTC. 2026-02-23T10:54:19.682 INFO:teuthology.orchestra.run.trial198.stdout:Package cephadm-2:20.3.0-5439.ge371433f.el9.noarch is already installed. 2026-02-23T10:54:19.701 INFO:teuthology.orchestra.run.trial198.stdout:Dependencies resolved. 2026-02-23T10:54:19.701 INFO:teuthology.orchestra.run.trial198.stdout:Nothing to do. 2026-02-23T10:54:19.701 INFO:teuthology.orchestra.run.trial198.stdout:Complete! 2026-02-23T10:54:19.722 DEBUG:teuthology.orchestra.run.trial198:> sudo yum -y install ceph-immutable-object-cache 2026-02-23T10:54:20.033 INFO:teuthology.orchestra.run.trial198.stdout:Last metadata expiration check: 0:00:04 ago on Mon 23 Feb 2026 10:54:16 AM UTC. 2026-02-23T10:54:20.098 INFO:teuthology.orchestra.run.trial198.stdout:Package ceph-immutable-object-cache-2:20.3.0-5439.ge371433f.el9.x86_64 is already installed. 2026-02-23T10:54:20.116 INFO:teuthology.orchestra.run.trial198.stdout:Dependencies resolved. 2026-02-23T10:54:20.117 INFO:teuthology.orchestra.run.trial198.stdout:Nothing to do. 2026-02-23T10:54:20.117 INFO:teuthology.orchestra.run.trial198.stdout:Complete! 2026-02-23T10:54:20.138 DEBUG:teuthology.orchestra.run.trial198:> sudo yum -y install ceph-mgr 2026-02-23T10:54:20.443 INFO:teuthology.orchestra.run.trial198.stdout:Last metadata expiration check: 0:00:04 ago on Mon 23 Feb 2026 10:54:16 AM UTC. 2026-02-23T10:54:20.508 INFO:teuthology.orchestra.run.trial198.stdout:Package ceph-mgr-2:20.3.0-5439.ge371433f.el9.x86_64 is already installed. 2026-02-23T10:54:20.527 INFO:teuthology.orchestra.run.trial198.stdout:Dependencies resolved. 2026-02-23T10:54:20.527 INFO:teuthology.orchestra.run.trial198.stdout:Nothing to do. 2026-02-23T10:54:20.527 INFO:teuthology.orchestra.run.trial198.stdout:Complete! 2026-02-23T10:54:20.549 DEBUG:teuthology.orchestra.run.trial198:> sudo yum -y install ceph-mgr-dashboard 2026-02-23T10:54:20.856 INFO:teuthology.orchestra.run.trial198.stdout:Last metadata expiration check: 0:00:04 ago on Mon 23 Feb 2026 10:54:16 AM UTC. 2026-02-23T10:54:20.920 INFO:teuthology.orchestra.run.trial198.stdout:Package ceph-mgr-dashboard-2:20.3.0-5439.ge371433f.el9.noarch is already installed. 2026-02-23T10:54:20.939 INFO:teuthology.orchestra.run.trial198.stdout:Dependencies resolved. 2026-02-23T10:54:20.940 INFO:teuthology.orchestra.run.trial198.stdout:Nothing to do. 2026-02-23T10:54:20.940 INFO:teuthology.orchestra.run.trial198.stdout:Complete! 2026-02-23T10:54:20.960 DEBUG:teuthology.orchestra.run.trial198:> sudo yum -y install ceph-mgr-diskprediction-local 2026-02-23T10:54:21.264 INFO:teuthology.orchestra.run.trial198.stdout:Last metadata expiration check: 0:00:05 ago on Mon 23 Feb 2026 10:54:16 AM UTC. 2026-02-23T10:54:21.329 INFO:teuthology.orchestra.run.trial198.stdout:Package ceph-mgr-diskprediction-local-2:20.3.0-5439.ge371433f.el9.noarch is already installed. 2026-02-23T10:54:21.348 INFO:teuthology.orchestra.run.trial198.stdout:Dependencies resolved. 2026-02-23T10:54:21.348 INFO:teuthology.orchestra.run.trial198.stdout:Nothing to do. 2026-02-23T10:54:21.349 INFO:teuthology.orchestra.run.trial198.stdout:Complete! 2026-02-23T10:54:21.370 DEBUG:teuthology.orchestra.run.trial198:> sudo yum -y install ceph-mgr-rook 2026-02-23T10:54:21.677 INFO:teuthology.orchestra.run.trial198.stdout:Last metadata expiration check: 0:00:05 ago on Mon 23 Feb 2026 10:54:16 AM UTC. 2026-02-23T10:54:21.741 INFO:teuthology.orchestra.run.trial198.stdout:Package ceph-mgr-rook-2:20.3.0-5439.ge371433f.el9.noarch is already installed. 2026-02-23T10:54:21.759 INFO:teuthology.orchestra.run.trial198.stdout:Dependencies resolved. 2026-02-23T10:54:21.760 INFO:teuthology.orchestra.run.trial198.stdout:Nothing to do. 2026-02-23T10:54:21.760 INFO:teuthology.orchestra.run.trial198.stdout:Complete! 2026-02-23T10:54:21.780 DEBUG:teuthology.orchestra.run.trial198:> sudo yum -y install ceph-mgr-cephadm 2026-02-23T10:54:22.088 INFO:teuthology.orchestra.run.trial198.stdout:Last metadata expiration check: 0:00:06 ago on Mon 23 Feb 2026 10:54:16 AM UTC. 2026-02-23T10:54:22.153 INFO:teuthology.orchestra.run.trial198.stdout:Package ceph-mgr-cephadm-2:20.3.0-5439.ge371433f.el9.noarch is already installed. 2026-02-23T10:54:22.171 INFO:teuthology.orchestra.run.trial198.stdout:Dependencies resolved. 2026-02-23T10:54:22.172 INFO:teuthology.orchestra.run.trial198.stdout:Nothing to do. 2026-02-23T10:54:22.172 INFO:teuthology.orchestra.run.trial198.stdout:Complete! 2026-02-23T10:54:22.194 DEBUG:teuthology.orchestra.run.trial198:> sudo yum -y install ceph-osd 2026-02-23T10:54:22.503 INFO:teuthology.orchestra.run.trial198.stdout:Last metadata expiration check: 0:00:06 ago on Mon 23 Feb 2026 10:54:16 AM UTC. 2026-02-23T10:54:22.567 INFO:teuthology.orchestra.run.trial198.stdout:Package ceph-osd-2:20.3.0-5439.ge371433f.el9.x86_64 is already installed. 2026-02-23T10:54:22.585 INFO:teuthology.orchestra.run.trial198.stdout:Dependencies resolved. 2026-02-23T10:54:22.585 INFO:teuthology.orchestra.run.trial198.stdout:Nothing to do. 2026-02-23T10:54:22.585 INFO:teuthology.orchestra.run.trial198.stdout:Complete! 2026-02-23T10:54:22.606 DEBUG:teuthology.orchestra.run.trial198:> sudo yum -y install ceph-osd-classic 2026-02-23T10:54:22.913 INFO:teuthology.orchestra.run.trial198.stdout:Last metadata expiration check: 0:00:06 ago on Mon 23 Feb 2026 10:54:16 AM UTC. 2026-02-23T10:54:22.978 INFO:teuthology.orchestra.run.trial198.stdout:Package ceph-osd-classic-2:20.3.0-5439.ge371433f.el9.x86_64 is already installed. 2026-02-23T10:54:22.996 INFO:teuthology.orchestra.run.trial198.stdout:Dependencies resolved. 2026-02-23T10:54:22.997 INFO:teuthology.orchestra.run.trial198.stdout:Nothing to do. 2026-02-23T10:54:22.997 INFO:teuthology.orchestra.run.trial198.stdout:Complete! 2026-02-23T10:54:23.019 DEBUG:teuthology.orchestra.run.trial198:> sudo yum -y install ceph-fuse 2026-02-23T10:54:23.332 INFO:teuthology.orchestra.run.trial198.stdout:Last metadata expiration check: 0:00:07 ago on Mon 23 Feb 2026 10:54:16 AM UTC. 2026-02-23T10:54:23.397 INFO:teuthology.orchestra.run.trial198.stdout:Package ceph-fuse-2:20.3.0-5439.ge371433f.el9.x86_64 is already installed. 2026-02-23T10:54:23.416 INFO:teuthology.orchestra.run.trial198.stdout:Dependencies resolved. 2026-02-23T10:54:23.417 INFO:teuthology.orchestra.run.trial198.stdout:Nothing to do. 2026-02-23T10:54:23.417 INFO:teuthology.orchestra.run.trial198.stdout:Complete! 2026-02-23T10:54:23.438 DEBUG:teuthology.orchestra.run.trial198:> sudo yum -y install ceph-volume 2026-02-23T10:54:23.751 INFO:teuthology.orchestra.run.trial198.stdout:Last metadata expiration check: 0:00:07 ago on Mon 23 Feb 2026 10:54:16 AM UTC. 2026-02-23T10:54:23.814 INFO:teuthology.orchestra.run.trial198.stdout:Package ceph-volume-2:20.3.0-5439.ge371433f.el9.noarch is already installed. 2026-02-23T10:54:23.833 INFO:teuthology.orchestra.run.trial198.stdout:Dependencies resolved. 2026-02-23T10:54:23.834 INFO:teuthology.orchestra.run.trial198.stdout:Nothing to do. 2026-02-23T10:54:23.834 INFO:teuthology.orchestra.run.trial198.stdout:Complete! 2026-02-23T10:54:23.855 DEBUG:teuthology.orchestra.run.trial198:> sudo yum -y install librados-devel 2026-02-23T10:54:24.160 INFO:teuthology.orchestra.run.trial198.stdout:Last metadata expiration check: 0:00:08 ago on Mon 23 Feb 2026 10:54:16 AM UTC. 2026-02-23T10:54:24.224 INFO:teuthology.orchestra.run.trial198.stdout:Package librados-devel-2:20.3.0-5439.ge371433f.el9.x86_64 is already installed. 2026-02-23T10:54:24.242 INFO:teuthology.orchestra.run.trial198.stdout:Dependencies resolved. 2026-02-23T10:54:24.243 INFO:teuthology.orchestra.run.trial198.stdout:Nothing to do. 2026-02-23T10:54:24.243 INFO:teuthology.orchestra.run.trial198.stdout:Complete! 2026-02-23T10:54:24.264 DEBUG:teuthology.orchestra.run.trial198:> sudo yum -y install libcephfs2 2026-02-23T10:54:24.568 INFO:teuthology.orchestra.run.trial198.stdout:Last metadata expiration check: 0:00:08 ago on Mon 23 Feb 2026 10:54:16 AM UTC. 2026-02-23T10:54:24.632 INFO:teuthology.orchestra.run.trial198.stdout:Package libcephfs2-2:20.3.0-5439.ge371433f.el9.x86_64 is already installed. 2026-02-23T10:54:24.650 INFO:teuthology.orchestra.run.trial198.stdout:Dependencies resolved. 2026-02-23T10:54:24.651 INFO:teuthology.orchestra.run.trial198.stdout:Nothing to do. 2026-02-23T10:54:24.651 INFO:teuthology.orchestra.run.trial198.stdout:Complete! 2026-02-23T10:54:24.672 DEBUG:teuthology.orchestra.run.trial198:> sudo yum -y install libcephfs-devel 2026-02-23T10:54:24.980 INFO:teuthology.orchestra.run.trial198.stdout:Last metadata expiration check: 0:00:08 ago on Mon 23 Feb 2026 10:54:16 AM UTC. 2026-02-23T10:54:25.045 INFO:teuthology.orchestra.run.trial198.stdout:Package libcephfs-devel-2:20.3.0-5439.ge371433f.el9.x86_64 is already installed. 2026-02-23T10:54:25.064 INFO:teuthology.orchestra.run.trial198.stdout:Dependencies resolved. 2026-02-23T10:54:25.065 INFO:teuthology.orchestra.run.trial198.stdout:Nothing to do. 2026-02-23T10:54:25.065 INFO:teuthology.orchestra.run.trial198.stdout:Complete! 2026-02-23T10:54:25.085 DEBUG:teuthology.orchestra.run.trial198:> sudo yum -y install librados2 2026-02-23T10:54:25.401 INFO:teuthology.orchestra.run.trial198.stdout:Last metadata expiration check: 0:00:09 ago on Mon 23 Feb 2026 10:54:16 AM UTC. 2026-02-23T10:54:25.467 INFO:teuthology.orchestra.run.trial198.stdout:Package librados2-2:20.3.0-5439.ge371433f.el9.x86_64 is already installed. 2026-02-23T10:54:25.486 INFO:teuthology.orchestra.run.trial198.stdout:Dependencies resolved. 2026-02-23T10:54:25.486 INFO:teuthology.orchestra.run.trial198.stdout:Nothing to do. 2026-02-23T10:54:25.486 INFO:teuthology.orchestra.run.trial198.stdout:Complete! 2026-02-23T10:54:25.510 DEBUG:teuthology.orchestra.run.trial198:> sudo yum -y install librbd1 2026-02-23T10:54:25.817 INFO:teuthology.orchestra.run.trial198.stdout:Last metadata expiration check: 0:00:09 ago on Mon 23 Feb 2026 10:54:16 AM UTC. 2026-02-23T10:54:25.882 INFO:teuthology.orchestra.run.trial198.stdout:Package librbd1-2:20.3.0-5439.ge371433f.el9.x86_64 is already installed. 2026-02-23T10:54:25.900 INFO:teuthology.orchestra.run.trial198.stdout:Dependencies resolved. 2026-02-23T10:54:25.901 INFO:teuthology.orchestra.run.trial198.stdout:Nothing to do. 2026-02-23T10:54:25.901 INFO:teuthology.orchestra.run.trial198.stdout:Complete! 2026-02-23T10:54:25.923 DEBUG:teuthology.orchestra.run.trial198:> sudo yum -y install python3-rados 2026-02-23T10:54:26.230 INFO:teuthology.orchestra.run.trial198.stdout:Last metadata expiration check: 0:00:10 ago on Mon 23 Feb 2026 10:54:16 AM UTC. 2026-02-23T10:54:26.294 INFO:teuthology.orchestra.run.trial198.stdout:Package python3-rados-2:20.3.0-5439.ge371433f.el9.x86_64 is already installed. 2026-02-23T10:54:26.312 INFO:teuthology.orchestra.run.trial198.stdout:Dependencies resolved. 2026-02-23T10:54:26.313 INFO:teuthology.orchestra.run.trial198.stdout:Nothing to do. 2026-02-23T10:54:26.313 INFO:teuthology.orchestra.run.trial198.stdout:Complete! 2026-02-23T10:54:26.334 DEBUG:teuthology.orchestra.run.trial198:> sudo yum -y install python3-rgw 2026-02-23T10:54:26.639 INFO:teuthology.orchestra.run.trial198.stdout:Last metadata expiration check: 0:00:10 ago on Mon 23 Feb 2026 10:54:16 AM UTC. 2026-02-23T10:54:26.703 INFO:teuthology.orchestra.run.trial198.stdout:Package python3-rgw-2:20.3.0-5439.ge371433f.el9.x86_64 is already installed. 2026-02-23T10:54:26.721 INFO:teuthology.orchestra.run.trial198.stdout:Dependencies resolved. 2026-02-23T10:54:26.722 INFO:teuthology.orchestra.run.trial198.stdout:Nothing to do. 2026-02-23T10:54:26.722 INFO:teuthology.orchestra.run.trial198.stdout:Complete! 2026-02-23T10:54:26.743 DEBUG:teuthology.orchestra.run.trial198:> sudo yum -y install python3-cephfs 2026-02-23T10:54:27.049 INFO:teuthology.orchestra.run.trial198.stdout:Last metadata expiration check: 0:00:11 ago on Mon 23 Feb 2026 10:54:16 AM UTC. 2026-02-23T10:54:27.114 INFO:teuthology.orchestra.run.trial198.stdout:Package python3-cephfs-2:20.3.0-5439.ge371433f.el9.x86_64 is already installed. 2026-02-23T10:54:27.132 INFO:teuthology.orchestra.run.trial198.stdout:Dependencies resolved. 2026-02-23T10:54:27.132 INFO:teuthology.orchestra.run.trial198.stdout:Nothing to do. 2026-02-23T10:54:27.132 INFO:teuthology.orchestra.run.trial198.stdout:Complete! 2026-02-23T10:54:27.154 DEBUG:teuthology.orchestra.run.trial198:> sudo yum -y install python3-rbd 2026-02-23T10:54:27.463 INFO:teuthology.orchestra.run.trial198.stdout:Last metadata expiration check: 0:00:11 ago on Mon 23 Feb 2026 10:54:16 AM UTC. 2026-02-23T10:54:27.527 INFO:teuthology.orchestra.run.trial198.stdout:Package python3-rbd-2:20.3.0-5439.ge371433f.el9.x86_64 is already installed. 2026-02-23T10:54:27.545 INFO:teuthology.orchestra.run.trial198.stdout:Dependencies resolved. 2026-02-23T10:54:27.546 INFO:teuthology.orchestra.run.trial198.stdout:Nothing to do. 2026-02-23T10:54:27.546 INFO:teuthology.orchestra.run.trial198.stdout:Complete! 2026-02-23T10:54:27.566 DEBUG:teuthology.orchestra.run.trial198:> sudo yum -y install rbd-fuse 2026-02-23T10:54:27.880 INFO:teuthology.orchestra.run.trial198.stdout:Last metadata expiration check: 0:00:11 ago on Mon 23 Feb 2026 10:54:16 AM UTC. 2026-02-23T10:54:27.945 INFO:teuthology.orchestra.run.trial198.stdout:Package rbd-fuse-2:20.3.0-5439.ge371433f.el9.x86_64 is already installed. 2026-02-23T10:54:27.963 INFO:teuthology.orchestra.run.trial198.stdout:Dependencies resolved. 2026-02-23T10:54:27.963 INFO:teuthology.orchestra.run.trial198.stdout:Nothing to do. 2026-02-23T10:54:27.963 INFO:teuthology.orchestra.run.trial198.stdout:Complete! 2026-02-23T10:54:27.985 DEBUG:teuthology.orchestra.run.trial198:> sudo yum -y install rbd-mirror 2026-02-23T10:54:28.290 INFO:teuthology.orchestra.run.trial198.stdout:Last metadata expiration check: 0:00:12 ago on Mon 23 Feb 2026 10:54:16 AM UTC. 2026-02-23T10:54:28.355 INFO:teuthology.orchestra.run.trial198.stdout:Package rbd-mirror-2:20.3.0-5439.ge371433f.el9.x86_64 is already installed. 2026-02-23T10:54:28.373 INFO:teuthology.orchestra.run.trial198.stdout:Dependencies resolved. 2026-02-23T10:54:28.374 INFO:teuthology.orchestra.run.trial198.stdout:Nothing to do. 2026-02-23T10:54:28.374 INFO:teuthology.orchestra.run.trial198.stdout:Complete! 2026-02-23T10:54:28.395 DEBUG:teuthology.orchestra.run.trial198:> sudo yum -y install rbd-nbd 2026-02-23T10:54:28.704 INFO:teuthology.orchestra.run.trial198.stdout:Last metadata expiration check: 0:00:12 ago on Mon 23 Feb 2026 10:54:16 AM UTC. 2026-02-23T10:54:28.769 INFO:teuthology.orchestra.run.trial198.stdout:Package rbd-nbd-2:20.3.0-5439.ge371433f.el9.x86_64 is already installed. 2026-02-23T10:54:28.787 INFO:teuthology.orchestra.run.trial198.stdout:Dependencies resolved. 2026-02-23T10:54:28.788 INFO:teuthology.orchestra.run.trial198.stdout:Nothing to do. 2026-02-23T10:54:28.788 INFO:teuthology.orchestra.run.trial198.stdout:Complete! 2026-02-23T10:54:28.811 DEBUG:teuthology.parallel:result is None 2026-02-23T10:54:28.811 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=debug&distros=centos%2F9%2Fx86_64&sha1=e371433fcd1390a0b82cfb18effc422c1f5d8c1d 2026-02-23T10:54:28.970 DEBUG:teuthology.orchestra.run.trial198:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2026-02-23T10:54:28.992 INFO:teuthology.orchestra.run.trial198.stdout:20.3.0-5439.ge371433f.el9 2026-02-23T10:54:28.992 INFO:teuthology.packaging:The installed version of ceph is 20.3.0-5439.ge371433f.el9 2026-02-23T10:54:28.992 INFO:teuthology.task.install:The correct ceph version 20.3.0-5439.ge371433f is installed. 2026-02-23T10:54:28.995 INFO:teuthology.task.install.util:Utilities already shipped, skip it... 2026-02-23T10:54:28.995 INFO:teuthology.run_tasks:Running task ceph... 2026-02-23T10:54:29.095 INFO:tasks.ceph:Making ceph log dir writeable by non-root... 2026-02-23T10:54:29.095 DEBUG:teuthology.orchestra.run.trial198:> sudo chmod 777 /var/log/ceph 2026-02-23T10:54:29.121 INFO:tasks.ceph:Disabling ceph logrotate... 2026-02-23T10:54:29.121 DEBUG:teuthology.orchestra.run.trial198:> sudo rm -f -- /etc/logrotate.d/ceph 2026-02-23T10:54:29.185 INFO:tasks.ceph:Creating extra log directories... 2026-02-23T10:54:29.185 DEBUG:teuthology.orchestra.run.trial198:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2026-02-23T10:54:29.261 INFO:tasks.ceph:Creating ceph cluster ceph... 2026-02-23T10:54:29.261 INFO:tasks.ceph:config {'conf': {'mon': {'mon min osdmap epochs': 50, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'paxos service trim min': 10, 'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'osd pool default crimson': True}, 'osd': {'debug monc': 20, 'bluestore block size': 96636764160, 'bluestore compression algorithm': 'snappy', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'bluestore rocksdb cf': False, 'crimson cpu num': 4, 'crimson osd obc lru size': 10, 'debug alienstore': 20, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 20, 'debug osd': 20, 'debug rocksdb': 10, 'err to stderr': True, 'log flush on exit': True, 'log to file': False, 'log to stderr': True, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd shutdown pgref assert': True, 'seastore cachepin size pershard': '64M', 'seastore max concurrent transactions': 8}, 'global': {'enable experimental unrecoverable data corrupting features': 'crimson', 'ms client mode': 'crc', 'ms cluster mode': 'crc', 'ms mon client mode': 'crc', 'ms mon cluster mode': 'crc', 'ms mon service mode': 'crc', 'ms service mode': 'crc'}, 'mgr': {'debug mgr': 20, 'debug ms': 1}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(POOL_APP_NOT_ENABLED\\)'], 'cpu_profile': set(), 'cluster': 'ceph', 'mon_bind_msgr2': True, 'mon_bind_addrvec': True} 2026-02-23T10:54:29.262 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/mnt/teuthology/shraddhaag-2026-02-23_09:58:48-crimson-rados-main-distro-debug-trial/66138', 'branch': 'main', 'description': 'crimson-rados/rbd/{0-crimson_install clusters/crimson-fixed-1 crimson-supported-all-distro/centos_latest crimson_qa_overrides deploy/ceph objectstore/crimson_bluestore tasks/rbd_cls_tests}', 'email': None, 'first_in_suite': False, 'flavor': 'debug', 'job_id': '66138', 'kernel': {'branch': 'distro', 'kdb': True, 'sha1': 'distro'}, 'last_in_suite': False, 'machine_type': 'trial', 'name': 'shraddhaag-2026-02-23_09:58:48-crimson-rados-main-distro-debug-trial', 'no_nested_subset': False, 'os_type': 'centos', 'os_version': '9.stream', 'overrides': {'admin_socket': {'branch': 'main'}, 'ceph': {'conf': {'global': {'enable experimental unrecoverable data corrupting features': 'crimson', 'ms client mode': 'crc', 'ms cluster mode': 'crc', 'ms mon client mode': 'crc', 'ms mon cluster mode': 'crc', 'ms mon service mode': 'crc', 'ms service mode': 'crc'}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'osd pool default crimson': True}, 'osd': {'bluestore block size': 96636764160, 'bluestore compression algorithm': 'snappy', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'bluestore rocksdb cf': False, 'crimson cpu num': 4, 'crimson osd obc lru size': 10, 'debug alienstore': 20, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 20, 'debug osd': 20, 'debug rocksdb': 10, 'err to stderr': True, 'log flush on exit': True, 'log to file': False, 'log to stderr': True, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd shutdown pgref assert': True, 'seastore cachepin size pershard': '64M', 'seastore max concurrent transactions': 8}}, 'crimson_compat': True, 'flavor': 'debug', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(POOL_APP_NOT_ENABLED\\)'], 'sha1': 'e371433fcd1390a0b82cfb18effc422c1f5d8c1d'}, 'ceph-deploy': {'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'global': {'osd crush chooseleaf type': 0, 'osd pool default pg num': 128, 'osd pool default pgp num': 128, 'osd pool default size': 2}, 'mon': {}}}, 'install': {'ceph': {'flavor': 'debug', 'sha1': 'e371433fcd1390a0b82cfb18effc422c1f5d8c1d'}}, 'workunit': {'branch': 'wip-shraddhaag-enable-cephadm-tests', 'env': {'CRIMSON_COMPAT': '1'}, 'sha1': 'e371433fcd1390a0b82cfb18effc422c1f5d8c1d'}}, 'owner': 'scheduled_shraddhaag@soko04.front.sepia.ceph.com', 'priority': 49, 'repo': 'https://git.ceph.com/ceph-ci.git', 'roles': [['mon.a', 'mgr.x', 'osd.0', 'osd.1', 'osd.2', 'client.0']], 'seed': 66, 'sha1': 'e371433fcd1390a0b82cfb18effc422c1f5d8c1d', 'sleep_before_teardown': 0, 'suite': 'crimson-rados', 'suite_branch': 'wip-shraddhaag-enable-cephadm-tests', 'suite_path': '/home/teuthworker/src/github.com_shraddhaag_ceph_e371433fcd1390a0b82cfb18effc422c1f5d8c1d/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://github.com/shraddhaag/ceph.git', 'suite_sha1': 'e371433fcd1390a0b82cfb18effc422c1f5d8c1d', 'targets': {'trial198.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBHtvwhffeaME518p75g+XtQwTPPiA5DEOAj0oujaqjsIvUXvQpVvVpvPzoBOuIouJhlPNX1VUcy0PDrX8gWDbq0='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'branch': 'distro', 'kdb': True, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': {'extra_packages': {'deb': ['ceph-osd-crimson', 'ceph-osd-crimson-dbg'], 'rpm': ['ceph-osd-crimson', 'ceph-osd-crimson-debuginfo']}, 'flavor': 'debug', 'sha1': 'e371433fcd1390a0b82cfb18effc422c1f5d8c1d'}}, {'pexec': {'all': ['sudo update-alternatives --set ceph-osd /usr/bin/ceph-osd-crimson']}}, {'install': None}, {'ceph': {'conf': {'mon': {'mon min osdmap epochs': 50, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'paxos service trim min': 10, 'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'osd pool default crimson': True}, 'osd': {'debug monc': 20, 'bluestore block size': 96636764160, 'bluestore compression algorithm': 'snappy', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'bluestore rocksdb cf': False, 'crimson cpu num': 4, 'crimson osd obc lru size': 10, 'debug alienstore': 20, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 20, 'debug osd': 20, 'debug rocksdb': 10, 'err to stderr': True, 'log flush on exit': True, 'log to file': False, 'log to stderr': True, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd shutdown pgref assert': True, 'seastore cachepin size pershard': '64M', 'seastore max concurrent transactions': 8}, 'global': {'enable experimental unrecoverable data corrupting features': 'crimson', 'ms client mode': 'crc', 'ms cluster mode': 'crc', 'ms mon client mode': 'crc', 'ms mon cluster mode': 'crc', 'ms mon service mode': 'crc', 'ms service mode': 'crc'}, 'mgr': {'debug mgr': 20, 'debug ms': 1}}, 'crimson_compat': True, 'flavor': 'debug', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(POOL_APP_NOT_ENABLED\\)'], 'sha1': 'e371433fcd1390a0b82cfb18effc422c1f5d8c1d', 'cluster': 'ceph'}}, {'workunit': {'clients': {'client.0': ['cls/test_cls_rbd.sh', 'cls/test_cls_lock.sh', 'cls/test_cls_journal.sh']}}}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': []}, 'teuthology_branch': 'main', 'teuthology_sha1': 'fa17720d0088c3ac28e473468bfc79eeaff5cd38', 'timestamp': '2026-02-23_09:58:48', 'tube': 'trial', 'user': 'shraddhaag', 'verbose': False, 'worker_log': '/home/teuthworker/mnt/teuthology/worker_logs/dispatcher.trial.3318653'} 2026-02-23T10:54:29.262 DEBUG:teuthology.orchestra.run.trial198:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2026-02-23T10:54:29.309 DEBUG:teuthology.orchestra.run.trial198:> sudo install -d -m0777 -- /var/run/ceph 2026-02-23T10:54:29.374 DEBUG:teuthology.orchestra.run.trial198:> set -ex 2026-02-23T10:54:29.374 DEBUG:teuthology.orchestra.run.trial198:> dd if=/scratch_devs of=/dev/stdout 2026-02-23T10:54:29.430 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2026-02-23T10:54:29.430 DEBUG:teuthology.orchestra.run.trial198:> stat /dev/vg_nvme/lv_1 2026-02-23T10:54:29.486 INFO:teuthology.orchestra.run.trial198.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2026-02-23T10:54:29.486 INFO:teuthology.orchestra.run.trial198.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2026-02-23T10:54:29.486 INFO:teuthology.orchestra.run.trial198.stdout:Device: 6h/6d Inode: 909 Links: 1 2026-02-23T10:54:29.486 INFO:teuthology.orchestra.run.trial198.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2026-02-23T10:54:29.486 INFO:teuthology.orchestra.run.trial198.stdout:Context: system_u:object_r:device_t:s0 2026-02-23T10:54:29.486 INFO:teuthology.orchestra.run.trial198.stdout:Access: 2026-02-23 10:52:15.132438570 +0000 2026-02-23T10:54:29.486 INFO:teuthology.orchestra.run.trial198.stdout:Modify: 2026-02-23 10:49:45.564992450 +0000 2026-02-23T10:54:29.486 INFO:teuthology.orchestra.run.trial198.stdout:Change: 2026-02-23 10:49:45.564992450 +0000 2026-02-23T10:54:29.486 INFO:teuthology.orchestra.run.trial198.stdout: Birth: 2026-02-23 10:49:45.564992450 +0000 2026-02-23T10:54:29.487 DEBUG:teuthology.orchestra.run.trial198:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2026-02-23T10:54:29.548 INFO:teuthology.orchestra.run.trial198.stderr:1+0 records in 2026-02-23T10:54:29.548 INFO:teuthology.orchestra.run.trial198.stderr:1+0 records out 2026-02-23T10:54:29.548 INFO:teuthology.orchestra.run.trial198.stderr:512 bytes copied, 0.000142394 s, 3.6 MB/s 2026-02-23T10:54:29.550 DEBUG:teuthology.orchestra.run.trial198:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2026-02-23T10:54:29.607 DEBUG:teuthology.orchestra.run.trial198:> stat /dev/vg_nvme/lv_2 2026-02-23T10:54:29.665 INFO:teuthology.orchestra.run.trial198.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2026-02-23T10:54:29.665 INFO:teuthology.orchestra.run.trial198.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2026-02-23T10:54:29.665 INFO:teuthology.orchestra.run.trial198.stdout:Device: 6h/6d Inode: 935 Links: 1 2026-02-23T10:54:29.666 INFO:teuthology.orchestra.run.trial198.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2026-02-23T10:54:29.666 INFO:teuthology.orchestra.run.trial198.stdout:Context: system_u:object_r:device_t:s0 2026-02-23T10:54:29.666 INFO:teuthology.orchestra.run.trial198.stdout:Access: 2026-02-23 10:52:15.132438570 +0000 2026-02-23T10:54:29.666 INFO:teuthology.orchestra.run.trial198.stdout:Modify: 2026-02-23 10:49:45.784983908 +0000 2026-02-23T10:54:29.666 INFO:teuthology.orchestra.run.trial198.stdout:Change: 2026-02-23 10:49:45.784983908 +0000 2026-02-23T10:54:29.666 INFO:teuthology.orchestra.run.trial198.stdout: Birth: 2026-02-23 10:49:45.784983908 +0000 2026-02-23T10:54:29.666 DEBUG:teuthology.orchestra.run.trial198:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2026-02-23T10:54:29.727 INFO:teuthology.orchestra.run.trial198.stderr:1+0 records in 2026-02-23T10:54:29.727 INFO:teuthology.orchestra.run.trial198.stderr:1+0 records out 2026-02-23T10:54:29.727 INFO:teuthology.orchestra.run.trial198.stderr:512 bytes copied, 0.000141663 s, 3.6 MB/s 2026-02-23T10:54:29.729 DEBUG:teuthology.orchestra.run.trial198:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2026-02-23T10:54:29.785 DEBUG:teuthology.orchestra.run.trial198:> stat /dev/vg_nvme/lv_3 2026-02-23T10:54:29.843 INFO:teuthology.orchestra.run.trial198.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2026-02-23T10:54:29.843 INFO:teuthology.orchestra.run.trial198.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2026-02-23T10:54:29.843 INFO:teuthology.orchestra.run.trial198.stdout:Device: 6h/6d Inode: 967 Links: 1 2026-02-23T10:54:29.843 INFO:teuthology.orchestra.run.trial198.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2026-02-23T10:54:29.843 INFO:teuthology.orchestra.run.trial198.stdout:Context: system_u:object_r:device_t:s0 2026-02-23T10:54:29.843 INFO:teuthology.orchestra.run.trial198.stdout:Access: 2026-02-23 10:52:15.132438570 +0000 2026-02-23T10:54:29.844 INFO:teuthology.orchestra.run.trial198.stdout:Modify: 2026-02-23 10:49:46.028974434 +0000 2026-02-23T10:54:29.844 INFO:teuthology.orchestra.run.trial198.stdout:Change: 2026-02-23 10:49:46.028974434 +0000 2026-02-23T10:54:29.844 INFO:teuthology.orchestra.run.trial198.stdout: Birth: 2026-02-23 10:49:46.028974434 +0000 2026-02-23T10:54:29.844 DEBUG:teuthology.orchestra.run.trial198:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2026-02-23T10:54:29.907 INFO:teuthology.orchestra.run.trial198.stderr:1+0 records in 2026-02-23T10:54:29.907 INFO:teuthology.orchestra.run.trial198.stderr:1+0 records out 2026-02-23T10:54:29.908 INFO:teuthology.orchestra.run.trial198.stderr:512 bytes copied, 0.0001458 s, 3.5 MB/s 2026-02-23T10:54:29.909 DEBUG:teuthology.orchestra.run.trial198:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2026-02-23T10:54:29.967 DEBUG:teuthology.orchestra.run.trial198:> stat /dev/vg_nvme/lv_4 2026-02-23T10:54:30.025 INFO:teuthology.orchestra.run.trial198.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2026-02-23T10:54:30.026 INFO:teuthology.orchestra.run.trial198.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2026-02-23T10:54:30.026 INFO:teuthology.orchestra.run.trial198.stdout:Device: 6h/6d Inode: 993 Links: 1 2026-02-23T10:54:30.026 INFO:teuthology.orchestra.run.trial198.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2026-02-23T10:54:30.026 INFO:teuthology.orchestra.run.trial198.stdout:Context: system_u:object_r:device_t:s0 2026-02-23T10:54:30.026 INFO:teuthology.orchestra.run.trial198.stdout:Access: 2026-02-23 10:52:15.133438533 +0000 2026-02-23T10:54:30.026 INFO:teuthology.orchestra.run.trial198.stdout:Modify: 2026-02-23 10:49:46.275964843 +0000 2026-02-23T10:54:30.026 INFO:teuthology.orchestra.run.trial198.stdout:Change: 2026-02-23 10:49:46.275964843 +0000 2026-02-23T10:54:30.026 INFO:teuthology.orchestra.run.trial198.stdout: Birth: 2026-02-23 10:49:46.275964843 +0000 2026-02-23T10:54:30.027 DEBUG:teuthology.orchestra.run.trial198:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2026-02-23T10:54:30.088 INFO:teuthology.orchestra.run.trial198.stderr:1+0 records in 2026-02-23T10:54:30.088 INFO:teuthology.orchestra.run.trial198.stderr:1+0 records out 2026-02-23T10:54:30.089 INFO:teuthology.orchestra.run.trial198.stderr:512 bytes copied, 0.000150139 s, 3.4 MB/s 2026-02-23T10:54:30.090 DEBUG:teuthology.orchestra.run.trial198:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2026-02-23T10:54:30.147 INFO:tasks.ceph:osd dev map: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2026-02-23T10:54:30.147 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@trial198.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'}} 2026-02-23T10:54:30.147 INFO:tasks.ceph:Generating config... 2026-02-23T10:54:30.149 INFO:tasks.ceph:[mon] mon min osdmap epochs = 50 2026-02-23T10:54:30.150 INFO:tasks.ceph:[mon] mon osdmap full prune interval = 2 2026-02-23T10:54:30.150 INFO:tasks.ceph:[mon] mon osdmap full prune min = 15 2026-02-23T10:54:30.150 INFO:tasks.ceph:[mon] mon osdmap full prune txsize = 2 2026-02-23T10:54:30.150 INFO:tasks.ceph:[mon] paxos service trim min = 10 2026-02-23T10:54:30.150 INFO:tasks.ceph:[mon] debug mon = 20 2026-02-23T10:54:30.150 INFO:tasks.ceph:[mon] debug ms = 1 2026-02-23T10:54:30.150 INFO:tasks.ceph:[mon] debug paxos = 20 2026-02-23T10:54:30.150 INFO:tasks.ceph:[mon] osd pool default crimson = True 2026-02-23T10:54:30.150 INFO:tasks.ceph:[osd] debug monc = 20 2026-02-23T10:54:30.150 INFO:tasks.ceph:[osd] bluestore block size = 96636764160 2026-02-23T10:54:30.151 INFO:tasks.ceph:[osd] bluestore compression algorithm = snappy 2026-02-23T10:54:30.151 INFO:tasks.ceph:[osd] bluestore compression mode = aggressive 2026-02-23T10:54:30.151 INFO:tasks.ceph:[osd] bluestore fsck on mount = True 2026-02-23T10:54:30.151 INFO:tasks.ceph:[osd] bluestore rocksdb cf = False 2026-02-23T10:54:30.151 INFO:tasks.ceph:[osd] crimson cpu num = 4 2026-02-23T10:54:30.151 INFO:tasks.ceph:[osd] crimson osd obc lru size = 10 2026-02-23T10:54:30.151 INFO:tasks.ceph:[osd] debug alienstore = 20 2026-02-23T10:54:30.151 INFO:tasks.ceph:[osd] debug bluefs = 20 2026-02-23T10:54:30.151 INFO:tasks.ceph:[osd] debug bluestore = 20 2026-02-23T10:54:30.151 INFO:tasks.ceph:[osd] debug ms = 20 2026-02-23T10:54:30.151 INFO:tasks.ceph:[osd] debug osd = 20 2026-02-23T10:54:30.151 INFO:tasks.ceph:[osd] debug rocksdb = 10 2026-02-23T10:54:30.152 INFO:tasks.ceph:[osd] err to stderr = True 2026-02-23T10:54:30.152 INFO:tasks.ceph:[osd] log flush on exit = True 2026-02-23T10:54:30.152 INFO:tasks.ceph:[osd] log to file = False 2026-02-23T10:54:30.152 INFO:tasks.ceph:[osd] log to stderr = True 2026-02-23T10:54:30.152 INFO:tasks.ceph:[osd] mon osd backfillfull_ratio = 0.85 2026-02-23T10:54:30.152 INFO:tasks.ceph:[osd] mon osd full ratio = 0.9 2026-02-23T10:54:30.152 INFO:tasks.ceph:[osd] mon osd nearfull ratio = 0.8 2026-02-23T10:54:30.152 INFO:tasks.ceph:[osd] osd failsafe full ratio = 0.95 2026-02-23T10:54:30.152 INFO:tasks.ceph:[osd] osd objectstore = bluestore 2026-02-23T10:54:30.152 INFO:tasks.ceph:[osd] osd shutdown pgref assert = True 2026-02-23T10:54:30.152 INFO:tasks.ceph:[osd] seastore cachepin size pershard = 64M 2026-02-23T10:54:30.152 INFO:tasks.ceph:[osd] seastore max concurrent transactions = 8 2026-02-23T10:54:30.153 INFO:tasks.ceph:[global] enable experimental unrecoverable data corrupting features = crimson 2026-02-23T10:54:30.153 INFO:tasks.ceph:[global] ms client mode = crc 2026-02-23T10:54:30.153 INFO:tasks.ceph:[global] ms cluster mode = crc 2026-02-23T10:54:30.153 INFO:tasks.ceph:[global] ms mon client mode = crc 2026-02-23T10:54:30.153 INFO:tasks.ceph:[global] ms mon cluster mode = crc 2026-02-23T10:54:30.153 INFO:tasks.ceph:[global] ms mon service mode = crc 2026-02-23T10:54:30.153 INFO:tasks.ceph:[global] ms service mode = crc 2026-02-23T10:54:30.153 INFO:tasks.ceph:[mgr] debug mgr = 20 2026-02-23T10:54:30.153 INFO:tasks.ceph:[mgr] debug ms = 1 2026-02-23T10:54:30.153 INFO:tasks.ceph:Setting up mon.a... 2026-02-23T10:54:30.154 DEBUG:teuthology.orchestra.run.trial198:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/ceph.keyring 2026-02-23T10:54:30.227 INFO:teuthology.orchestra.run.trial198.stdout:creating /etc/ceph/ceph.keyring 2026-02-23T10:54:30.230 DEBUG:teuthology.orchestra.run.trial198:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/ceph.keyring 2026-02-23T10:54:30.315 DEBUG:teuthology.orchestra.run.trial198:> sudo chmod 0644 /etc/ceph/ceph.keyring 2026-02-23T10:54:30.381 DEBUG:tasks.ceph:Ceph mon addresses: [('mon.a', '10.20.193.198')] 2026-02-23T10:54:30.382 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'cephx', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=isa technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '10.20.193.198', 'enable experimental unrecoverable data corrupting features': 'crimson', 'ms client mode': 'crc', 'ms cluster mode': 'crc', 'ms mon client mode': 'crc', 'ms mon cluster mode': 'crc', 'ms mon service mode': 'crc', 'ms service mode': 'crc'}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd mclock profile': 'high_recovery_ops', 'osd mclock skip benchmark': 'true', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd debug trim objects': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'debug monc': 20, 'bluestore block size': 96636764160, 'bluestore compression algorithm': 'snappy', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'bluestore rocksdb cf': False, 'crimson cpu num': 4, 'crimson osd obc lru size': 10, 'debug alienstore': 20, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 20, 'debug osd': 20, 'debug rocksdb': 10, 'err to stderr': True, 'log flush on exit': True, 'log to file': False, 'log to stderr': True, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd shutdown pgref assert': True, 'seastore cachepin size pershard': '64M', 'seastore max concurrent transactions': 8}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false'}, 'mon': {'debug ms': 1, 'debug mon': 20, 'debug paxos': 20, 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'mon_down_mkfs_grace': '2m', 'mon_warn_on_filestore_osds': 'false', 'mon min osdmap epochs': 50, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'paxos service trim min': 10, 'osd pool default crimson': True}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok'}, 'mon.a': {}} 2026-02-23T10:54:30.384 DEBUG:teuthology.orchestra.run.trial198:> set -ex 2026-02-23T10:54:30.384 DEBUG:teuthology.orchestra.run.trial198:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2026-02-23T10:54:30.436 DEBUG:teuthology.orchestra.run.trial198:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --add a 10.20.193.198 --print /home/ubuntu/cephtest/ceph.monmap 2026-02-23T10:54:30.516 INFO:teuthology.orchestra.run.trial198.stderr:2026-02-23T10:54:30.514+0000 7f3ac042af40 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T10:54:30.517 INFO:teuthology.orchestra.run.trial198.stderr:2026-02-23T10:54:30.514+0000 7f3ac042af40 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T10:54:30.517 INFO:teuthology.orchestra.run.trial198.stderr:2026-02-23T10:54:30.515+0000 7f3ac042af40 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T10:54:30.517 INFO:teuthology.orchestra.run.trial198.stdout:monmaptool: monmap file /home/ubuntu/cephtest/ceph.monmap 2026-02-23T10:54:30.517 INFO:teuthology.orchestra.run.trial198.stdout:monmaptool: generated fsid 8ebafdf4-9523-4bab-a465-a90635e02d85 2026-02-23T10:54:30.517 INFO:teuthology.orchestra.run.trial198.stdout:setting min_mon_release = tentacle 2026-02-23T10:54:30.517 INFO:teuthology.orchestra.run.trial198.stdout:epoch 0 2026-02-23T10:54:30.517 INFO:teuthology.orchestra.run.trial198.stdout:fsid 8ebafdf4-9523-4bab-a465-a90635e02d85 2026-02-23T10:54:30.517 INFO:teuthology.orchestra.run.trial198.stdout:last_changed 2026-02-23T10:54:30.516866+0000 2026-02-23T10:54:30.517 INFO:teuthology.orchestra.run.trial198.stdout:created 2026-02-23T10:54:30.516866+0000 2026-02-23T10:54:30.517 INFO:teuthology.orchestra.run.trial198.stdout:min_mon_release 20 (tentacle) 2026-02-23T10:54:30.517 INFO:teuthology.orchestra.run.trial198.stdout:election_strategy: 1 2026-02-23T10:54:30.517 INFO:teuthology.orchestra.run.trial198.stdout:0: [v2:10.20.193.198:3300/0,v1:10.20.193.198:6789/0] mon.a 2026-02-23T10:54:30.517 INFO:teuthology.orchestra.run.trial198.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/ceph.monmap (1 monitors) 2026-02-23T10:54:30.518 DEBUG:teuthology.orchestra.run.trial198:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2026-02-23T10:54:30.575 INFO:tasks.ceph:Writing /etc/ceph/ceph.conf for FSID 8ebafdf4-9523-4bab-a465-a90635e02d85... 2026-02-23T10:54:30.576 DEBUG:teuthology.orchestra.run.trial198:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2026-02-23T10:54:30.655 INFO:teuthology.orchestra.run.trial198.stdout:[global] 2026-02-23T10:54:30.655 INFO:teuthology.orchestra.run.trial198.stdout: chdir = "" 2026-02-23T10:54:30.655 INFO:teuthology.orchestra.run.trial198.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2026-02-23T10:54:30.655 INFO:teuthology.orchestra.run.trial198.stdout: auth supported = cephx 2026-02-23T10:54:30.656 INFO:teuthology.orchestra.run.trial198.stdout: 2026-02-23T10:54:30.656 INFO:teuthology.orchestra.run.trial198.stdout: filestore xattr use omap = true 2026-02-23T10:54:30.656 INFO:teuthology.orchestra.run.trial198.stdout: 2026-02-23T10:54:30.656 INFO:teuthology.orchestra.run.trial198.stdout: mon clock drift allowed = 1.000 2026-02-23T10:54:30.656 INFO:teuthology.orchestra.run.trial198.stdout: 2026-02-23T10:54:30.656 INFO:teuthology.orchestra.run.trial198.stdout: osd crush chooseleaf type = 0 2026-02-23T10:54:30.656 INFO:teuthology.orchestra.run.trial198.stdout: auth debug = true 2026-02-23T10:54:30.656 INFO:teuthology.orchestra.run.trial198.stdout: 2026-02-23T10:54:30.656 INFO:teuthology.orchestra.run.trial198.stdout: ms die on old message = true 2026-02-23T10:54:30.656 INFO:teuthology.orchestra.run.trial198.stdout: ms die on bug = true 2026-02-23T10:54:30.656 INFO:teuthology.orchestra.run.trial198.stdout: 2026-02-23T10:54:30.656 INFO:teuthology.orchestra.run.trial198.stdout: mon max pg per osd = 10000 # >= luminous 2026-02-23T10:54:30.656 INFO:teuthology.orchestra.run.trial198.stdout: mon pg warn max object skew = 0 2026-02-23T10:54:30.656 INFO:teuthology.orchestra.run.trial198.stdout: 2026-02-23T10:54:30.656 INFO:teuthology.orchestra.run.trial198.stdout: # disable pg_autoscaler by default for new pools 2026-02-23T10:54:30.656 INFO:teuthology.orchestra.run.trial198.stdout: osd_pool_default_pg_autoscale_mode = off 2026-02-23T10:54:30.656 INFO:teuthology.orchestra.run.trial198.stdout: 2026-02-23T10:54:30.656 INFO:teuthology.orchestra.run.trial198.stdout: osd pool default size = 2 2026-02-23T10:54:30.656 INFO:teuthology.orchestra.run.trial198.stdout: 2026-02-23T10:54:30.656 INFO:teuthology.orchestra.run.trial198.stdout: mon osd allow primary affinity = true 2026-02-23T10:54:30.656 INFO:teuthology.orchestra.run.trial198.stdout: mon osd allow pg remap = true 2026-02-23T10:54:30.656 INFO:teuthology.orchestra.run.trial198.stdout: mon warn on legacy crush tunables = false 2026-02-23T10:54:30.656 INFO:teuthology.orchestra.run.trial198.stdout: mon warn on crush straw calc version zero = false 2026-02-23T10:54:30.656 INFO:teuthology.orchestra.run.trial198.stdout: mon warn on no sortbitwise = false 2026-02-23T10:54:30.656 INFO:teuthology.orchestra.run.trial198.stdout: mon warn on osd down out interval zero = false 2026-02-23T10:54:30.656 INFO:teuthology.orchestra.run.trial198.stdout: mon warn on too few osds = false 2026-02-23T10:54:30.656 INFO:teuthology.orchestra.run.trial198.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2026-02-23T10:54:30.656 INFO:teuthology.orchestra.run.trial198.stdout: mon_warn_on_pool_no_redundancy = false 2026-02-23T10:54:30.656 INFO:teuthology.orchestra.run.trial198.stdout: mon_allow_pool_size_one = true 2026-02-23T10:54:30.656 INFO:teuthology.orchestra.run.trial198.stdout: 2026-02-23T10:54:30.656 INFO:teuthology.orchestra.run.trial198.stdout: osd pool default erasure code profile = plugin=isa technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2026-02-23T10:54:30.656 INFO:teuthology.orchestra.run.trial198.stdout: 2026-02-23T10:54:30.657 INFO:teuthology.orchestra.run.trial198.stdout: osd default data pool replay window = 5 2026-02-23T10:54:30.657 INFO:teuthology.orchestra.run.trial198.stdout: 2026-02-23T10:54:30.657 INFO:teuthology.orchestra.run.trial198.stdout: mon allow pool delete = true 2026-02-23T10:54:30.657 INFO:teuthology.orchestra.run.trial198.stdout: 2026-02-23T10:54:30.657 INFO:teuthology.orchestra.run.trial198.stdout: mon cluster log file level = debug 2026-02-23T10:54:30.657 INFO:teuthology.orchestra.run.trial198.stdout: debug asserts on shutdown = true 2026-02-23T10:54:30.657 INFO:teuthology.orchestra.run.trial198.stdout: mon health detail to clog = false 2026-02-23T10:54:30.657 INFO:teuthology.orchestra.run.trial198.stdout: mon host = 10.20.193.198 2026-02-23T10:54:30.657 INFO:teuthology.orchestra.run.trial198.stdout: enable experimental unrecoverable data corrupting features = crimson 2026-02-23T10:54:30.657 INFO:teuthology.orchestra.run.trial198.stdout: ms client mode = crc 2026-02-23T10:54:30.657 INFO:teuthology.orchestra.run.trial198.stdout: ms cluster mode = crc 2026-02-23T10:54:30.657 INFO:teuthology.orchestra.run.trial198.stdout: ms mon client mode = crc 2026-02-23T10:54:30.658 INFO:teuthology.orchestra.run.trial198.stdout: ms mon cluster mode = crc 2026-02-23T10:54:30.658 INFO:teuthology.orchestra.run.trial198.stdout: ms mon service mode = crc 2026-02-23T10:54:30.658 INFO:teuthology.orchestra.run.trial198.stdout: ms service mode = crc 2026-02-23T10:54:30.658 INFO:teuthology.orchestra.run.trial198.stdout: fsid = 8ebafdf4-9523-4bab-a465-a90635e02d85 2026-02-23T10:54:30.658 INFO:teuthology.orchestra.run.trial198.stdout: 2026-02-23T10:54:30.658 INFO:teuthology.orchestra.run.trial198.stdout:[osd] 2026-02-23T10:54:30.658 INFO:teuthology.orchestra.run.trial198.stdout: osd journal size = 100 2026-02-23T10:54:30.658 INFO:teuthology.orchestra.run.trial198.stdout: 2026-02-23T10:54:30.658 INFO:teuthology.orchestra.run.trial198.stdout: osd scrub load threshold = 5.0 2026-02-23T10:54:30.658 INFO:teuthology.orchestra.run.trial198.stdout: osd scrub max interval = 600 2026-02-23T10:54:30.658 INFO:teuthology.orchestra.run.trial198.stdout: osd mclock profile = high_recovery_ops 2026-02-23T10:54:30.658 INFO:teuthology.orchestra.run.trial198.stdout: osd mclock skip benchmark = true 2026-02-23T10:54:30.659 INFO:teuthology.orchestra.run.trial198.stdout: 2026-02-23T10:54:30.659 INFO:teuthology.orchestra.run.trial198.stdout: osd recover clone overlap = true 2026-02-23T10:54:30.659 INFO:teuthology.orchestra.run.trial198.stdout: osd recovery max chunk = 1048576 2026-02-23T10:54:30.659 INFO:teuthology.orchestra.run.trial198.stdout: 2026-02-23T10:54:30.659 INFO:teuthology.orchestra.run.trial198.stdout: osd debug shutdown = true 2026-02-23T10:54:30.659 INFO:teuthology.orchestra.run.trial198.stdout: osd debug op order = true 2026-02-23T10:54:30.659 INFO:teuthology.orchestra.run.trial198.stdout: osd debug verify stray on activate = true 2026-02-23T10:54:30.659 INFO:teuthology.orchestra.run.trial198.stdout: osd debug trim objects = true 2026-02-23T10:54:30.659 INFO:teuthology.orchestra.run.trial198.stdout: 2026-02-23T10:54:30.659 INFO:teuthology.orchestra.run.trial198.stdout: osd open classes on start = true 2026-02-23T10:54:30.659 INFO:teuthology.orchestra.run.trial198.stdout: osd debug pg log writeout = true 2026-02-23T10:54:30.659 INFO:teuthology.orchestra.run.trial198.stdout: 2026-02-23T10:54:30.659 INFO:teuthology.orchestra.run.trial198.stdout: osd deep scrub update digest min age = 30 2026-02-23T10:54:30.660 INFO:teuthology.orchestra.run.trial198.stdout: 2026-02-23T10:54:30.660 INFO:teuthology.orchestra.run.trial198.stdout: osd map max advance = 10 2026-02-23T10:54:30.660 INFO:teuthology.orchestra.run.trial198.stdout: 2026-02-23T10:54:30.660 INFO:teuthology.orchestra.run.trial198.stdout: journal zero on create = true 2026-02-23T10:54:30.660 INFO:teuthology.orchestra.run.trial198.stdout: 2026-02-23T10:54:30.660 INFO:teuthology.orchestra.run.trial198.stdout: filestore ondisk finisher threads = 3 2026-02-23T10:54:30.660 INFO:teuthology.orchestra.run.trial198.stdout: filestore apply finisher threads = 3 2026-02-23T10:54:30.660 INFO:teuthology.orchestra.run.trial198.stdout: 2026-02-23T10:54:30.660 INFO:teuthology.orchestra.run.trial198.stdout: bdev debug aio = true 2026-02-23T10:54:30.660 INFO:teuthology.orchestra.run.trial198.stdout: osd debug misdirected ops = true 2026-02-23T10:54:30.660 INFO:teuthology.orchestra.run.trial198.stdout: debug monc = 20 2026-02-23T10:54:30.660 INFO:teuthology.orchestra.run.trial198.stdout: bluestore block size = 96636764160 2026-02-23T10:54:30.660 INFO:teuthology.orchestra.run.trial198.stdout: bluestore compression algorithm = snappy 2026-02-23T10:54:30.661 INFO:teuthology.orchestra.run.trial198.stdout: bluestore compression mode = aggressive 2026-02-23T10:54:30.661 INFO:teuthology.orchestra.run.trial198.stdout: bluestore fsck on mount = True 2026-02-23T10:54:30.661 INFO:teuthology.orchestra.run.trial198.stdout: bluestore rocksdb cf = False 2026-02-23T10:54:30.661 INFO:teuthology.orchestra.run.trial198.stdout: crimson cpu num = 4 2026-02-23T10:54:30.661 INFO:teuthology.orchestra.run.trial198.stdout: crimson osd obc lru size = 10 2026-02-23T10:54:30.661 INFO:teuthology.orchestra.run.trial198.stdout: debug alienstore = 20 2026-02-23T10:54:30.661 INFO:teuthology.orchestra.run.trial198.stdout: debug bluefs = 20 2026-02-23T10:54:30.661 INFO:teuthology.orchestra.run.trial198.stdout: debug bluestore = 20 2026-02-23T10:54:30.661 INFO:teuthology.orchestra.run.trial198.stdout: debug ms = 20 2026-02-23T10:54:30.661 INFO:teuthology.orchestra.run.trial198.stdout: debug osd = 20 2026-02-23T10:54:30.661 INFO:teuthology.orchestra.run.trial198.stdout: debug rocksdb = 10 2026-02-23T10:54:30.661 INFO:teuthology.orchestra.run.trial198.stdout: err to stderr = True 2026-02-23T10:54:30.661 INFO:teuthology.orchestra.run.trial198.stdout: log flush on exit = True 2026-02-23T10:54:30.662 INFO:teuthology.orchestra.run.trial198.stdout: log to file = False 2026-02-23T10:54:30.662 INFO:teuthology.orchestra.run.trial198.stdout: log to stderr = True 2026-02-23T10:54:30.662 INFO:teuthology.orchestra.run.trial198.stdout: mon osd backfillfull_ratio = 0.85 2026-02-23T10:54:30.662 INFO:teuthology.orchestra.run.trial198.stdout: mon osd full ratio = 0.9 2026-02-23T10:54:30.662 INFO:teuthology.orchestra.run.trial198.stdout: mon osd nearfull ratio = 0.8 2026-02-23T10:54:30.662 INFO:teuthology.orchestra.run.trial198.stdout: osd failsafe full ratio = 0.95 2026-02-23T10:54:30.662 INFO:teuthology.orchestra.run.trial198.stdout: osd objectstore = bluestore 2026-02-23T10:54:30.662 INFO:teuthology.orchestra.run.trial198.stdout: osd shutdown pgref assert = True 2026-02-23T10:54:30.662 INFO:teuthology.orchestra.run.trial198.stdout: seastore cachepin size pershard = 64M 2026-02-23T10:54:30.662 INFO:teuthology.orchestra.run.trial198.stdout: seastore max concurrent transactions = 8 2026-02-23T10:54:30.662 INFO:teuthology.orchestra.run.trial198.stdout: 2026-02-23T10:54:30.662 INFO:teuthology.orchestra.run.trial198.stdout:[mgr] 2026-02-23T10:54:30.663 INFO:teuthology.orchestra.run.trial198.stdout: debug ms = 1 2026-02-23T10:54:30.663 INFO:teuthology.orchestra.run.trial198.stdout: debug mgr = 20 2026-02-23T10:54:30.663 INFO:teuthology.orchestra.run.trial198.stdout: debug mon = 20 2026-02-23T10:54:30.663 INFO:teuthology.orchestra.run.trial198.stdout: debug auth = 20 2026-02-23T10:54:30.663 INFO:teuthology.orchestra.run.trial198.stdout: mon reweight min pgs per osd = 4 2026-02-23T10:54:30.663 INFO:teuthology.orchestra.run.trial198.stdout: mon reweight min bytes per osd = 10 2026-02-23T10:54:30.663 INFO:teuthology.orchestra.run.trial198.stdout: mgr/telemetry/nag = false 2026-02-23T10:54:30.663 INFO:teuthology.orchestra.run.trial198.stdout: 2026-02-23T10:54:30.663 INFO:teuthology.orchestra.run.trial198.stdout:[mon] 2026-02-23T10:54:30.663 INFO:teuthology.orchestra.run.trial198.stdout: debug ms = 1 2026-02-23T10:54:30.663 INFO:teuthology.orchestra.run.trial198.stdout: debug mon = 20 2026-02-23T10:54:30.663 INFO:teuthology.orchestra.run.trial198.stdout: debug paxos = 20 2026-02-23T10:54:30.663 INFO:teuthology.orchestra.run.trial198.stdout: debug auth = 20 2026-02-23T10:54:30.663 INFO:teuthology.orchestra.run.trial198.stdout: mon data avail warn = 5 2026-02-23T10:54:30.664 INFO:teuthology.orchestra.run.trial198.stdout: mon mgr mkfs grace = 240 2026-02-23T10:54:30.664 INFO:teuthology.orchestra.run.trial198.stdout: mon reweight min pgs per osd = 4 2026-02-23T10:54:30.664 INFO:teuthology.orchestra.run.trial198.stdout: mon osd reporter subtree level = osd 2026-02-23T10:54:30.664 INFO:teuthology.orchestra.run.trial198.stdout: mon osd prime pg temp = true 2026-02-23T10:54:30.664 INFO:teuthology.orchestra.run.trial198.stdout: mon reweight min bytes per osd = 10 2026-02-23T10:54:30.664 INFO:teuthology.orchestra.run.trial198.stdout: 2026-02-23T10:54:30.664 INFO:teuthology.orchestra.run.trial198.stdout: # rotate auth tickets quickly to exercise renewal paths 2026-02-23T10:54:30.664 INFO:teuthology.orchestra.run.trial198.stdout: auth mon ticket ttl = 660 # 11m 2026-02-23T10:54:30.664 INFO:teuthology.orchestra.run.trial198.stdout: auth service ticket ttl = 240 # 4m 2026-02-23T10:54:30.664 INFO:teuthology.orchestra.run.trial198.stdout: 2026-02-23T10:54:30.664 INFO:teuthology.orchestra.run.trial198.stdout: # don't complain about insecure global_id in the test suite 2026-02-23T10:54:30.664 INFO:teuthology.orchestra.run.trial198.stdout: mon_warn_on_insecure_global_id_reclaim = false 2026-02-23T10:54:30.665 INFO:teuthology.orchestra.run.trial198.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2026-02-23T10:54:30.665 INFO:teuthology.orchestra.run.trial198.stdout: 2026-02-23T10:54:30.665 INFO:teuthology.orchestra.run.trial198.stdout: # 1m isn't quite enough 2026-02-23T10:54:30.665 INFO:teuthology.orchestra.run.trial198.stdout: mon_down_mkfs_grace = 2m 2026-02-23T10:54:30.665 INFO:teuthology.orchestra.run.trial198.stdout: 2026-02-23T10:54:30.665 INFO:teuthology.orchestra.run.trial198.stdout: mon_warn_on_filestore_osds = false 2026-02-23T10:54:30.665 INFO:teuthology.orchestra.run.trial198.stdout: mon min osdmap epochs = 50 2026-02-23T10:54:30.665 INFO:teuthology.orchestra.run.trial198.stdout: mon osdmap full prune interval = 2 2026-02-23T10:54:30.665 INFO:teuthology.orchestra.run.trial198.stdout: mon osdmap full prune min = 15 2026-02-23T10:54:30.665 INFO:teuthology.orchestra.run.trial198.stdout: mon osdmap full prune txsize = 2 2026-02-23T10:54:30.665 INFO:teuthology.orchestra.run.trial198.stdout: paxos service trim min = 10 2026-02-23T10:54:30.665 INFO:teuthology.orchestra.run.trial198.stdout: osd pool default crimson = True 2026-02-23T10:54:30.665 INFO:teuthology.orchestra.run.trial198.stdout: 2026-02-23T10:54:30.666 INFO:teuthology.orchestra.run.trial198.stdout:[client] 2026-02-23T10:54:30.666 INFO:teuthology.orchestra.run.trial198.stdout: rgw cache enabled = true 2026-02-23T10:54:30.666 INFO:teuthology.orchestra.run.trial198.stdout: rgw enable ops log = true 2026-02-23T10:54:30.666 INFO:teuthology.orchestra.run.trial198.stdout: rgw enable usage log = true 2026-02-23T10:54:30.666 INFO:teuthology.orchestra.run.trial198.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2026-02-23T10:54:30.666 INFO:teuthology.orchestra.run.trial198.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2026-02-23T10:54:30.666 INFO:teuthology.orchestra.run.trial198.stdout:[mon.a] 2026-02-23T10:54:30.667 INFO:tasks.ceph:Creating admin key on mon.a... 2026-02-23T10:54:30.668 DEBUG:teuthology.orchestra.run.trial198:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/ceph.keyring 2026-02-23T10:54:30.758 INFO:tasks.ceph:Copying monmap to all nodes... 2026-02-23T10:54:30.758 DEBUG:teuthology.orchestra.run.trial198:> set -ex 2026-02-23T10:54:30.758 DEBUG:teuthology.orchestra.run.trial198:> dd if=/etc/ceph/ceph.keyring of=/dev/stdout 2026-02-23T10:54:30.773 DEBUG:teuthology.orchestra.run.trial198:> set -ex 2026-02-23T10:54:30.773 DEBUG:teuthology.orchestra.run.trial198:> dd if=/home/ubuntu/cephtest/ceph.monmap of=/dev/stdout 2026-02-23T10:54:30.829 INFO:tasks.ceph:Sending monmap to node ubuntu@trial198.front.sepia.ceph.com 2026-02-23T10:54:30.829 DEBUG:teuthology.orchestra.run.trial198:> set -ex 2026-02-23T10:54:30.830 DEBUG:teuthology.orchestra.run.trial198:> sudo dd of=/etc/ceph/ceph.keyring 2026-02-23T10:54:30.830 DEBUG:teuthology.orchestra.run.trial198:> sudo chmod 0644 /etc/ceph/ceph.keyring 2026-02-23T10:54:30.902 DEBUG:teuthology.orchestra.run.trial198:> set -ex 2026-02-23T10:54:30.903 DEBUG:teuthology.orchestra.run.trial198:> dd of=/home/ubuntu/cephtest/ceph.monmap 2026-02-23T10:54:30.959 INFO:tasks.ceph:Setting up mon nodes... 2026-02-23T10:54:30.959 INFO:tasks.ceph:Setting up mgr nodes... 2026-02-23T10:54:30.959 DEBUG:teuthology.orchestra.run.trial198:> sudo mkdir -p /var/lib/ceph/mgr/ceph-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/ceph-x/keyring 2026-02-23T10:54:31.050 INFO:teuthology.orchestra.run.trial198.stdout:creating /var/lib/ceph/mgr/ceph-x/keyring 2026-02-23T10:54:31.052 INFO:tasks.ceph:Setting up mds nodes... 2026-02-23T10:54:31.053 INFO:tasks.ceph_client:Setting up client nodes... 2026-02-23T10:54:31.053 DEBUG:teuthology.orchestra.run.trial198:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/ceph.client.0.keyring && sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2026-02-23T10:54:31.093 INFO:teuthology.orchestra.run.trial198.stdout:creating /etc/ceph/ceph.client.0.keyring 2026-02-23T10:54:31.105 INFO:tasks.ceph:Running mkfs on osd nodes... 2026-02-23T10:54:31.105 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@trial198.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'}} 2026-02-23T10:54:31.105 DEBUG:teuthology.orchestra.run.trial198:> sudo mkdir -p /var/lib/ceph/osd/ceph-0 2026-02-23T10:54:31.168 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2026-02-23T10:54:31.168 INFO:tasks.ceph:role: osd.0 2026-02-23T10:54:31.168 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@trial198.front.sepia.ceph.com 2026-02-23T10:54:31.168 DEBUG:teuthology.orchestra.run.trial198:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2026-02-23T10:54:31.233 INFO:teuthology.orchestra.run.trial198.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=45875200 blks 2026-02-23T10:54:31.233 INFO:teuthology.orchestra.run.trial198.stdout: = sectsz=4096 attr=2, projid32bit=1 2026-02-23T10:54:31.233 INFO:teuthology.orchestra.run.trial198.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2026-02-23T10:54:31.233 INFO:teuthology.orchestra.run.trial198.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2026-02-23T10:54:31.234 INFO:teuthology.orchestra.run.trial198.stdout:data = bsize=4096 blocks=183500800, imaxpct=25 2026-02-23T10:54:31.234 INFO:teuthology.orchestra.run.trial198.stdout: = sunit=0 swidth=0 blks 2026-02-23T10:54:31.234 INFO:teuthology.orchestra.run.trial198.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2026-02-23T10:54:31.234 INFO:teuthology.orchestra.run.trial198.stdout:log =internal log bsize=4096 blocks=89600, version=2 2026-02-23T10:54:31.234 INFO:teuthology.orchestra.run.trial198.stdout: = sectsz=4096 sunit=1 blks, lazy-count=1 2026-02-23T10:54:31.234 INFO:teuthology.orchestra.run.trial198.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2026-02-23T10:54:31.803 INFO:teuthology.orchestra.run.trial198.stdout:Discarding blocks...Done. 2026-02-23T10:54:31.804 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@trial198.front.sepia.ceph.com -o noatime 2026-02-23T10:54:31.805 DEBUG:teuthology.orchestra.run.trial198:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-0 2026-02-23T10:54:31.879 DEBUG:teuthology.orchestra.run.trial198:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-0 2026-02-23T10:54:31.948 DEBUG:teuthology.orchestra.run.trial198:> sudo mkdir -p /var/lib/ceph/osd/ceph-1 2026-02-23T10:54:32.011 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2026-02-23T10:54:32.011 INFO:tasks.ceph:role: osd.1 2026-02-23T10:54:32.012 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@trial198.front.sepia.ceph.com 2026-02-23T10:54:32.012 DEBUG:teuthology.orchestra.run.trial198:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2026-02-23T10:54:32.077 INFO:teuthology.orchestra.run.trial198.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=45875200 blks 2026-02-23T10:54:32.077 INFO:teuthology.orchestra.run.trial198.stdout: = sectsz=4096 attr=2, projid32bit=1 2026-02-23T10:54:32.077 INFO:teuthology.orchestra.run.trial198.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2026-02-23T10:54:32.077 INFO:teuthology.orchestra.run.trial198.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2026-02-23T10:54:32.077 INFO:teuthology.orchestra.run.trial198.stdout:data = bsize=4096 blocks=183500800, imaxpct=25 2026-02-23T10:54:32.077 INFO:teuthology.orchestra.run.trial198.stdout: = sunit=0 swidth=0 blks 2026-02-23T10:54:32.077 INFO:teuthology.orchestra.run.trial198.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2026-02-23T10:54:32.077 INFO:teuthology.orchestra.run.trial198.stdout:log =internal log bsize=4096 blocks=89600, version=2 2026-02-23T10:54:32.077 INFO:teuthology.orchestra.run.trial198.stdout: = sectsz=4096 sunit=1 blks, lazy-count=1 2026-02-23T10:54:32.077 INFO:teuthology.orchestra.run.trial198.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2026-02-23T10:54:32.639 INFO:teuthology.orchestra.run.trial198.stdout:Discarding blocks...Done. 2026-02-23T10:54:32.641 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@trial198.front.sepia.ceph.com -o noatime 2026-02-23T10:54:32.641 DEBUG:teuthology.orchestra.run.trial198:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-1 2026-02-23T10:54:32.709 DEBUG:teuthology.orchestra.run.trial198:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-1 2026-02-23T10:54:32.776 DEBUG:teuthology.orchestra.run.trial198:> sudo mkdir -p /var/lib/ceph/osd/ceph-2 2026-02-23T10:54:32.840 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2026-02-23T10:54:32.840 INFO:tasks.ceph:role: osd.2 2026-02-23T10:54:32.840 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@trial198.front.sepia.ceph.com 2026-02-23T10:54:32.840 DEBUG:teuthology.orchestra.run.trial198:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2026-02-23T10:54:32.905 INFO:teuthology.orchestra.run.trial198.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=45875200 blks 2026-02-23T10:54:32.905 INFO:teuthology.orchestra.run.trial198.stdout: = sectsz=4096 attr=2, projid32bit=1 2026-02-23T10:54:32.905 INFO:teuthology.orchestra.run.trial198.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2026-02-23T10:54:32.905 INFO:teuthology.orchestra.run.trial198.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2026-02-23T10:54:32.905 INFO:teuthology.orchestra.run.trial198.stdout:data = bsize=4096 blocks=183500800, imaxpct=25 2026-02-23T10:54:32.905 INFO:teuthology.orchestra.run.trial198.stdout: = sunit=0 swidth=0 blks 2026-02-23T10:54:32.905 INFO:teuthology.orchestra.run.trial198.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2026-02-23T10:54:32.905 INFO:teuthology.orchestra.run.trial198.stdout:log =internal log bsize=4096 blocks=89600, version=2 2026-02-23T10:54:32.906 INFO:teuthology.orchestra.run.trial198.stdout: = sectsz=4096 sunit=1 blks, lazy-count=1 2026-02-23T10:54:32.906 INFO:teuthology.orchestra.run.trial198.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2026-02-23T10:54:33.481 INFO:teuthology.orchestra.run.trial198.stdout:Discarding blocks...Done. 2026-02-23T10:54:33.483 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@trial198.front.sepia.ceph.com -o noatime 2026-02-23T10:54:33.483 DEBUG:teuthology.orchestra.run.trial198:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-2 2026-02-23T10:54:33.551 DEBUG:teuthology.orchestra.run.trial198:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-2 2026-02-23T10:54:33.617 DEBUG:teuthology.orchestra.run.trial198:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/ceph.monmap 2026-02-23T10:54:33.766 INFO:teuthology.orchestra.run.trial198.stderr:INFO  osd - OSD::main: parsing early config 2026-02-23T10:54:33.823 INFO:teuthology.orchestra.run.trial198.stderr:INFO 2026-02-23 10:54:33,822 seastar - Reactor backend: linux-aio 2026-02-23T10:54:33.832 INFO:teuthology.orchestra.run.trial198.stderr:INFO 2026-02-23 10:54:33,832 [shard 0:main] io - IO queue was unable to find a suitable maximum request length, the search was cut-off early at: 16MB 2026-02-23T10:54:33.832 INFO:teuthology.orchestra.run.trial198.stderr:INFO 2026-02-23 10:54:33,832 [shard 0:main] io - IO queue was unable to find a suitable maximum request length, the search was cut-off early at: 16MB 2026-02-23T10:54:33.834 INFO:teuthology.orchestra.run.trial198.stderr:==39352==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2026-02-23T10:54:33.867 INFO:teuthology.orchestra.run.trial198.stderr:Reactor stalled for 32 ms on shard 0, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0x9f10dfc 0x9d045c3 0x9d40a13 0x9d43fc6 0x9d44234 0x9d4437c 0x9d4483b /lib64/libc.so.6+0x3fc2f 0x358512d 0x35cb04d 0x35cb807 0x35cbb09 0x35cbc9a 0x35cbf43 0x35cc24a 0x35cc390 0x357c2a6 0x358288e 0x3582ad4 0x3582c28 0x3582cd6 0x3582fa0 0x35831a3 0x3583424 0x35835e6 0x3583705 0x357c2a6 0x357c652 0x357c898 0x357c96d 0x35d4ada 0x35d4cf7 0x35d4e6b 0x35d529b 0x35d54da 0x35d575a 0x35d593a 0x35d5a5c 0x359268e 0x3592a9d 0x3592d54 0x3592ebd 0x35932fa 0x35936a0 0x3593920 0x3593b00 0x3593c0f 0x359268e 0x3594527 0x35947de 0x3594a47 0x3595239 0x359572e 0x35ce25b 0x35ce5ba 0x35ce932 0x35cedd0 0x35cf20f 0x3596cf0 0x3597d27 0x3597fde 0x359847d 0x3598599 0x359884d 0x3598bdf 0x3598f12 0x359926f 0x35994a7 0x3599ae2 0x3599e12 0x359a178 0x359a625 0x359aa64 0x359ac90 0x359b09f 0x359b356 0x359b4a7 0x359b778 0x359bb03 0x359be36 0x359c329 0x359c70b 0x359c897 0x359cccf 0x359d03b 0x359d3a0 0x359d6c3 0x359d7a5 0x359da33 0x359dc0a 0x359de71 0x35d1229 0x35d1979 0x35d1e65 0x35d21df 0x35d245e 0x35d2513 0x357b132 0x357ba95 2026-02-23T10:54:33.889 INFO:teuthology.orchestra.run.trial198.stderr:INFO 2026-02-23 10:54:33,889 [shard 0:main] osd - bypassing the config fetch due to --no-mon-config 2026-02-23T10:54:33.890 INFO:teuthology.orchestra.run.trial198.stderr:INFO 2026-02-23 10:54:33,889 [shard 0:main] osd - Configure option_name crimson_reactor_task_quota_ms with value : 0.500000 2026-02-23T10:54:33.890 INFO:teuthology.orchestra.run.trial198.stderr:INFO 2026-02-23 10:54:33,889 [shard 0:main] osd - Configure option_name crimson_reactor_idle_poll_time_us with value : 200 2026-02-23T10:54:33.890 INFO:teuthology.orchestra.run.trial198.stderr:INFO 2026-02-23 10:54:33,889 [shard 0:main] osd - get_early_config: set --thread-affinity 0 --smp 4 2026-02-23T10:54:33.890 INFO:teuthology.orchestra.run.trial198.stderr:DEBUG 2026-02-23 10:54:33,890 [shard 0:main] seastar - reactor::drain 2026-02-23T10:54:35.967 INFO:teuthology.orchestra.run.trial198.stderr:----------------------------------------------------- 2026-02-23T10:54:35.967 INFO:teuthology.orchestra.run.trial198.stderr:Suppressions used: 2026-02-23T10:54:35.967 INFO:teuthology.orchestra.run.trial198.stderr: count bytes template 2026-02-23T10:54:35.967 INFO:teuthology.orchestra.run.trial198.stderr: 1 8 MallocExtension::Register 2026-02-23T10:54:35.967 INFO:teuthology.orchestra.run.trial198.stderr:----------------------------------------------------- 2026-02-23T10:54:35.967 INFO:teuthology.orchestra.run.trial198.stderr: 2026-02-23T10:54:35.986 INFO:teuthology.orchestra.run.trial198.stderr:INFO  osd - OSD::main: early config parsed successfully 2026-02-23T10:54:35.987 INFO:teuthology.orchestra.run.trial198.stderr:INFO  osd - OSD::main: initializing seastar app_template 2026-02-23T10:54:36.031 INFO:teuthology.orchestra.run.trial198.stderr:INFO  osd - OSD::main: registering CLI options 2026-02-23T10:54:36.032 INFO:teuthology.orchestra.run.trial198.stderr:INFO  osd - OSD::main: entering seastar runtime 2026-02-23T10:54:36.038 INFO:teuthology.orchestra.run.trial198.stderr:INFO 2026-02-23 10:54:36,038 seastar - Reactor backend: linux-aio 2026-02-23T10:54:36.050 INFO:teuthology.orchestra.run.trial198.stderr:INFO 2026-02-23 10:54:36,050 [shard 3:main] io - IO queue was unable to find a suitable maximum request length, the search was cut-off early at: 16MB 2026-02-23T10:54:36.050 INFO:teuthology.orchestra.run.trial198.stderr:INFO 2026-02-23 10:54:36,050 [shard 3:main] io - IO queue was unable to find a suitable maximum request length, the search was cut-off early at: 16MB 2026-02-23T10:54:36.054 INFO:teuthology.orchestra.run.trial198.stderr:==39350==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2026-02-23T10:54:36.054 INFO:teuthology.orchestra.run.trial198.stderr:INFO 2026-02-23 10:54:36,054 [shard 0:main] osd - OSD::main: seastar runtime started 2026-02-23T10:54:36.111 INFO:teuthology.orchestra.run.trial198.stderr:DEBUG 2026-02-23 10:54:36,111 [shard 0:main] osd - OSD::main: initializing logger output 2026-02-23T10:54:39.522 INFO:teuthology.orchestra.run.trial198.stderr:----------------------------------------------------- 2026-02-23T10:54:39.523 INFO:teuthology.orchestra.run.trial198.stderr:Suppressions used: 2026-02-23T10:54:39.523 INFO:teuthology.orchestra.run.trial198.stderr: count bytes template 2026-02-23T10:54:39.523 INFO:teuthology.orchestra.run.trial198.stderr: 1 8 MallocExtension::Register 2026-02-23T10:54:39.523 INFO:teuthology.orchestra.run.trial198.stderr:----------------------------------------------------- 2026-02-23T10:54:39.523 INFO:teuthology.orchestra.run.trial198.stderr: 2026-02-23T10:54:39.552 DEBUG:teuthology.orchestra.run.trial198:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2026-02-23T10:54:39.578 DEBUG:teuthology.orchestra.run.trial198:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/ceph.monmap 2026-02-23T10:54:39.729 INFO:teuthology.orchestra.run.trial198.stderr:INFO  osd - OSD::main: parsing early config 2026-02-23T10:54:39.793 INFO:teuthology.orchestra.run.trial198.stderr:INFO 2026-02-23 10:54:39,792 seastar - Reactor backend: linux-aio 2026-02-23T10:54:39.803 INFO:teuthology.orchestra.run.trial198.stderr:INFO 2026-02-23 10:54:39,802 [shard 0:main] io - IO queue was unable to find a suitable maximum request length, the search was cut-off early at: 16MB 2026-02-23T10:54:39.803 INFO:teuthology.orchestra.run.trial198.stderr:INFO 2026-02-23 10:54:39,802 [shard 0:main] io - IO queue was unable to find a suitable maximum request length, the search was cut-off early at: 16MB 2026-02-23T10:54:39.805 INFO:teuthology.orchestra.run.trial198.stderr:==39554==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2026-02-23T10:54:39.860 INFO:teuthology.orchestra.run.trial198.stderr:INFO 2026-02-23 10:54:39,859 [shard 0:main] osd - bypassing the config fetch due to --no-mon-config 2026-02-23T10:54:39.860 INFO:teuthology.orchestra.run.trial198.stderr:INFO 2026-02-23 10:54:39,859 [shard 0:main] osd - Configure option_name crimson_reactor_task_quota_ms with value : 0.500000 2026-02-23T10:54:39.860 INFO:teuthology.orchestra.run.trial198.stderr:INFO 2026-02-23 10:54:39,859 [shard 0:main] osd - Configure option_name crimson_reactor_idle_poll_time_us with value : 200 2026-02-23T10:54:39.860 INFO:teuthology.orchestra.run.trial198.stderr:INFO 2026-02-23 10:54:39,859 [shard 0:main] osd - get_early_config: set --thread-affinity 0 --smp 4 2026-02-23T10:54:39.860 INFO:teuthology.orchestra.run.trial198.stderr:DEBUG 2026-02-23 10:54:39,860 [shard 0:main] seastar - reactor::drain 2026-02-23T10:54:41.951 INFO:teuthology.orchestra.run.trial198.stderr:----------------------------------------------------- 2026-02-23T10:54:41.951 INFO:teuthology.orchestra.run.trial198.stderr:Suppressions used: 2026-02-23T10:54:41.951 INFO:teuthology.orchestra.run.trial198.stderr: count bytes template 2026-02-23T10:54:41.952 INFO:teuthology.orchestra.run.trial198.stderr: 1 8 MallocExtension::Register 2026-02-23T10:54:41.952 INFO:teuthology.orchestra.run.trial198.stderr:----------------------------------------------------- 2026-02-23T10:54:41.952 INFO:teuthology.orchestra.run.trial198.stderr: 2026-02-23T10:54:41.972 INFO:teuthology.orchestra.run.trial198.stderr:INFO  osd - OSD::main: early config parsed successfully 2026-02-23T10:54:41.972 INFO:teuthology.orchestra.run.trial198.stderr:INFO  osd - OSD::main: initializing seastar app_template 2026-02-23T10:54:42.015 INFO:teuthology.orchestra.run.trial198.stderr:INFO  osd - OSD::main: registering CLI options 2026-02-23T10:54:42.016 INFO:teuthology.orchestra.run.trial198.stderr:INFO  osd - OSD::main: entering seastar runtime 2026-02-23T10:54:42.022 INFO:teuthology.orchestra.run.trial198.stderr:INFO 2026-02-23 10:54:42,022 seastar - Reactor backend: linux-aio 2026-02-23T10:54:42.035 INFO:teuthology.orchestra.run.trial198.stderr:INFO 2026-02-23 10:54:42,034 [shard 2:main] io - IO queue was unable to find a suitable maximum request length, the search was cut-off early at: 16MB 2026-02-23T10:54:42.035 INFO:teuthology.orchestra.run.trial198.stderr:INFO 2026-02-23 10:54:42,034 [shard 2:main] io - IO queue was unable to find a suitable maximum request length, the search was cut-off early at: 16MB 2026-02-23T10:54:42.039 INFO:teuthology.orchestra.run.trial198.stderr:==39552==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2026-02-23T10:54:42.039 INFO:teuthology.orchestra.run.trial198.stderr:INFO 2026-02-23 10:54:42,038 [shard 0:main] osd - OSD::main: seastar runtime started 2026-02-23T10:54:42.098 INFO:teuthology.orchestra.run.trial198.stderr:DEBUG 2026-02-23 10:54:42,097 [shard 0:main] osd - OSD::main: initializing logger output 2026-02-23T10:54:45.417 INFO:teuthology.orchestra.run.trial198.stderr:----------------------------------------------------- 2026-02-23T10:54:45.417 INFO:teuthology.orchestra.run.trial198.stderr:Suppressions used: 2026-02-23T10:54:45.417 INFO:teuthology.orchestra.run.trial198.stderr: count bytes template 2026-02-23T10:54:45.417 INFO:teuthology.orchestra.run.trial198.stderr: 1 8 MallocExtension::Register 2026-02-23T10:54:45.417 INFO:teuthology.orchestra.run.trial198.stderr:----------------------------------------------------- 2026-02-23T10:54:45.417 INFO:teuthology.orchestra.run.trial198.stderr: 2026-02-23T10:54:45.448 DEBUG:teuthology.orchestra.run.trial198:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2026-02-23T10:54:45.474 DEBUG:teuthology.orchestra.run.trial198:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/ceph.monmap 2026-02-23T10:54:45.626 INFO:teuthology.orchestra.run.trial198.stderr:INFO  osd - OSD::main: parsing early config 2026-02-23T10:54:45.680 INFO:teuthology.orchestra.run.trial198.stderr:INFO 2026-02-23 10:54:45,679 seastar - Reactor backend: linux-aio 2026-02-23T10:54:45.698 INFO:teuthology.orchestra.run.trial198.stderr:INFO 2026-02-23 10:54:45,697 [shard 0:main] io - IO queue was unable to find a suitable maximum request length, the search was cut-off early at: 16MB 2026-02-23T10:54:45.698 INFO:teuthology.orchestra.run.trial198.stderr:INFO 2026-02-23 10:54:45,697 [shard 0:main] io - IO queue was unable to find a suitable maximum request length, the search was cut-off early at: 16MB 2026-02-23T10:54:45.700 INFO:teuthology.orchestra.run.trial198.stderr:==39756==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2026-02-23T10:54:45.755 INFO:teuthology.orchestra.run.trial198.stderr:INFO 2026-02-23 10:54:45,754 [shard 0:main] osd - bypassing the config fetch due to --no-mon-config 2026-02-23T10:54:45.755 INFO:teuthology.orchestra.run.trial198.stderr:INFO 2026-02-23 10:54:45,754 [shard 0:main] osd - Configure option_name crimson_reactor_task_quota_ms with value : 0.500000 2026-02-23T10:54:45.755 INFO:teuthology.orchestra.run.trial198.stderr:INFO 2026-02-23 10:54:45,754 [shard 0:main] osd - Configure option_name crimson_reactor_idle_poll_time_us with value : 200 2026-02-23T10:54:45.755 INFO:teuthology.orchestra.run.trial198.stderr:INFO 2026-02-23 10:54:45,754 [shard 0:main] osd - get_early_config: set --thread-affinity 0 --smp 4 2026-02-23T10:54:45.755 INFO:teuthology.orchestra.run.trial198.stderr:DEBUG 2026-02-23 10:54:45,754 [shard 0:main] seastar - reactor::drain 2026-02-23T10:54:47.842 INFO:teuthology.orchestra.run.trial198.stderr:----------------------------------------------------- 2026-02-23T10:54:47.842 INFO:teuthology.orchestra.run.trial198.stderr:Suppressions used: 2026-02-23T10:54:47.843 INFO:teuthology.orchestra.run.trial198.stderr: count bytes template 2026-02-23T10:54:47.843 INFO:teuthology.orchestra.run.trial198.stderr: 1 8 MallocExtension::Register 2026-02-23T10:54:47.843 INFO:teuthology.orchestra.run.trial198.stderr:----------------------------------------------------- 2026-02-23T10:54:47.843 INFO:teuthology.orchestra.run.trial198.stderr: 2026-02-23T10:54:47.862 INFO:teuthology.orchestra.run.trial198.stderr:INFO  osd - OSD::main: early config parsed successfully 2026-02-23T10:54:47.863 INFO:teuthology.orchestra.run.trial198.stderr:INFO  osd - OSD::main: initializing seastar app_template 2026-02-23T10:54:47.911 INFO:teuthology.orchestra.run.trial198.stderr:INFO  osd - OSD::main: registering CLI options 2026-02-23T10:54:47.912 INFO:teuthology.orchestra.run.trial198.stderr:INFO  osd - OSD::main: entering seastar runtime 2026-02-23T10:54:47.918 INFO:teuthology.orchestra.run.trial198.stderr:INFO 2026-02-23 10:54:47,918 seastar - Reactor backend: linux-aio 2026-02-23T10:54:47.931 INFO:teuthology.orchestra.run.trial198.stderr:INFO 2026-02-23 10:54:47,930 [shard 2:main] io - IO queue was unable to find a suitable maximum request length, the search was cut-off early at: 16MB 2026-02-23T10:54:47.931 INFO:teuthology.orchestra.run.trial198.stderr:INFO 2026-02-23 10:54:47,930 [shard 2:main] io - IO queue was unable to find a suitable maximum request length, the search was cut-off early at: 16MB 2026-02-23T10:54:47.934 INFO:teuthology.orchestra.run.trial198.stderr:==39754==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2026-02-23T10:54:47.935 INFO:teuthology.orchestra.run.trial198.stderr:INFO 2026-02-23 10:54:47,934 [shard 0:main] osd - OSD::main: seastar runtime started 2026-02-23T10:54:47.995 INFO:teuthology.orchestra.run.trial198.stderr:DEBUG 2026-02-23 10:54:47,995 [shard 0:main] osd - OSD::main: initializing logger output 2026-02-23T10:54:51.170 INFO:teuthology.orchestra.run.trial198.stderr:----------------------------------------------------- 2026-02-23T10:54:51.171 INFO:teuthology.orchestra.run.trial198.stderr:Suppressions used: 2026-02-23T10:54:51.171 INFO:teuthology.orchestra.run.trial198.stderr: count bytes template 2026-02-23T10:54:51.171 INFO:teuthology.orchestra.run.trial198.stderr: 1 8 MallocExtension::Register 2026-02-23T10:54:51.171 INFO:teuthology.orchestra.run.trial198.stderr:----------------------------------------------------- 2026-02-23T10:54:51.171 INFO:teuthology.orchestra.run.trial198.stderr: 2026-02-23T10:54:51.202 DEBUG:teuthology.orchestra.run.trial198:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2026-02-23T10:54:51.227 INFO:tasks.ceph:Reading keys from all nodes... 2026-02-23T10:54:51.227 DEBUG:teuthology.orchestra.run.trial198:> set -ex 2026-02-23T10:54:51.227 DEBUG:teuthology.orchestra.run.trial198:> sudo dd if=/var/lib/ceph/mgr/ceph-x/keyring of=/dev/stdout 2026-02-23T10:54:51.291 DEBUG:teuthology.orchestra.run.trial198:> set -ex 2026-02-23T10:54:51.291 DEBUG:teuthology.orchestra.run.trial198:> sudo dd if=/var/lib/ceph/osd/ceph-0/keyring of=/dev/stdout 2026-02-23T10:54:51.356 DEBUG:teuthology.orchestra.run.trial198:> set -ex 2026-02-23T10:54:51.357 DEBUG:teuthology.orchestra.run.trial198:> sudo dd if=/var/lib/ceph/osd/ceph-1/keyring of=/dev/stdout 2026-02-23T10:54:51.419 DEBUG:teuthology.orchestra.run.trial198:> set -ex 2026-02-23T10:54:51.419 DEBUG:teuthology.orchestra.run.trial198:> sudo dd if=/var/lib/ceph/osd/ceph-2/keyring of=/dev/stdout 2026-02-23T10:54:51.484 DEBUG:teuthology.orchestra.run.trial198:> set -ex 2026-02-23T10:54:51.485 DEBUG:teuthology.orchestra.run.trial198:> dd if=/etc/ceph/ceph.client.0.keyring of=/dev/stdout 2026-02-23T10:54:51.542 INFO:tasks.ceph:Adding keys to all mons... 2026-02-23T10:54:51.542 DEBUG:teuthology.orchestra.run.trial198:> sudo tee -a /etc/ceph/ceph.keyring 2026-02-23T10:54:51.605 INFO:teuthology.orchestra.run.trial198.stdout:[mgr.x] 2026-02-23T10:54:51.605 INFO:teuthology.orchestra.run.trial198.stdout: key = AQDnMZxp3Yz2AhAA2Ht/iCqUqVsaOtmUBtlqqw== 2026-02-23T10:54:51.605 INFO:teuthology.orchestra.run.trial198.stdout:[osd.0] 2026-02-23T10:54:51.605 INFO:teuthology.orchestra.run.trial198.stdout: key = AQDsMZxp+UDfBhAArut42XTGmhOmQWcUt5uxrQ== 2026-02-23T10:54:51.605 INFO:teuthology.orchestra.run.trial198.stdout:[osd.1] 2026-02-23T10:54:51.605 INFO:teuthology.orchestra.run.trial198.stdout: key = AQDyMZxpp+INBhAABXnsUTywowb+0Hndyjl6Zg== 2026-02-23T10:54:51.605 INFO:teuthology.orchestra.run.trial198.stdout:[osd.2] 2026-02-23T10:54:51.606 INFO:teuthology.orchestra.run.trial198.stdout: key = AQD3MZxp3seJOxAAZlNSGlckM1n2Br4ps0tfRw== 2026-02-23T10:54:51.606 INFO:teuthology.orchestra.run.trial198.stdout:[client.0] 2026-02-23T10:54:51.606 INFO:teuthology.orchestra.run.trial198.stdout: key = AQDnMZxpWBiPBRAAWoTW+rNQo1aBYeSihPKipQ== 2026-02-23T10:54:51.607 DEBUG:teuthology.orchestra.run.trial198:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2026-02-23T10:54:51.694 DEBUG:teuthology.orchestra.run.trial198:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2026-02-23T10:54:51.742 DEBUG:teuthology.orchestra.run.trial198:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2026-02-23T10:54:51.791 DEBUG:teuthology.orchestra.run.trial198:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2026-02-23T10:54:51.838 DEBUG:teuthology.orchestra.run.trial198:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2026-02-23T10:54:51.888 INFO:tasks.ceph:Running mkfs on mon nodes... 2026-02-23T10:54:51.888 DEBUG:teuthology.orchestra.run.trial198:> sudo mkdir -p /var/lib/ceph/mon/ceph-a 2026-02-23T10:54:51.912 DEBUG:teuthology.orchestra.run.trial198:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i a --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2026-02-23T10:54:51.999 INFO:teuthology.orchestra.run.trial198.stderr:2026-02-23T10:54:51.997+0000 7fba0289bc00 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T10:54:51.999 INFO:teuthology.orchestra.run.trial198.stderr:2026-02-23T10:54:51.997+0000 7fba0289bc00 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T10:54:52.011 DEBUG:teuthology.orchestra.run.trial198:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-a 2026-02-23T10:54:52.036 DEBUG:teuthology.orchestra.run.trial198:> rm -- /home/ubuntu/cephtest/ceph.monmap 2026-02-23T10:54:52.093 INFO:tasks.ceph:Starting mon daemons in cluster ceph... 2026-02-23T10:54:52.093 INFO:tasks.ceph.mon.a:Restarting daemon 2026-02-23T10:54:52.093 DEBUG:teuthology.orchestra.run.trial198:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2026-02-23T10:54:52.135 INFO:tasks.ceph.mon.a:Started 2026-02-23T10:54:52.135 INFO:tasks.ceph:Starting mgr daemons in cluster ceph... 2026-02-23T10:54:52.136 INFO:tasks.ceph.mgr.x:Restarting daemon 2026-02-23T10:54:52.136 DEBUG:teuthology.orchestra.run.trial198:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i x 2026-02-23T10:54:52.138 INFO:tasks.ceph.mgr.x:Started 2026-02-23T10:54:52.139 DEBUG:tasks.ceph:set 0 configs 2026-02-23T10:54:52.139 DEBUG:teuthology.orchestra.run.trial198:> sudo ceph --cluster ceph config dump 2026-02-23T10:54:52.198 INFO:tasks.ceph.mon.a.trial198.stderr:2026-02-23T10:54:52.195+0000 7fb1ca233c00 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T10:54:52.198 INFO:tasks.ceph.mon.a.trial198.stderr:2026-02-23T10:54:52.195+0000 7fb1ca233c00 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T10:54:52.198 INFO:tasks.ceph.mon.a.trial198.stderr:2026-02-23T10:54:52.196+0000 7fb1ca233c00 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T10:54:52.202 INFO:tasks.ceph.mgr.x.trial198.stderr:2026-02-23T10:54:52.200+0000 7fdc5f030fc0 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T10:54:52.215 INFO:teuthology.orchestra.run.trial198.stderr:2026-02-23T10:54:52.210+0000 7f6d54164640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T10:54:52.218 INFO:teuthology.orchestra.run.trial198.stderr:2026-02-23T10:54:52.216+0000 7f6d54164640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T10:54:52.251 INFO:teuthology.orchestra.run.trial198.stdout:WHO MASK LEVEL OPTION VALUE RO 2026-02-23T10:54:52.259 INFO:tasks.ceph:Setting crush tunables to default 2026-02-23T10:54:52.259 DEBUG:teuthology.orchestra.run.trial198:> sudo ceph --cluster ceph osd crush tunables default 2026-02-23T10:54:52.335 INFO:teuthology.orchestra.run.trial198.stderr:2026-02-23T10:54:52.329+0000 7f005f30b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T10:54:52.337 INFO:teuthology.orchestra.run.trial198.stderr:2026-02-23T10:54:52.335+0000 7f005e082640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T10:54:52.374 INFO:teuthology.orchestra.run.trial198.stderr:adjusted tunables profile to default 2026-02-23T10:54:52.383 INFO:tasks.ceph:check_enable_crimson: True 2026-02-23T10:54:52.383 INFO:tasks.ceph:check_enable_crimson: setting set-allow-crimson 2026-02-23T10:54:52.383 DEBUG:teuthology.orchestra.run.trial198:> sudo ceph --cluster ceph osd set-allow-crimson --yes-i-really-mean-it 2026-02-23T10:54:52.406 INFO:tasks.ceph.mgr.x.trial198.stderr:2026-02-23T10:54:52.404+0000 7fdc5f030fc0 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T10:54:52.406 INFO:tasks.ceph.mgr.x.trial198.stderr:2026-02-23T10:54:52.404+0000 7fdc5f030fc0 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T10:54:52.457 INFO:teuthology.orchestra.run.trial198.stderr:2026-02-23T10:54:52.453+0000 7f274eb0b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T10:54:52.460 INFO:teuthology.orchestra.run.trial198.stderr:2026-02-23T10:54:52.458+0000 7f274eb0b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T10:54:52.500 INFO:tasks.ceph.mon.a.trial198.stderr:2026-02-23T10:54:52.498+0000 7fb1c15f1640 -1 WARNING: experimental feature 'crimson' is enabled 2026-02-23T10:54:52.500 INFO:tasks.ceph.mon.a.trial198.stderr:Please be aware that this feature is experimental, untested, 2026-02-23T10:54:52.500 INFO:tasks.ceph.mon.a.trial198.stderr:unsupported, and may result in data corruption, data loss, 2026-02-23T10:54:52.500 INFO:tasks.ceph.mon.a.trial198.stderr:and/or irreparable damage to your cluster. Do not use 2026-02-23T10:54:52.500 INFO:tasks.ceph.mon.a.trial198.stderr:feature with important data. 2026-02-23T10:54:52.500 INFO:tasks.ceph.mon.a.trial198.stderr: 2026-02-23T10:54:53.235 INFO:tasks.ceph.mgr.x.trial198.stderr:/usr/lib64/python3.9/site-packages/scipy/__init__.py:73: UserWarning: NumPy was imported from a Python sub-interpreter but NumPy does not properly support sub-interpreters. This will likely work for most users but might cause hard to track down issues or subtle bugs. A common user of the rare sub-interpreter feature is wsgi which also allows single-interpreter mode. 2026-02-23T10:54:53.235 INFO:tasks.ceph.mgr.x.trial198.stderr:Improvements in the case of bugs are welcome, but is not on the NumPy roadmap, and full support may require significant effort to achieve. 2026-02-23T10:54:53.235 INFO:tasks.ceph.mgr.x.trial198.stderr: from numpy import show_config as show_numpy_config 2026-02-23T10:54:53.384 INFO:tasks.ceph:Starting osd daemons in cluster ceph... 2026-02-23T10:54:53.384 DEBUG:teuthology.orchestra.run.trial198:> set -ex 2026-02-23T10:54:53.384 DEBUG:teuthology.orchestra.run.trial198:> sudo dd if=/var/lib/ceph/osd/ceph-0/fsid of=/dev/stdout 2026-02-23T10:54:53.449 DEBUG:teuthology.orchestra.run.trial198:> set -ex 2026-02-23T10:54:53.449 DEBUG:teuthology.orchestra.run.trial198:> sudo dd if=/var/lib/ceph/osd/ceph-1/fsid of=/dev/stdout 2026-02-23T10:54:53.515 DEBUG:teuthology.orchestra.run.trial198:> set -ex 2026-02-23T10:54:53.515 DEBUG:teuthology.orchestra.run.trial198:> sudo dd if=/var/lib/ceph/osd/ceph-2/fsid of=/dev/stdout 2026-02-23T10:54:53.580 DEBUG:teuthology.orchestra.run.trial198:> sudo ceph --cluster ceph osd new 4acd148c-ad73-48ca-abb1-37def718455b 0 2026-02-23T10:54:53.694 INFO:teuthology.orchestra.run.trial198.stderr:2026-02-23T10:54:53.689+0000 7fc861a5a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T10:54:53.697 INFO:teuthology.orchestra.run.trial198.stderr:2026-02-23T10:54:53.695+0000 7fc861a5a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T10:54:53.735 INFO:teuthology.orchestra.run.trial198.stdout:0 2026-02-23T10:54:53.743 DEBUG:teuthology.orchestra.run.trial198:> sudo ceph --cluster ceph osd new f6b3e06e-2b8b-4194-ba73-c0edaaff09dc 1 2026-02-23T10:54:53.817 INFO:teuthology.orchestra.run.trial198.stderr:2026-02-23T10:54:53.812+0000 7fc921dc1640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T10:54:53.819 INFO:teuthology.orchestra.run.trial198.stderr:2026-02-23T10:54:53.817+0000 7fc921dc1640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T10:54:53.858 INFO:teuthology.orchestra.run.trial198.stdout:1 2026-02-23T10:54:53.866 DEBUG:teuthology.orchestra.run.trial198:> sudo ceph --cluster ceph osd new d28833e3-00d5-418c-8ba8-81a1c18e33e2 2 2026-02-23T10:54:53.941 INFO:teuthology.orchestra.run.trial198.stderr:2026-02-23T10:54:53.935+0000 7f2d0ba5a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T10:54:53.943 INFO:teuthology.orchestra.run.trial198.stderr:2026-02-23T10:54:53.941+0000 7f2d0ba5a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T10:54:53.981 INFO:teuthology.orchestra.run.trial198.stdout:2 2026-02-23T10:54:53.990 DEBUG:teuthology.orchestra.run.trial198:> sudo install -b -o ubuntu -g ubuntu /dev/null /var/log/ceph/ceph-osd.0.log 2026-02-23T10:54:54.018 INFO:tasks.ceph.osd.0:Restarting daemon 2026-02-23T10:54:54.018 DEBUG:teuthology.orchestra.run.trial198:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 0 2>> /var/log/ceph/ceph-osd.0.log 2026-02-23T10:54:54.060 INFO:tasks.ceph.osd.0:Started 2026-02-23T10:54:54.060 DEBUG:teuthology.orchestra.run.trial198:> sudo install -b -o ubuntu -g ubuntu /dev/null /var/log/ceph/ceph-osd.1.log 2026-02-23T10:54:54.090 INFO:tasks.ceph.osd.1:Restarting daemon 2026-02-23T10:54:54.090 DEBUG:teuthology.orchestra.run.trial198:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 1 2>> /var/log/ceph/ceph-osd.1.log 2026-02-23T10:54:54.132 INFO:tasks.ceph.osd.1:Started 2026-02-23T10:54:54.133 DEBUG:teuthology.orchestra.run.trial198:> sudo install -b -o ubuntu -g ubuntu /dev/null /var/log/ceph/ceph-osd.2.log 2026-02-23T10:54:54.161 INFO:tasks.ceph.osd.2:Restarting daemon 2026-02-23T10:54:54.161 DEBUG:teuthology.orchestra.run.trial198:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 2 2>> /var/log/ceph/ceph-osd.2.log 2026-02-23T10:54:54.203 INFO:tasks.ceph.osd.2:Started 2026-02-23T10:54:54.204 DEBUG:teuthology.orchestra.run.trial198:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2026-02-23T10:54:54.280 INFO:teuthology.orchestra.run.trial198.stderr:2026-02-23T10:54:54.275+0000 7fc3d499c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T10:54:54.282 INFO:teuthology.orchestra.run.trial198.stderr:2026-02-23T10:54:54.280+0000 7fc3d499c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T10:54:54.319 INFO:teuthology.orchestra.run.trial198.stdout: 2026-02-23T10:54:54.319 INFO:teuthology.orchestra.run.trial198.stdout:{"epoch":6,"fsid":"8ebafdf4-9523-4bab-a465-a90635e02d85","created":"2026-02-23T10:54:52.211292+0000","modified":"2026-02-23T10:54:53.980646+0000","last_up_change":"0.000000","last_in_change":"2026-02-23T10:54:53.980646+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[],"osds":[{"osd":0,"uuid":"4acd148c-ad73-48ca-abb1-37def718455b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"f6b3e06e-2b8b-4194-ba73-c0edaaff09dc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"d28833e3-00d5-418c-8ba8-81a1c18e33e2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T10:54:54.328 INFO:tasks.ceph.ceph_manager.ceph:[] 2026-02-23T10:54:54.328 INFO:tasks.ceph:Waiting for OSDs to come up 2026-02-23T10:54:54.630 DEBUG:teuthology.orchestra.run.trial198:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2026-02-23T10:54:54.698 INFO:teuthology.misc.health.trial198.stderr:2026-02-23T10:54:54.693+0000 7f6dbb30b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T10:54:54.700 INFO:teuthology.misc.health.trial198.stderr:2026-02-23T10:54:54.698+0000 7f6dbb30b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T10:54:54.734 INFO:teuthology.misc.health.trial198.stdout: 2026-02-23T10:54:54.735 INFO:teuthology.misc.health.trial198.stdout:{"epoch":6,"fsid":"8ebafdf4-9523-4bab-a465-a90635e02d85","created":"2026-02-23T10:54:52.211292+0000","modified":"2026-02-23T10:54:53.980646+0000","last_up_change":"0.000000","last_in_change":"2026-02-23T10:54:53.980646+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[],"osds":[{"osd":0,"uuid":"4acd148c-ad73-48ca-abb1-37def718455b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"f6b3e06e-2b8b-4194-ba73-c0edaaff09dc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"d28833e3-00d5-418c-8ba8-81a1c18e33e2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T10:54:54.742 DEBUG:teuthology.misc:0 of 3 OSDs are up 2026-02-23T10:54:55.458 INFO:tasks.ceph.mgr.x.trial198.stderr:2026-02-23T10:54:55.456+0000 7fdb7bb95640 -1 mgr.server handle_report got status from non-daemon mon.a 2026-02-23T10:55:01.045 DEBUG:teuthology.orchestra.run.trial198:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2026-02-23T10:55:01.114 INFO:teuthology.misc.health.trial198.stderr:2026-02-23T10:55:01.111+0000 7f8dd11d3640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T10:55:01.116 INFO:teuthology.misc.health.trial198.stderr:2026-02-23T10:55:01.115+0000 7f8dd11d3640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T10:55:01.232 INFO:teuthology.misc.health.trial198.stdout: 2026-02-23T10:55:01.232 INFO:teuthology.misc.health.trial198.stdout:{"epoch":7,"fsid":"8ebafdf4-9523-4bab-a465-a90635e02d85","created":"2026-02-23T10:54:52.211292+0000","modified":"2026-02-23T10:55:00.771156+0000","last_up_change":"0.000000","last_in_change":"2026-02-23T10:54:53.980646+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":3,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[],"osds":[{"osd":0,"uuid":"4acd148c-ad73-48ca-abb1-37def718455b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"f6b3e06e-2b8b-4194-ba73-c0edaaff09dc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"d28833e3-00d5-418c-8ba8-81a1c18e33e2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T10:55:01.241 DEBUG:teuthology.misc:0 of 3 OSDs are up 2026-02-23T10:55:07.544 DEBUG:teuthology.orchestra.run.trial198:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2026-02-23T10:55:07.615 INFO:teuthology.misc.health.trial198.stderr:2026-02-23T10:55:07.612+0000 7fc30f70b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T10:55:07.617 INFO:teuthology.misc.health.trial198.stderr:2026-02-23T10:55:07.616+0000 7fc30f70b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T10:55:07.732 INFO:teuthology.misc.health.trial198.stdout: 2026-02-23T10:55:07.733 INFO:teuthology.misc.health.trial198.stdout:{"epoch":13,"fsid":"8ebafdf4-9523-4bab-a465-a90635e02d85","created":"2026-02-23T10:54:52.211292+0000","modified":"2026-02-23T10:55:06.794539+0000","last_up_change":"2026-02-23T10:55:03.782459+0000","last_in_change":"2026-02-23T10:54:53.980646+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T10:55:04.470198+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"13","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.9900000095367432,"score_stable":2.9900000095367432,"optimal_score":0.67000001668930054,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"4acd148c-ad73-48ca-abb1-37def718455b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.198:6803","nonce":1261899012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.198:6802","nonce":1261899012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.198:6807","nonce":1261899012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.198:6804","nonce":1261899012}]},"public_addr":"10.20.193.198:6803/1261899012","cluster_addr":"10.20.193.198:6802/1261899012","heartbeat_back_addr":"10.20.193.198:6807/1261899012","heartbeat_front_addr":"10.20.193.198:6804/1261899012","state":["exists","up"]},{"osd":1,"uuid":"f6b3e06e-2b8b-4194-ba73-c0edaaff09dc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":10,"up_thru":11,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.198:6805","nonce":3805786617}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.198:6806","nonce":3805786617}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.198:6811","nonce":3805786617}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.198:6808","nonce":3805786617}]},"public_addr":"10.20.193.198:6805/3805786617","cluster_addr":"10.20.193.198:6806/3805786617","heartbeat_back_addr":"10.20.193.198:6811/3805786617","heartbeat_front_addr":"10.20.193.198:6808/3805786617","state":["exists","up"]},{"osd":2,"uuid":"d28833e3-00d5-418c-8ba8-81a1c18e33e2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":10,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.198:6809","nonce":677970331}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.198:6810","nonce":677970331}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.198:6813","nonce":677970331}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.198:6812","nonce":677970331}]},"public_addr":"10.20.193.198:6809/677970331","cluster_addr":"10.20.193.198:6810/677970331","heartbeat_back_addr":"10.20.193.198:6813/677970331","heartbeat_front_addr":"10.20.193.198:6812/677970331","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T10:55:07.741 DEBUG:teuthology.misc:3 of 3 OSDs are up 2026-02-23T10:55:07.742 INFO:tasks.ceph:Creating RBD pool 2026-02-23T10:55:07.742 DEBUG:teuthology.orchestra.run.trial198:> sudo ceph --cluster ceph osd pool create rbd 8 2026-02-23T10:55:07.815 INFO:teuthology.orchestra.run.trial198.stderr:2026-02-23T10:55:07.811+0000 7f60462cb640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T10:55:07.818 INFO:teuthology.orchestra.run.trial198.stderr:2026-02-23T10:55:07.817+0000 7f60462cb640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T10:55:08.802 INFO:teuthology.orchestra.run.trial198.stderr:pool 'rbd' created 2026-02-23T10:55:08.815 DEBUG:teuthology.orchestra.run.trial198:> rbd --cluster ceph pool init rbd 2026-02-23T10:55:08.855 INFO:teuthology.orchestra.run.trial198.stderr:2026-02-23T10:55:08.854+0000 7f5e2f419240 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T10:55:08.855 INFO:teuthology.orchestra.run.trial198.stderr:2026-02-23T10:55:08.854+0000 7f5e2f419240 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T10:55:08.858 INFO:teuthology.orchestra.run.trial198.stderr:2026-02-23T10:55:08.857+0000 7f5e2f419240 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T10:55:11.830 INFO:tasks.ceph:Starting mds daemons in cluster ceph... 2026-02-23T10:55:11.831 DEBUG:teuthology.orchestra.run.trial198:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config log 1 --format=json 2026-02-23T10:55:11.831 INFO:tasks.daemonwatchdog.daemon_watchdog:watchdog starting 2026-02-23T10:55:11.947 INFO:teuthology.orchestra.run.trial198.stderr:2026-02-23T10:55:11.942+0000 7fc16041a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T10:55:11.949 INFO:teuthology.orchestra.run.trial198.stderr:2026-02-23T10:55:11.948+0000 7fc16041a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T10:55:12.062 INFO:teuthology.orchestra.run.trial198.stdout: 2026-02-23T10:55:12.071 INFO:teuthology.orchestra.run.trial198.stdout:[{"version":1,"timestamp":"0.000000","name":"","changes":[]}] 2026-02-23T10:55:12.071 INFO:tasks.ceph_manager:config epoch is 1 2026-02-23T10:55:12.072 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2026-02-23T10:55:12.072 INFO:tasks.ceph.ceph_manager.ceph:waiting for mgr available 2026-02-23T10:55:12.072 DEBUG:teuthology.orchestra.run.trial198:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr dump --format=json 2026-02-23T10:55:12.147 INFO:teuthology.orchestra.run.trial198.stderr:2026-02-23T10:55:12.144+0000 7f849b25a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T10:55:12.149 INFO:teuthology.orchestra.run.trial198.stderr:2026-02-23T10:55:12.148+0000 7f849b25a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T10:55:12.294 INFO:teuthology.orchestra.run.trial198.stdout: 2026-02-23T10:55:12.303 INFO:teuthology.orchestra.run.trial198.stdout:{"epoch":5,"flags":0,"active_gid":4105,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.198:6800","nonce":222776289},{"type":"v1","addr":"10.20.193.198:6801","nonce":222776289}]},"active_addr":"10.20.193.198:6801/222776289","active_change":"2026-02-23T10:54:54.452367+0000","active_mgr_features":4544132024016699391,"available":true,"standbys":[],"modules":["iostat","nfs"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to, use commas to separate multiple","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"update_pg_upmap_activity":{"name":"update_pg_upmap_activity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Updates pg_upmap activity stats to be used in `balancer status detail`","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often the agent on each host will gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"The first TCP port the agent will try to bind to. Up to 1000 subsequent ports will be attempted if this port cannot be bound","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this option can facilitate debugging daemons that encounter runtime problems.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"How frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"Fraction of total system memory to divide among autotuned daemons. Converged systems hosting both Ceph and compute should set a lower value; systems with more than 10 GiB of memory per OSD may benefit from a slightly larger value so that the abundance of RAM can be fully utilized","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command persistent logging","long_desc":"","tags":[],"see_also":[]},"certificate_automated_rotation_enabled":{"name":"certificate_automated_rotation_enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"This flag controls whether cephadm automatically rotates certificates upon expiration.","long_desc":"","tags":[],"see_also":[]},"certificate_check_debug_mode":{"name":"certificate_check_debug_mode","type":"bool","level":"dev","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"FOR TESTING ONLY: This flag forces the certificate check instead of waiting for certificate_check_period.","long_desc":"","tags":[],"see_also":[]},"certificate_check_period":{"name":"certificate_check_period","type":"int","level":"advanced","flags":0,"default_value":"1","min":"0","max":"30","enum_allowed":[],"desc":"Specifies how often (in days) the certificate should be checked for validity.","long_desc":"","tags":[],"see_also":[]},"certificate_duration_days":{"name":"certificate_duration_days","type":"int","level":"advanced","flags":0,"default_value":"1095","min":"90","max":"3650","enum_allowed":[],"desc":"Specifies the duration of self-signed certificates generated and signed by the cephadm root CA","long_desc":"","tags":[],"see_also":[]},"certificate_renewal_threshold_days":{"name":"certificate_renewal_threshold_days","type":"int","level":"advanced","flags":0,"default_value":"30","min":"10","max":"90","enum_allowed":[],"desc":"Specifies the lead time in days to initiate certificate renewal before expiration.","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently Podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.28.1","min":"","max":"","enum_allowed":[],"desc":"Alertmanager container image","long_desc":"","tags":[],"see_also":[]},"container_image_alloy":{"name":"container_image_alloy","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/alloy:latest","min":"","max":"","enum_allowed":[],"desc":"Alloy container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"Elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:12.3.1","min":"","max":"","enum_allowed":[],"desc":"Grafana container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"Haproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_nginx":{"name":"container_image_nginx","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nginx:sclorg-nginx-126","min":"","max":"","enum_allowed":[],"desc":"Nginx container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.9.1","min":"","max":"","enum_allowed":[],"desc":"Node exporter container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.6","min":"","max":"","enum_allowed":[],"desc":"Nvmeof container image","long_desc":"","tags":[],"see_also":[]},"container_image_oauth2_proxy":{"name":"container_image_oauth2_proxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/oauth2-proxy/oauth2-proxy:v7.6.0","min":"","max":"","enum_allowed":[],"desc":"Oauth2 proxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v3.6.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba":{"name":"container_image_samba","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-server:devbuilds-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba_metrics":{"name":"container_image_samba_metrics","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-metrics:devbuilds-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba metrics container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"Snmp gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"Seconds for which to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"quay.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"Seconds for which to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"Seconds for which to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"Location of dashboards to include in Grafana","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"How frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether the hardware monitoring daemon be deployed on every host?","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at the debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"Max number of daemons per service per host: guard against mistakes and runaways","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"Max number of OSDs that will be drained simultaneously when OSDs are removed","long_desc":"","tags":[],"see_also":[]},"max_parallel_osd_upgrades":{"name":"max_parallel_osd_upgrades","type":"int","level":"advanced","flags":0,"default_value":"16","min":"","max":"","enum_allowed":[],"desc":"Maximum number of OSD daemons upgraded in parallel.","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"Mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default IP address for RedFish API (OOB management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"Location of alerts to include in Prometheus Alertmanager","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry URL for login purposes. This is not the default registry.","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"Cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Customized SSH config file to distribute to managed hosts","long_desc":"","tags":[],"see_also":[]},"ssh_keepalive_count_max":{"name":"ssh_keepalive_count_max","type":"int","level":"advanced","flags":0,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"How many sequential SSH connection liveness check failures before the host is marked offline","long_desc":"","tags":[],"see_also":[]},"ssh_keepalive_interval":{"name":"ssh_keepalive_interval","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"How often SSH connections are checked for liveness","long_desc":"","tags":[],"see_also":[]},"stray_daemon_check_interval":{"name":"stray_daemon_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"How frequently cephadm should check for the presence of stray daemons","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use the cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest to ensure that all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MANAGED_BY_CLUSTERS":{"name":"MANAGED_BY_CLUSTERS","type":"str","level":"advanced","flags":0,"default_value":"[]","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MULTICLUSTER_CONFIG":{"name":"MULTICLUSTER_CONFIG","type":"str","level":"advanced","flags":0,"default_value":"{}","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROM_ALERT_CREDENTIAL_CACHE_TTL":{"name":"PROM_ALERT_CREDENTIAL_CACHE_TTL","type":"int","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_HOSTNAME_PER_DAEMON":{"name":"RGW_HOSTNAME_PER_DAEMON","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crypto_caller":{"name":"crypto_caller","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sso_oauth2":{"name":"sso_oauth2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"healthcheck_history_max_entries":{"name":"healthcheck_history_max_entries","type":"int","level":"advanced","flags":1,"default_value":"1000","min":"","max":"","enum_allowed":[],"desc":"Maximum number of health check history entries to keep","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"secondary_zone_period_retry_limit":{"name":"secondary_zone_period_retry_limit","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"RGW module period update retry limit for secondary site","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"prometheus_tls_secret_name":{"name":"prometheus_tls_secret_name","type":"str","level":"advanced","flags":0,"default_value":"rook-ceph-prometheus-server-tls","min":"","max":"","enum_allowed":[],"desc":"name of tls secret in k8s for prometheus","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"smb","can_run":true,"error_string":"","module_options":{"internal_store_backend":{"name":"internal_store_backend","type":"str","level":"dev","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"set internal store backend. for develoment and testing only","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"update_orchestration":{"name":"update_orchestration","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically update orchestration when smb resources are changed","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"pause_cloning":{"name":"pause_cloning","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Pause asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"pause_purging":{"name":"pause_purging","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Pause asynchronous subvolume purge threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"squid":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"tentacle":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"force_disabled_modules":{},"last_failure_osd_epoch":0,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"10.20.193.198:0","nonce":4107696505}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"10.20.193.198:0","nonce":4214729177}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"10.20.193.198:0","nonce":672903579}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"10.20.193.198:0","nonce":3891824874}]}]} 2026-02-23T10:55:12.307 INFO:tasks.ceph.ceph_manager.ceph:mgr available! 2026-02-23T10:55:12.307 INFO:tasks.ceph.ceph_manager.ceph:waiting for all up 2026-02-23T10:55:12.308 DEBUG:teuthology.orchestra.run.trial198:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2026-02-23T10:55:12.384 INFO:teuthology.orchestra.run.trial198.stderr:2026-02-23T10:55:12.381+0000 7ff18545a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T10:55:12.386 INFO:teuthology.orchestra.run.trial198.stderr:2026-02-23T10:55:12.385+0000 7ff18545a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T10:55:12.498 INFO:teuthology.orchestra.run.trial198.stdout: 2026-02-23T10:55:12.498 INFO:teuthology.orchestra.run.trial198.stdout:{"epoch":17,"fsid":"8ebafdf4-9523-4bab-a465-a90635e02d85","created":"2026-02-23T10:54:52.211292+0000","modified":"2026-02-23T10:55:11.807803+0000","last_up_change":"2026-02-23T10:55:03.782459+0000","last_in_change":"2026-02-23T10:54:53.980646+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T10:55:04.470198+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"13","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.9900000095367432,"score_stable":2.9900000095367432,"optimal_score":0.67000001668930054,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2026-02-23T10:55:07.934221+0000","flags":270337,"flags_names":"hashpspool,selfmanaged_snaps,crimson","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":2,"snap_epoch":17,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"4acd148c-ad73-48ca-abb1-37def718455b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":14,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.198:6803","nonce":1261899012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.198:6802","nonce":1261899012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.198:6807","nonce":1261899012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.198:6804","nonce":1261899012}]},"public_addr":"10.20.193.198:6803/1261899012","cluster_addr":"10.20.193.198:6802/1261899012","heartbeat_back_addr":"10.20.193.198:6807/1261899012","heartbeat_front_addr":"10.20.193.198:6804/1261899012","state":["exists","up"]},{"osd":1,"uuid":"f6b3e06e-2b8b-4194-ba73-c0edaaff09dc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":10,"up_thru":14,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.198:6805","nonce":3805786617}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.198:6806","nonce":3805786617}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.198:6811","nonce":3805786617}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.198:6808","nonce":3805786617}]},"public_addr":"10.20.193.198:6805/3805786617","cluster_addr":"10.20.193.198:6806/3805786617","heartbeat_back_addr":"10.20.193.198:6811/3805786617","heartbeat_front_addr":"10.20.193.198:6808/3805786617","state":["exists","up"]},{"osd":2,"uuid":"d28833e3-00d5-418c-8ba8-81a1c18e33e2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":10,"up_thru":14,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.198:6809","nonce":677970331}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.198:6810","nonce":677970331}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.198:6813","nonce":677970331}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.198:6812","nonce":677970331}]},"public_addr":"10.20.193.198:6809/677970331","cluster_addr":"10.20.193.198:6810/677970331","heartbeat_back_addr":"10.20.193.198:6813/677970331","heartbeat_front_addr":"10.20.193.198:6812/677970331","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":2,"snaps":[{"begin":2,"length":1}]}],"new_removed_snaps":[{"pool":2,"snaps":[{"begin":2,"length":1}]}],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T10:55:12.508 INFO:tasks.ceph.ceph_manager.ceph:all up! 2026-02-23T10:55:12.508 DEBUG:teuthology.orchestra.run.trial198:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2026-02-23T10:55:12.585 INFO:teuthology.orchestra.run.trial198.stderr:2026-02-23T10:55:12.582+0000 7eff0330b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T10:55:12.587 INFO:teuthology.orchestra.run.trial198.stderr:2026-02-23T10:55:12.587+0000 7eff0330b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T10:55:12.700 INFO:teuthology.orchestra.run.trial198.stdout: 2026-02-23T10:55:12.700 INFO:teuthology.orchestra.run.trial198.stdout:{"epoch":17,"fsid":"8ebafdf4-9523-4bab-a465-a90635e02d85","created":"2026-02-23T10:54:52.211292+0000","modified":"2026-02-23T10:55:11.807803+0000","last_up_change":"2026-02-23T10:55:03.782459+0000","last_in_change":"2026-02-23T10:54:53.980646+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T10:55:04.470198+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"13","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.9900000095367432,"score_stable":2.9900000095367432,"optimal_score":0.67000001668930054,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2026-02-23T10:55:07.934221+0000","flags":270337,"flags_names":"hashpspool,selfmanaged_snaps,crimson","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":2,"snap_epoch":17,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"4acd148c-ad73-48ca-abb1-37def718455b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":14,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.198:6803","nonce":1261899012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.198:6802","nonce":1261899012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.198:6807","nonce":1261899012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.198:6804","nonce":1261899012}]},"public_addr":"10.20.193.198:6803/1261899012","cluster_addr":"10.20.193.198:6802/1261899012","heartbeat_back_addr":"10.20.193.198:6807/1261899012","heartbeat_front_addr":"10.20.193.198:6804/1261899012","state":["exists","up"]},{"osd":1,"uuid":"f6b3e06e-2b8b-4194-ba73-c0edaaff09dc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":10,"up_thru":14,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.198:6805","nonce":3805786617}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.198:6806","nonce":3805786617}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.198:6811","nonce":3805786617}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.198:6808","nonce":3805786617}]},"public_addr":"10.20.193.198:6805/3805786617","cluster_addr":"10.20.193.198:6806/3805786617","heartbeat_back_addr":"10.20.193.198:6811/3805786617","heartbeat_front_addr":"10.20.193.198:6808/3805786617","state":["exists","up"]},{"osd":2,"uuid":"d28833e3-00d5-418c-8ba8-81a1c18e33e2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":10,"up_thru":14,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.198:6809","nonce":677970331}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.198:6810","nonce":677970331}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.198:6813","nonce":677970331}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.198:6812","nonce":677970331}]},"public_addr":"10.20.193.198:6809/677970331","cluster_addr":"10.20.193.198:6810/677970331","heartbeat_back_addr":"10.20.193.198:6813/677970331","heartbeat_front_addr":"10.20.193.198:6812/677970331","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":2,"snaps":[{"begin":2,"length":1}]}],"new_removed_snaps":[{"pool":2,"snaps":[{"begin":2,"length":1}]}],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T10:55:12.712 DEBUG:teuthology.orchestra.run.trial198:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2026-02-23T10:55:12.713 DEBUG:teuthology.orchestra.run.trial198:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2026-02-23T10:55:12.714 DEBUG:teuthology.orchestra.run.trial198:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2026-02-23T10:55:12.795 INFO:teuthology.orchestra.run.trial198.stderr:2026-02-23T10:55:12.791+0000 7fd3c6a1a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T10:55:12.796 INFO:teuthology.orchestra.run.trial198.stderr:2026-02-23T10:55:12.791+0000 7f671a76f640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T10:55:12.796 INFO:teuthology.orchestra.run.trial198.stderr:2026-02-23T10:55:12.791+0000 7f0b7f6cb640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T10:55:12.797 INFO:teuthology.orchestra.run.trial198.stderr:2026-02-23T10:55:12.796+0000 7f671a76f640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T10:55:12.798 INFO:teuthology.orchestra.run.trial198.stderr:2026-02-23T10:55:12.796+0000 7fd3c6a1a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T10:55:12.798 INFO:teuthology.orchestra.run.trial198.stderr:2026-02-23T10:55:12.797+0000 7f0b7f6cb640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T10:55:12.814 INFO:teuthology.orchestra.run.trial198.stdout:42949672969 2026-02-23T10:55:12.815 DEBUG:teuthology.orchestra.run.trial198:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2026-02-23T10:55:12.816 INFO:teuthology.orchestra.run.trial198.stdout:42949672969 2026-02-23T10:55:12.817 DEBUG:teuthology.orchestra.run.trial198:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2026-02-23T10:55:12.818 INFO:teuthology.orchestra.run.trial198.stdout:38654705674 2026-02-23T10:55:12.818 DEBUG:teuthology.orchestra.run.trial198:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2026-02-23T10:55:12.934 INFO:teuthology.orchestra.run.trial198.stderr:2026-02-23T10:55:12.931+0000 7f0b543fd640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T10:55:12.935 INFO:teuthology.orchestra.run.trial198.stderr:2026-02-23T10:55:12.932+0000 7f23a0ec2640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T10:55:12.935 INFO:teuthology.orchestra.run.trial198.stderr:2026-02-23T10:55:12.931+0000 7fac7eb0b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T10:55:12.937 INFO:teuthology.orchestra.run.trial198.stderr:2026-02-23T10:55:12.936+0000 7f0b543fd640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T10:55:12.938 INFO:teuthology.orchestra.run.trial198.stderr:2026-02-23T10:55:12.937+0000 7f23a0ec2640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T10:55:12.938 INFO:teuthology.orchestra.run.trial198.stderr:2026-02-23T10:55:12.937+0000 7fac7eb0b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T10:55:13.052 INFO:teuthology.orchestra.run.trial198.stdout:42949672965 2026-02-23T10:55:13.056 INFO:teuthology.orchestra.run.trial198.stdout:38654705674 2026-02-23T10:55:13.058 INFO:teuthology.orchestra.run.trial198.stdout:42949672965 2026-02-23T10:55:13.063 INFO:tasks.ceph.ceph_manager.ceph:need seq 42949672969 got 42949672965 for osd.1 2026-02-23T10:55:13.067 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654705674 got 38654705674 for osd.0 2026-02-23T10:55:13.067 DEBUG:teuthology.parallel:result is None 2026-02-23T10:55:13.069 INFO:tasks.ceph.ceph_manager.ceph:need seq 42949672969 got 42949672965 for osd.2 2026-02-23T10:55:14.065 DEBUG:teuthology.orchestra.run.trial198:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2026-02-23T10:55:14.069 DEBUG:teuthology.orchestra.run.trial198:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2026-02-23T10:55:14.144 INFO:teuthology.orchestra.run.trial198.stderr:2026-02-23T10:55:14.140+0000 7fb903ecb640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T10:55:14.146 INFO:teuthology.orchestra.run.trial198.stderr:2026-02-23T10:55:14.145+0000 7fb903ecb640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T10:55:14.188 INFO:teuthology.orchestra.run.trial198.stderr:2026-02-23T10:55:14.184+0000 7f1fbb90b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T10:55:14.191 INFO:teuthology.orchestra.run.trial198.stderr:2026-02-23T10:55:14.190+0000 7f1fbb90b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T10:55:14.260 INFO:teuthology.orchestra.run.trial198.stdout:42949672965 2026-02-23T10:55:14.271 INFO:tasks.ceph.ceph_manager.ceph:need seq 42949672969 got 42949672965 for osd.1 2026-02-23T10:55:14.306 INFO:teuthology.orchestra.run.trial198.stdout:42949672965 2026-02-23T10:55:14.317 INFO:tasks.ceph.ceph_manager.ceph:need seq 42949672969 got 42949672965 for osd.2 2026-02-23T10:55:15.271 DEBUG:teuthology.orchestra.run.trial198:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2026-02-23T10:55:15.318 DEBUG:teuthology.orchestra.run.trial198:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2026-02-23T10:55:15.352 INFO:teuthology.orchestra.run.trial198.stderr:2026-02-23T10:55:15.347+0000 7f68376cb640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T10:55:15.354 INFO:teuthology.orchestra.run.trial198.stderr:2026-02-23T10:55:15.353+0000 7f68376cb640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T10:55:15.396 INFO:teuthology.orchestra.run.trial198.stderr:2026-02-23T10:55:15.392+0000 7fcf443b8640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T10:55:15.398 INFO:teuthology.orchestra.run.trial198.stderr:2026-02-23T10:55:15.397+0000 7fcf443b8640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T10:55:15.469 INFO:teuthology.orchestra.run.trial198.stdout:42949672970 2026-02-23T10:55:15.480 INFO:tasks.ceph.ceph_manager.ceph:need seq 42949672969 got 42949672970 for osd.1 2026-02-23T10:55:15.480 DEBUG:teuthology.parallel:result is None 2026-02-23T10:55:15.510 INFO:teuthology.orchestra.run.trial198.stdout:42949672970 2026-02-23T10:55:15.521 INFO:tasks.ceph.ceph_manager.ceph:need seq 42949672969 got 42949672970 for osd.2 2026-02-23T10:55:15.522 DEBUG:teuthology.parallel:result is None 2026-02-23T10:55:15.522 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2026-02-23T10:55:15.522 DEBUG:teuthology.orchestra.run.trial198:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-23T10:55:15.640 INFO:teuthology.orchestra.run.trial198.stderr:2026-02-23T10:55:15.636+0000 7fcbb0b0b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T10:55:15.642 INFO:teuthology.orchestra.run.trial198.stderr:2026-02-23T10:55:15.641+0000 7fcbb0b0b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T10:55:15.751 INFO:teuthology.orchestra.run.trial198.stdout: 2026-02-23T10:55:15.751 INFO:teuthology.orchestra.run.trial198.stderr:dumped all 2026-02-23T10:55:15.761 INFO:teuthology.orchestra.run.trial198.stdout:{"pg_ready":true,"pg_map":{"version":15,"stamp":"2026-02-23T10:55:14.456739+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":53,"num_read_kb":46,"num_write":81,"num_write_kb":1108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":54,"ondisk_log_size":54,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":3,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":283115520,"kb_used":80852,"kb_used_data":548,"kb_used_omap":0,"kb_used_meta":80256,"kb_avail":283034668,"statfs":{"total":289910292480,"available":289827500032,"internally_reserved":0,"allocated":561152,"data_stored":1293892,"data_compressed":61150,"data_compressed_allocated":81920,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":82182144},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.654266"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":15,"reported_epoch":17,"state":"active+clean","last_fresh":"2026-02-23T10:55:11.815185+0000","last_change":"2026-02-23T10:55:11.822604+0000","last_active":"2026-02-23T10:55:11.815185+0000","last_peered":"2026-02-23T10:55:11.815185+0000","last_clean":"2026-02-23T10:55:11.815185+0000","last_became_active":"2026-02-23T10:55:09.839991+0000","last_became_peered":"2026-02-23T10:55:09.839991+0000","last_unstale":"2026-02-23T10:55:11.815185+0000","last_undegraded":"2026-02-23T10:55:11.815185+0000","last_fullsized":"2026-02-23T10:55:11.815185+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-23T10:55:08.799034+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-23T10:55:08.799034+0000","last_clean_scrub_stamp":"2026-02-23T10:55:08.799034+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":15,"reported_epoch":17,"state":"active+clean","last_fresh":"2026-02-23T10:55:11.817328+0000","last_change":"2026-02-23T10:55:11.819878+0000","last_active":"2026-02-23T10:55:11.817328+0000","last_peered":"2026-02-23T10:55:11.817328+0000","last_clean":"2026-02-23T10:55:11.817328+0000","last_became_active":"2026-02-23T10:55:09.841672+0000","last_became_peered":"2026-02-23T10:55:09.841672+0000","last_unstale":"2026-02-23T10:55:11.817328+0000","last_undegraded":"2026-02-23T10:55:11.817328+0000","last_fullsized":"2026-02-23T10:55:11.817328+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-23T10:55:08.799034+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-23T10:55:08.799034+0000","last_clean_scrub_stamp":"2026-02-23T10:55:08.799034+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":15,"reported_epoch":17,"state":"active+clean","last_fresh":"2026-02-23T10:55:11.815777+0000","last_change":"2026-02-23T10:55:11.822511+0000","last_active":"2026-02-23T10:55:11.815777+0000","last_peered":"2026-02-23T10:55:11.815777+0000","last_clean":"2026-02-23T10:55:11.815777+0000","last_became_active":"2026-02-23T10:55:09.834419+0000","last_became_peered":"2026-02-23T10:55:09.834419+0000","last_unstale":"2026-02-23T10:55:11.815777+0000","last_undegraded":"2026-02-23T10:55:11.815777+0000","last_fullsized":"2026-02-23T10:55:11.815777+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-23T10:55:08.799034+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-23T10:55:08.799034+0000","last_clean_scrub_stamp":"2026-02-23T10:55:08.799034+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":15,"reported_epoch":17,"state":"active+clean","last_fresh":"2026-02-23T10:55:11.817445+0000","last_change":"2026-02-23T10:55:11.819874+0000","last_active":"2026-02-23T10:55:11.817445+0000","last_peered":"2026-02-23T10:55:11.817445+0000","last_clean":"2026-02-23T10:55:11.817445+0000","last_became_active":"2026-02-23T10:55:09.846966+0000","last_became_peered":"2026-02-23T10:55:09.846966+0000","last_unstale":"2026-02-23T10:55:11.817445+0000","last_undegraded":"2026-02-23T10:55:11.817445+0000","last_fullsized":"2026-02-23T10:55:11.817445+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-23T10:55:08.799034+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-23T10:55:08.799034+0000","last_clean_scrub_stamp":"2026-02-23T10:55:08.799034+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"16'1","reported_seq":17,"reported_epoch":17,"state":"active+clean","last_fresh":"2026-02-23T10:55:11.817943+0000","last_change":"2026-02-23T10:55:11.817943+0000","last_active":"2026-02-23T10:55:11.817943+0000","last_peered":"2026-02-23T10:55:11.817943+0000","last_clean":"2026-02-23T10:55:11.817943+0000","last_became_active":"2026-02-23T10:55:09.839193+0000","last_became_peered":"2026-02-23T10:55:09.839193+0000","last_unstale":"2026-02-23T10:55:11.817943+0000","last_undegraded":"2026-02-23T10:55:11.817943+0000","last_fullsized":"2026-02-23T10:55:11.817943+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-23T10:55:08.799034+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-23T10:55:08.799034+0000","last_clean_scrub_stamp":"2026-02-23T10:55:08.799034+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":15,"reported_epoch":17,"state":"active+clean","last_fresh":"2026-02-23T10:55:11.815862+0000","last_change":"2026-02-23T10:55:11.817133+0000","last_active":"2026-02-23T10:55:11.815862+0000","last_peered":"2026-02-23T10:55:11.815862+0000","last_clean":"2026-02-23T10:55:11.815862+0000","last_became_active":"2026-02-23T10:55:09.839412+0000","last_became_peered":"2026-02-23T10:55:09.839412+0000","last_unstale":"2026-02-23T10:55:11.815862+0000","last_undegraded":"2026-02-23T10:55:11.815862+0000","last_fullsized":"2026-02-23T10:55:11.815862+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-23T10:55:08.799034+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-23T10:55:08.799034+0000","last_clean_scrub_stamp":"2026-02-23T10:55:08.799034+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":15,"reported_epoch":17,"state":"active+clean","last_fresh":"2026-02-23T10:55:11.814223+0000","last_change":"2026-02-23T10:55:11.816017+0000","last_active":"2026-02-23T10:55:11.814223+0000","last_peered":"2026-02-23T10:55:11.814223+0000","last_clean":"2026-02-23T10:55:11.814223+0000","last_became_active":"2026-02-23T10:55:09.834010+0000","last_became_peered":"2026-02-23T10:55:09.834010+0000","last_unstale":"2026-02-23T10:55:11.814223+0000","last_undegraded":"2026-02-23T10:55:11.814223+0000","last_fullsized":"2026-02-23T10:55:11.814223+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-23T10:55:08.799034+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-23T10:55:08.799034+0000","last_clean_scrub_stamp":"2026-02-23T10:55:08.799034+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"15'1","reported_seq":16,"reported_epoch":17,"state":"active+clean","last_fresh":"2026-02-23T10:55:11.816393+0000","last_change":"2026-02-23T10:55:11.817718+0000","last_active":"2026-02-23T10:55:11.816393+0000","last_peered":"2026-02-23T10:55:11.816393+0000","last_clean":"2026-02-23T10:55:11.816393+0000","last_became_active":"2026-02-23T10:55:09.824334+0000","last_became_peered":"2026-02-23T10:55:09.824334+0000","last_unstale":"2026-02-23T10:55:11.816393+0000","last_undegraded":"2026-02-23T10:55:11.816393+0000","last_fullsized":"2026-02-23T10:55:11.816393+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-23T10:55:08.799034+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-23T10:55:08.799034+0000","last_clean_scrub_stamp":"2026-02-23T10:55:08.799034+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"12'52","reported_seq":90,"reported_epoch":17,"state":"active+clean","last_fresh":"2026-02-23T10:55:11.816201+0000","last_change":"2026-02-23T10:55:05.812642+0000","last_active":"2026-02-23T10:55:11.816201+0000","last_peered":"2026-02-23T10:55:11.816201+0000","last_clean":"2026-02-23T10:55:11.816201+0000","last_became_active":"2026-02-23T10:55:05.809738+0000","last_became_peered":"2026-02-23T10:55:05.809738+0000","last_unstale":"2026-02-23T10:55:11.816201+0000","last_undegraded":"2026-02-23T10:55:11.816201+0000","last_fullsized":"2026-02-23T10:55:11.816201+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-23T10:55:04.788039+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-23T10:55:04.788039+0000","last_clean_scrub_stamp":"2026-02-23T10:55:04.788039+0000","objects_scrubbed":0,"log_size":52,"log_dups_size":0,"ondisk_log_size":52,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":53,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":53,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":590368,"data_compressed":30575,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},"log_size":52,"ondisk_log_size":52,"up":2,"acting":2,"num_store_stats":1}],"osd_stats":[{"osd":0,"up_from":9,"seq":38654705674,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":26972,"kb_used_data":204,"kb_used_omap":0,"kb_used_meta":26752,"kb_avail":94344868,"statfs":{"total":96636764160,"available":96609144832,"internally_reserved":0,"allocated":208896,"data_stored":628093,"data_compressed":30575,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":27394048},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":10,"seq":42949672970,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":26908,"kb_used_data":140,"kb_used_omap":0,"kb_used_meta":26752,"kb_avail":94344932,"statfs":{"total":96636764160,"available":96609210368,"internally_reserved":0,"allocated":143360,"data_stored":37706,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27394048},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":10,"seq":42949672970,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":26972,"kb_used_data":204,"kb_used_omap":0,"kb_used_meta":26752,"kb_avail":94344868,"statfs":{"total":96636764160,"available":96609144832,"internally_reserved":0,"allocated":208896,"data_stored":628093,"data_compressed":30575,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":27394048},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":590368,"data_compressed":30575,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-23T10:55:15.762 DEBUG:teuthology.orchestra.run.trial198:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-23T10:55:15.839 INFO:teuthology.orchestra.run.trial198.stderr:2026-02-23T10:55:15.836+0000 7f08e0c5a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T10:55:15.842 INFO:teuthology.orchestra.run.trial198.stderr:2026-02-23T10:55:15.841+0000 7f08e0c5a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T10:55:15.954 INFO:teuthology.orchestra.run.trial198.stdout: 2026-02-23T10:55:15.955 INFO:teuthology.orchestra.run.trial198.stderr:dumped all 2026-02-23T10:55:15.963 INFO:teuthology.orchestra.run.trial198.stdout:{"pg_ready":true,"pg_map":{"version":15,"stamp":"2026-02-23T10:55:14.456739+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":53,"num_read_kb":46,"num_write":81,"num_write_kb":1108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":54,"ondisk_log_size":54,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":3,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":283115520,"kb_used":80852,"kb_used_data":548,"kb_used_omap":0,"kb_used_meta":80256,"kb_avail":283034668,"statfs":{"total":289910292480,"available":289827500032,"internally_reserved":0,"allocated":561152,"data_stored":1293892,"data_compressed":61150,"data_compressed_allocated":81920,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":82182144},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.654266"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":15,"reported_epoch":17,"state":"active+clean","last_fresh":"2026-02-23T10:55:11.815185+0000","last_change":"2026-02-23T10:55:11.822604+0000","last_active":"2026-02-23T10:55:11.815185+0000","last_peered":"2026-02-23T10:55:11.815185+0000","last_clean":"2026-02-23T10:55:11.815185+0000","last_became_active":"2026-02-23T10:55:09.839991+0000","last_became_peered":"2026-02-23T10:55:09.839991+0000","last_unstale":"2026-02-23T10:55:11.815185+0000","last_undegraded":"2026-02-23T10:55:11.815185+0000","last_fullsized":"2026-02-23T10:55:11.815185+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-23T10:55:08.799034+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-23T10:55:08.799034+0000","last_clean_scrub_stamp":"2026-02-23T10:55:08.799034+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":15,"reported_epoch":17,"state":"active+clean","last_fresh":"2026-02-23T10:55:11.817328+0000","last_change":"2026-02-23T10:55:11.819878+0000","last_active":"2026-02-23T10:55:11.817328+0000","last_peered":"2026-02-23T10:55:11.817328+0000","last_clean":"2026-02-23T10:55:11.817328+0000","last_became_active":"2026-02-23T10:55:09.841672+0000","last_became_peered":"2026-02-23T10:55:09.841672+0000","last_unstale":"2026-02-23T10:55:11.817328+0000","last_undegraded":"2026-02-23T10:55:11.817328+0000","last_fullsized":"2026-02-23T10:55:11.817328+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-23T10:55:08.799034+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-23T10:55:08.799034+0000","last_clean_scrub_stamp":"2026-02-23T10:55:08.799034+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":15,"reported_epoch":17,"state":"active+clean","last_fresh":"2026-02-23T10:55:11.815777+0000","last_change":"2026-02-23T10:55:11.822511+0000","last_active":"2026-02-23T10:55:11.815777+0000","last_peered":"2026-02-23T10:55:11.815777+0000","last_clean":"2026-02-23T10:55:11.815777+0000","last_became_active":"2026-02-23T10:55:09.834419+0000","last_became_peered":"2026-02-23T10:55:09.834419+0000","last_unstale":"2026-02-23T10:55:11.815777+0000","last_undegraded":"2026-02-23T10:55:11.815777+0000","last_fullsized":"2026-02-23T10:55:11.815777+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-23T10:55:08.799034+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-23T10:55:08.799034+0000","last_clean_scrub_stamp":"2026-02-23T10:55:08.799034+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":15,"reported_epoch":17,"state":"active+clean","last_fresh":"2026-02-23T10:55:11.817445+0000","last_change":"2026-02-23T10:55:11.819874+0000","last_active":"2026-02-23T10:55:11.817445+0000","last_peered":"2026-02-23T10:55:11.817445+0000","last_clean":"2026-02-23T10:55:11.817445+0000","last_became_active":"2026-02-23T10:55:09.846966+0000","last_became_peered":"2026-02-23T10:55:09.846966+0000","last_unstale":"2026-02-23T10:55:11.817445+0000","last_undegraded":"2026-02-23T10:55:11.817445+0000","last_fullsized":"2026-02-23T10:55:11.817445+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-23T10:55:08.799034+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-23T10:55:08.799034+0000","last_clean_scrub_stamp":"2026-02-23T10:55:08.799034+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"16'1","reported_seq":17,"reported_epoch":17,"state":"active+clean","last_fresh":"2026-02-23T10:55:11.817943+0000","last_change":"2026-02-23T10:55:11.817943+0000","last_active":"2026-02-23T10:55:11.817943+0000","last_peered":"2026-02-23T10:55:11.817943+0000","last_clean":"2026-02-23T10:55:11.817943+0000","last_became_active":"2026-02-23T10:55:09.839193+0000","last_became_peered":"2026-02-23T10:55:09.839193+0000","last_unstale":"2026-02-23T10:55:11.817943+0000","last_undegraded":"2026-02-23T10:55:11.817943+0000","last_fullsized":"2026-02-23T10:55:11.817943+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-23T10:55:08.799034+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-23T10:55:08.799034+0000","last_clean_scrub_stamp":"2026-02-23T10:55:08.799034+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":15,"reported_epoch":17,"state":"active+clean","last_fresh":"2026-02-23T10:55:11.815862+0000","last_change":"2026-02-23T10:55:11.817133+0000","last_active":"2026-02-23T10:55:11.815862+0000","last_peered":"2026-02-23T10:55:11.815862+0000","last_clean":"2026-02-23T10:55:11.815862+0000","last_became_active":"2026-02-23T10:55:09.839412+0000","last_became_peered":"2026-02-23T10:55:09.839412+0000","last_unstale":"2026-02-23T10:55:11.815862+0000","last_undegraded":"2026-02-23T10:55:11.815862+0000","last_fullsized":"2026-02-23T10:55:11.815862+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-23T10:55:08.799034+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-23T10:55:08.799034+0000","last_clean_scrub_stamp":"2026-02-23T10:55:08.799034+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":15,"reported_epoch":17,"state":"active+clean","last_fresh":"2026-02-23T10:55:11.814223+0000","last_change":"2026-02-23T10:55:11.816017+0000","last_active":"2026-02-23T10:55:11.814223+0000","last_peered":"2026-02-23T10:55:11.814223+0000","last_clean":"2026-02-23T10:55:11.814223+0000","last_became_active":"2026-02-23T10:55:09.834010+0000","last_became_peered":"2026-02-23T10:55:09.834010+0000","last_unstale":"2026-02-23T10:55:11.814223+0000","last_undegraded":"2026-02-23T10:55:11.814223+0000","last_fullsized":"2026-02-23T10:55:11.814223+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-23T10:55:08.799034+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-23T10:55:08.799034+0000","last_clean_scrub_stamp":"2026-02-23T10:55:08.799034+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"15'1","reported_seq":16,"reported_epoch":17,"state":"active+clean","last_fresh":"2026-02-23T10:55:11.816393+0000","last_change":"2026-02-23T10:55:11.817718+0000","last_active":"2026-02-23T10:55:11.816393+0000","last_peered":"2026-02-23T10:55:11.816393+0000","last_clean":"2026-02-23T10:55:11.816393+0000","last_became_active":"2026-02-23T10:55:09.824334+0000","last_became_peered":"2026-02-23T10:55:09.824334+0000","last_unstale":"2026-02-23T10:55:11.816393+0000","last_undegraded":"2026-02-23T10:55:11.816393+0000","last_fullsized":"2026-02-23T10:55:11.816393+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-23T10:55:08.799034+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-23T10:55:08.799034+0000","last_clean_scrub_stamp":"2026-02-23T10:55:08.799034+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"12'52","reported_seq":90,"reported_epoch":17,"state":"active+clean","last_fresh":"2026-02-23T10:55:11.816201+0000","last_change":"2026-02-23T10:55:05.812642+0000","last_active":"2026-02-23T10:55:11.816201+0000","last_peered":"2026-02-23T10:55:11.816201+0000","last_clean":"2026-02-23T10:55:11.816201+0000","last_became_active":"2026-02-23T10:55:05.809738+0000","last_became_peered":"2026-02-23T10:55:05.809738+0000","last_unstale":"2026-02-23T10:55:11.816201+0000","last_undegraded":"2026-02-23T10:55:11.816201+0000","last_fullsized":"2026-02-23T10:55:11.816201+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-23T10:55:04.788039+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-23T10:55:04.788039+0000","last_clean_scrub_stamp":"2026-02-23T10:55:04.788039+0000","objects_scrubbed":0,"log_size":52,"log_dups_size":0,"ondisk_log_size":52,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":53,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":53,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":590368,"data_compressed":30575,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},"log_size":52,"ondisk_log_size":52,"up":2,"acting":2,"num_store_stats":1}],"osd_stats":[{"osd":0,"up_from":9,"seq":38654705674,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":26972,"kb_used_data":204,"kb_used_omap":0,"kb_used_meta":26752,"kb_avail":94344868,"statfs":{"total":96636764160,"available":96609144832,"internally_reserved":0,"allocated":208896,"data_stored":628093,"data_compressed":30575,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":27394048},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":10,"seq":42949672970,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":26908,"kb_used_data":140,"kb_used_omap":0,"kb_used_meta":26752,"kb_avail":94344932,"statfs":{"total":96636764160,"available":96609210368,"internally_reserved":0,"allocated":143360,"data_stored":37706,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27394048},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":10,"seq":42949672970,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":26972,"kb_used_data":204,"kb_used_omap":0,"kb_used_meta":26752,"kb_avail":94344868,"statfs":{"total":96636764160,"available":96609144832,"internally_reserved":0,"allocated":208896,"data_stored":628093,"data_compressed":30575,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":27394048},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":590368,"data_compressed":30575,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-23T10:55:15.964 INFO:tasks.ceph.ceph_manager.ceph:clean! 2026-02-23T10:55:15.965 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2026-02-23T10:55:15.965 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy 2026-02-23T10:55:15.965 DEBUG:teuthology.orchestra.run.trial198:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2026-02-23T10:55:16.041 INFO:teuthology.orchestra.run.trial198.stderr:2026-02-23T10:55:16.038+0000 7fa36d1dc640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T10:55:16.043 INFO:teuthology.orchestra.run.trial198.stderr:2026-02-23T10:55:16.042+0000 7fa36d1dc640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T10:55:16.181 INFO:teuthology.orchestra.run.trial198.stdout: 2026-02-23T10:55:16.182 INFO:teuthology.orchestra.run.trial198.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2026-02-23T10:55:16.191 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy done 2026-02-23T10:55:16.191 INFO:teuthology.run_tasks:Running task workunit... 2026-02-23T10:55:16.215 INFO:tasks.workunit:Pulling workunits from ref e371433fcd1390a0b82cfb18effc422c1f5d8c1d 2026-02-23T10:55:16.216 INFO:tasks.workunit:Making a separate scratch dir for every client... 2026-02-23T10:55:16.216 DEBUG:teuthology.orchestra.run.trial198:> stat -- /home/ubuntu/cephtest/mnt.0 2026-02-23T10:55:16.232 DEBUG:teuthology.orchestra.run:got remote process result: 1 2026-02-23T10:55:16.233 INFO:teuthology.orchestra.run.trial198.stderr:stat: cannot statx '/home/ubuntu/cephtest/mnt.0': No such file or directory 2026-02-23T10:55:16.233 DEBUG:teuthology.orchestra.run.trial198:> mkdir -- /home/ubuntu/cephtest/mnt.0 2026-02-23T10:55:16.289 INFO:tasks.workunit:Created dir /home/ubuntu/cephtest/mnt.0 2026-02-23T10:55:16.289 DEBUG:teuthology.orchestra.run.trial198:> cd -- /home/ubuntu/cephtest/mnt.0 && mkdir -- client.0 2026-02-23T10:55:16.346 INFO:tasks.workunit:timeout=3h 2026-02-23T10:55:16.346 INFO:tasks.workunit:cleanup=True 2026-02-23T10:55:16.347 DEBUG:teuthology.orchestra.run.trial198:> rm -rf /home/ubuntu/cephtest/clone.client.0 && git clone https://github.com/shraddhaag/ceph.git /home/ubuntu/cephtest/clone.client.0 && cd /home/ubuntu/cephtest/clone.client.0 && git checkout e371433fcd1390a0b82cfb18effc422c1f5d8c1d 2026-02-23T10:55:16.409 INFO:tasks.workunit.client.0.trial198.stderr:Cloning into '/home/ubuntu/cephtest/clone.client.0'... 2026-02-23T10:55:38.281 INFO:tasks.workunit.client.0.trial198.stderr:Note: switching to 'e371433fcd1390a0b82cfb18effc422c1f5d8c1d'. 2026-02-23T10:55:38.282 INFO:tasks.workunit.client.0.trial198.stderr: 2026-02-23T10:55:38.282 INFO:tasks.workunit.client.0.trial198.stderr:You are in 'detached HEAD' state. You can look around, make experimental 2026-02-23T10:55:38.282 INFO:tasks.workunit.client.0.trial198.stderr:changes and commit them, and you can discard any commits you make in this 2026-02-23T10:55:38.282 INFO:tasks.workunit.client.0.trial198.stderr:state without impacting any branches by switching back to a branch. 2026-02-23T10:55:38.282 INFO:tasks.workunit.client.0.trial198.stderr: 2026-02-23T10:55:38.282 INFO:tasks.workunit.client.0.trial198.stderr:If you want to create a new branch to retain commits you create, you may 2026-02-23T10:55:38.282 INFO:tasks.workunit.client.0.trial198.stderr:do so (now or later) by using -c with the switch command. Example: 2026-02-23T10:55:38.282 INFO:tasks.workunit.client.0.trial198.stderr: 2026-02-23T10:55:38.282 INFO:tasks.workunit.client.0.trial198.stderr: git switch -c 2026-02-23T10:55:38.283 INFO:tasks.workunit.client.0.trial198.stderr: 2026-02-23T10:55:38.283 INFO:tasks.workunit.client.0.trial198.stderr:Or undo this operation with: 2026-02-23T10:55:38.283 INFO:tasks.workunit.client.0.trial198.stderr: 2026-02-23T10:55:38.283 INFO:tasks.workunit.client.0.trial198.stderr: git switch - 2026-02-23T10:55:38.283 INFO:tasks.workunit.client.0.trial198.stderr: 2026-02-23T10:55:38.283 INFO:tasks.workunit.client.0.trial198.stderr:Turn off this advice by setting config variable advice.detachedHead to false 2026-02-23T10:55:38.283 INFO:tasks.workunit.client.0.trial198.stderr: 2026-02-23T10:55:38.283 INFO:tasks.workunit.client.0.trial198.stderr:HEAD is now at e371433fcd1 qa/suites/crimson-rados: enable cephadm tests 2026-02-23T10:55:38.285 DEBUG:teuthology.orchestra.run.trial198:> cd -- /home/ubuntu/cephtest/clone.client.0/qa/workunits && if test -e Makefile ; then make ; fi && find -executable -type f -printf '%P\0' >/home/ubuntu/cephtest/workunits.list.client.0 2026-02-23T10:55:38.343 INFO:tasks.workunit.client.0.trial198.stdout:for d in direct_io fs ; do ( cd $d ; make all ) ; done 2026-02-23T10:55:38.345 INFO:tasks.workunit.client.0.trial198.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2026-02-23T10:55:38.345 INFO:tasks.workunit.client.0.trial198.stdout:cc -Wall -Wextra -D_GNU_SOURCE direct_io_test.c -o direct_io_test 2026-02-23T10:55:38.436 INFO:tasks.workunit.client.0.trial198.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_sync_io.c -o test_sync_io 2026-02-23T10:55:38.467 INFO:tasks.workunit.client.0.trial198.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_short_dio_read.c -o test_short_dio_read 2026-02-23T10:55:38.494 INFO:tasks.workunit.client.0.trial198.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2026-02-23T10:55:38.495 INFO:tasks.workunit.client.0.trial198.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2026-02-23T10:55:38.495 INFO:tasks.workunit.client.0.trial198.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_o_trunc.c -o test_o_trunc 2026-02-23T10:55:38.523 INFO:tasks.workunit.client.0.trial198.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2026-02-23T10:55:38.527 DEBUG:teuthology.orchestra.run.trial198:> set -ex 2026-02-23T10:55:38.527 DEBUG:teuthology.orchestra.run.trial198:> dd if=/home/ubuntu/cephtest/workunits.list.client.0 of=/dev/stdout 2026-02-23T10:55:38.584 INFO:tasks.workunit:Running workunits matching cls/test_cls_rbd.sh on client.0... 2026-02-23T10:55:38.587 INFO:tasks.workunit:Running workunit cls/test_cls_rbd.sh... 2026-02-23T10:55:38.587 DEBUG:teuthology.orchestra.run.trial198:workunit test cls/test_cls_rbd.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=e371433fcd1390a0b82cfb18effc422c1f5d8c1d TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 CRIMSON_COMPAT=1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/cls/test_cls_rbd.sh 2026-02-23T10:55:38.657 INFO:tasks.workunit.client.0.trial198.stdout:Running main() from gmock_main.cc 2026-02-23T10:55:38.658 INFO:tasks.workunit.client.0.trial198.stdout:[==========] Running 71 tests from 1 test suite. 2026-02-23T10:55:38.658 INFO:tasks.workunit.client.0.trial198.stdout:[----------] Global test environment set-up. 2026-02-23T10:55:38.658 INFO:tasks.workunit.client.0.trial198.stdout:[----------] 71 tests from TestClsRbd 2026-02-23T10:55:39.836 INFO:tasks.workunit.client.0.trial198.stdout:[ RUN ] TestClsRbd.get_all_features 2026-02-23T10:55:40.024 INFO:tasks.workunit.client.0.trial198.stdout:[ OK ] TestClsRbd.get_all_features (187 ms) 2026-02-23T10:55:40.024 INFO:tasks.workunit.client.0.trial198.stdout:[ RUN ] TestClsRbd.copyup 2026-02-23T10:55:40.137 INFO:tasks.workunit.client.0.trial198.stdout:[ OK ] TestClsRbd.copyup (112 ms) 2026-02-23T10:55:40.137 INFO:tasks.workunit.client.0.trial198.stdout:[ RUN ] TestClsRbd.sparse_copyup 2026-02-23T10:55:40.201 INFO:tasks.workunit.client.0.trial198.stdout:[ OK ] TestClsRbd.sparse_copyup (64 ms) 2026-02-23T10:55:40.201 INFO:tasks.workunit.client.0.trial198.stdout:[ RUN ] TestClsRbd.get_and_set_id 2026-02-23T10:55:40.237 INFO:tasks.workunit.client.0.trial198.stdout:[ OK ] TestClsRbd.get_and_set_id (36 ms) 2026-02-23T10:55:40.238 INFO:tasks.workunit.client.0.trial198.stdout:[ RUN ] TestClsRbd.add_remove_child 2026-02-23T10:55:40.295 INFO:tasks.workunit.client.0.trial198.stdout:[ OK ] TestClsRbd.add_remove_child (57 ms) 2026-02-23T10:55:40.295 INFO:tasks.workunit.client.0.trial198.stdout:[ RUN ] TestClsRbd.directory_methods 2026-02-23T10:55:40.444 INFO:tasks.workunit.client.0.trial198.stdout:[ OK ] TestClsRbd.directory_methods (149 ms) 2026-02-23T10:55:40.444 INFO:tasks.workunit.client.0.trial198.stdout:[ RUN ] TestClsRbd.create 2026-02-23T10:55:40.506 INFO:tasks.workunit.client.0.trial198.stdout:[ OK ] TestClsRbd.create (61 ms) 2026-02-23T10:55:40.506 INFO:tasks.workunit.client.0.trial198.stdout:[ RUN ] TestClsRbd.get_features 2026-02-23T10:55:40.515 INFO:tasks.workunit.client.0.trial198.stdout:[ OK ] TestClsRbd.get_features (8 ms) 2026-02-23T10:55:40.515 INFO:tasks.workunit.client.0.trial198.stdout:[ RUN ] TestClsRbd.get_object_prefix 2026-02-23T10:55:40.523 INFO:tasks.workunit.client.0.trial198.stdout:[ OK ] TestClsRbd.get_object_prefix (8 ms) 2026-02-23T10:55:40.523 INFO:tasks.workunit.client.0.trial198.stdout:[ RUN ] TestClsRbd.get_create_timestamp 2026-02-23T10:55:40.531 INFO:tasks.workunit.client.0.trial198.stdout:[ OK ] TestClsRbd.get_create_timestamp (8 ms) 2026-02-23T10:55:40.532 INFO:tasks.workunit.client.0.trial198.stdout:[ RUN ] TestClsRbd.get_access_timestamp 2026-02-23T10:55:40.539 INFO:tasks.workunit.client.0.trial198.stdout:[ OK ] TestClsRbd.get_access_timestamp (7 ms) 2026-02-23T10:55:40.539 INFO:tasks.workunit.client.0.trial198.stdout:[ RUN ] TestClsRbd.get_modify_timestamp 2026-02-23T10:55:40.547 INFO:tasks.workunit.client.0.trial198.stdout:[ OK ] TestClsRbd.get_modify_timestamp (7 ms) 2026-02-23T10:55:40.547 INFO:tasks.workunit.client.0.trial198.stdout:[ RUN ] TestClsRbd.get_data_pool 2026-02-23T10:55:40.564 INFO:tasks.workunit.client.0.trial198.stdout:[ OK ] TestClsRbd.get_data_pool (17 ms) 2026-02-23T10:55:40.565 INFO:tasks.workunit.client.0.trial198.stdout:[ RUN ] TestClsRbd.get_size 2026-02-23T10:55:40.589 INFO:tasks.workunit.client.0.trial198.stdout:[ OK ] TestClsRbd.get_size (24 ms) 2026-02-23T10:55:40.589 INFO:tasks.workunit.client.0.trial198.stdout:[ RUN ] TestClsRbd.set_size 2026-02-23T10:55:40.617 INFO:tasks.workunit.client.0.trial198.stdout:[ OK ] TestClsRbd.set_size (28 ms) 2026-02-23T10:55:40.617 INFO:tasks.workunit.client.0.trial198.stdout:[ RUN ] TestClsRbd.protection_status 2026-02-23T10:55:40.717 INFO:tasks.workunit.client.0.trial198.stdout:[ OK ] TestClsRbd.protection_status (100 ms) 2026-02-23T10:55:40.717 INFO:tasks.workunit.client.0.trial198.stdout:[ RUN ] TestClsRbd.snapshot_limits 2026-02-23T10:55:40.759 INFO:tasks.workunit.client.0.trial198.stdout:[ OK ] TestClsRbd.snapshot_limits (42 ms) 2026-02-23T10:55:40.759 INFO:tasks.workunit.client.0.trial198.stdout:[ RUN ] TestClsRbd.parents_v1 2026-02-23T10:55:41.022 INFO:tasks.workunit.client.0.trial198.stdout:[ OK ] TestClsRbd.parents_v1 (262 ms) 2026-02-23T10:55:41.023 INFO:tasks.workunit.client.0.trial198.stdout:[ RUN ] TestClsRbd.parents_v2 2026-02-23T10:55:41.201 INFO:tasks.workunit.client.0.trial198.stdout:[ OK ] TestClsRbd.parents_v2 (178 ms) 2026-02-23T10:55:41.201 INFO:tasks.workunit.client.0.trial198.stdout:[ RUN ] TestClsRbd.snapshots 2026-02-23T10:55:41.343 INFO:tasks.workunit.client.0.trial198.stdout:[ OK ] TestClsRbd.snapshots (142 ms) 2026-02-23T10:55:41.343 INFO:tasks.workunit.client.0.trial198.stdout:[ RUN ] TestClsRbd.snapid_race 2026-02-23T10:55:41.364 INFO:tasks.workunit.client.0.trial198.stdout:[ OK ] TestClsRbd.snapid_race (20 ms) 2026-02-23T10:55:41.365 INFO:tasks.workunit.client.0.trial198.stdout:[ RUN ] TestClsRbd.stripingv2 2026-02-23T10:55:41.414 INFO:tasks.workunit.client.0.trial198.stdout:[ OK ] TestClsRbd.stripingv2 (49 ms) 2026-02-23T10:55:41.414 INFO:tasks.workunit.client.0.trial198.stdout:[ RUN ] TestClsRbd.object_map_save 2026-02-23T10:55:41.422 INFO:tasks.workunit.client.0.trial198.stdout:[ OK ] TestClsRbd.object_map_save (8 ms) 2026-02-23T10:55:41.423 INFO:tasks.workunit.client.0.trial198.stdout:[ RUN ] TestClsRbd.object_map_resize 2026-02-23T10:55:41.450 INFO:tasks.workunit.client.0.trial198.stdout:[ OK ] TestClsRbd.object_map_resize (28 ms) 2026-02-23T10:55:41.451 INFO:tasks.workunit.client.0.trial198.stdout:[ RUN ] TestClsRbd.object_map_update 2026-02-23T10:55:41.475 INFO:tasks.workunit.client.0.trial198.stdout:[ OK ] TestClsRbd.object_map_update (24 ms) 2026-02-23T10:55:41.475 INFO:tasks.workunit.client.0.trial198.stdout:[ RUN ] TestClsRbd.object_map_load_enoent 2026-02-23T10:55:41.477 INFO:tasks.workunit.client.0.trial198.stdout:[ OK ] TestClsRbd.object_map_load_enoent (1 ms) 2026-02-23T10:55:41.477 INFO:tasks.workunit.client.0.trial198.stdout:[ RUN ] TestClsRbd.object_map_snap_add 2026-02-23T10:55:41.499 INFO:tasks.workunit.client.0.trial198.stdout:[ OK ] TestClsRbd.object_map_snap_add (22 ms) 2026-02-23T10:55:41.499 INFO:tasks.workunit.client.0.trial198.stdout:[ RUN ] TestClsRbd.object_map_snap_remove 2026-02-23T10:55:41.523 INFO:tasks.workunit.client.0.trial198.stdout:[ OK ] TestClsRbd.object_map_snap_remove (23 ms) 2026-02-23T10:55:41.524 INFO:tasks.workunit.client.0.trial198.stdout:[ RUN ] TestClsRbd.flags 2026-02-23T10:55:41.555 INFO:tasks.workunit.client.0.trial198.stdout:[ OK ] TestClsRbd.flags (31 ms) 2026-02-23T10:55:41.555 INFO:tasks.workunit.client.0.trial198.stdout:[ RUN ] TestClsRbd.metadata 2026-02-23T10:55:41.652 INFO:tasks.workunit.client.0.trial198.stdout:[ OK ] TestClsRbd.metadata (97 ms) 2026-02-23T10:55:41.653 INFO:tasks.workunit.client.0.trial198.stdout:[ RUN ] TestClsRbd.set_features 2026-02-23T10:55:41.692 INFO:tasks.workunit.client.0.trial198.stdout:[ OK ] TestClsRbd.set_features (39 ms) 2026-02-23T10:55:41.692 INFO:tasks.workunit.client.0.trial198.stdout:[ RUN ] TestClsRbd.mirror 2026-02-23T10:55:41.900 INFO:tasks.workunit.client.0.trial198.stdout:[ OK ] TestClsRbd.mirror (207 ms) 2026-02-23T10:55:41.900 INFO:tasks.workunit.client.0.trial198.stdout:[ RUN ] TestClsRbd.mirror_image 2026-02-23T10:55:41.989 INFO:tasks.workunit.client.0.trial198.stdout:[ OK ] TestClsRbd.mirror_image (88 ms) 2026-02-23T10:55:41.989 INFO:tasks.workunit.client.0.trial198.stdout:[ RUN ] TestClsRbd.mirror_image_status 2026-02-23T10:55:57.903 INFO:tasks.workunit.client.0.trial198.stdout:[ OK ] TestClsRbd.mirror_image_status (15914 ms) 2026-02-23T10:55:57.903 INFO:tasks.workunit.client.0.trial198.stdout:[ RUN ] TestClsRbd.mirror_image_map 2026-02-23T10:55:58.493 INFO:tasks.workunit.client.0.trial198.stdout:[ OK ] TestClsRbd.mirror_image_map (590 ms) 2026-02-23T10:55:58.494 INFO:tasks.workunit.client.0.trial198.stdout:[ RUN ] TestClsRbd.mirror_instances 2026-02-23T10:55:58.544 INFO:tasks.workunit.client.0.trial198.stdout:[ OK ] TestClsRbd.mirror_instances (50 ms) 2026-02-23T10:55:58.545 INFO:tasks.workunit.client.0.trial198.stdout:[ RUN ] TestClsRbd.mirror_snapshot 2026-02-23T10:55:58.608 INFO:tasks.workunit.client.0.trial198.stdout:[ OK ] TestClsRbd.mirror_snapshot (63 ms) 2026-02-23T10:55:58.608 INFO:tasks.workunit.client.0.trial198.stdout:[ RUN ] TestClsRbd.group_dir_list 2026-02-23T10:55:58.622 INFO:tasks.workunit.client.0.trial198.stdout:[ OK ] TestClsRbd.group_dir_list (14 ms) 2026-02-23T10:55:58.622 INFO:tasks.workunit.client.0.trial198.stdout:[ RUN ] TestClsRbd.group_dir_add 2026-02-23T10:55:58.635 INFO:tasks.workunit.client.0.trial198.stdout:[ OK ] TestClsRbd.group_dir_add (13 ms) 2026-02-23T10:55:58.636 INFO:tasks.workunit.client.0.trial198.stdout:[ RUN ] TestClsRbd.dir_add_already_existing 2026-02-23T10:55:58.657 INFO:tasks.workunit.client.0.trial198.stdout:[ OK ] TestClsRbd.dir_add_already_existing (22 ms) 2026-02-23T10:55:58.658 INFO:tasks.workunit.client.0.trial198.stdout:[ RUN ] TestClsRbd.group_dir_rename 2026-02-23T10:55:58.694 INFO:tasks.workunit.client.0.trial198.stdout:[ OK ] TestClsRbd.group_dir_rename (37 ms) 2026-02-23T10:55:58.694 INFO:tasks.workunit.client.0.trial198.stdout:[ RUN ] TestClsRbd.group_dir_remove 2026-02-23T10:55:58.713 INFO:tasks.workunit.client.0.trial198.stdout:[ OK ] TestClsRbd.group_dir_remove (18 ms) 2026-02-23T10:55:58.713 INFO:tasks.workunit.client.0.trial198.stdout:[ RUN ] TestClsRbd.group_dir_remove_missing 2026-02-23T10:55:58.737 INFO:tasks.workunit.client.0.trial198.stdout:[ OK ] TestClsRbd.group_dir_remove_missing (23 ms) 2026-02-23T10:55:58.737 INFO:tasks.workunit.client.0.trial198.stdout:[ RUN ] TestClsRbd.group_image_add 2026-02-23T10:55:58.748 INFO:tasks.workunit.client.0.trial198.stdout:[ OK ] TestClsRbd.group_image_add (11 ms) 2026-02-23T10:55:58.748 INFO:tasks.workunit.client.0.trial198.stdout:[ RUN ] TestClsRbd.group_image_remove 2026-02-23T10:55:58.769 INFO:tasks.workunit.client.0.trial198.stdout:[ OK ] TestClsRbd.group_image_remove (20 ms) 2026-02-23T10:55:58.769 INFO:tasks.workunit.client.0.trial198.stdout:[ RUN ] TestClsRbd.group_image_list 2026-02-23T10:55:58.784 INFO:tasks.workunit.client.0.trial198.stdout:[ OK ] TestClsRbd.group_image_list (15 ms) 2026-02-23T10:55:58.785 INFO:tasks.workunit.client.0.trial198.stdout:[ RUN ] TestClsRbd.group_image_clean 2026-02-23T10:55:58.806 INFO:tasks.workunit.client.0.trial198.stdout:[ OK ] TestClsRbd.group_image_clean (22 ms) 2026-02-23T10:55:58.807 INFO:tasks.workunit.client.0.trial198.stdout:[ RUN ] TestClsRbd.image_group_add 2026-02-23T10:55:58.820 INFO:tasks.workunit.client.0.trial198.stdout:[ OK ] TestClsRbd.image_group_add (13 ms) 2026-02-23T10:55:58.820 INFO:tasks.workunit.client.0.trial198.stdout:[ RUN ] TestClsRbd.image_group_remove 2026-02-23T10:55:58.837 INFO:tasks.workunit.client.0.trial198.stdout:[ OK ] TestClsRbd.image_group_remove (17 ms) 2026-02-23T10:55:58.837 INFO:tasks.workunit.client.0.trial198.stdout:[ RUN ] TestClsRbd.image_group_get 2026-02-23T10:55:58.851 INFO:tasks.workunit.client.0.trial198.stdout:[ OK ] TestClsRbd.image_group_get (14 ms) 2026-02-23T10:55:58.852 INFO:tasks.workunit.client.0.trial198.stdout:[ RUN ] TestClsRbd.group_snap_set_empty_name 2026-02-23T10:55:58.861 INFO:tasks.workunit.client.0.trial198.stdout:[ OK ] TestClsRbd.group_snap_set_empty_name (10 ms) 2026-02-23T10:55:58.862 INFO:tasks.workunit.client.0.trial198.stdout:[ RUN ] TestClsRbd.group_snap_set_empty_id 2026-02-23T10:55:58.872 INFO:tasks.workunit.client.0.trial198.stdout:[ OK ] TestClsRbd.group_snap_set_empty_id (11 ms) 2026-02-23T10:55:58.872 INFO:tasks.workunit.client.0.trial198.stdout:[ RUN ] TestClsRbd.group_snap_set_duplicate_id 2026-02-23T10:55:58.888 INFO:tasks.workunit.client.0.trial198.stdout:[ OK ] TestClsRbd.group_snap_set_duplicate_id (16 ms) 2026-02-23T10:55:58.889 INFO:tasks.workunit.client.0.trial198.stdout:[ RUN ] TestClsRbd.group_snap_set_duplicate_name 2026-02-23T10:55:58.902 INFO:tasks.workunit.client.0.trial198.stdout:[ OK ] TestClsRbd.group_snap_set_duplicate_name (14 ms) 2026-02-23T10:55:58.903 INFO:tasks.workunit.client.0.trial198.stdout:[ RUN ] TestClsRbd.group_snap_set 2026-02-23T10:55:58.916 INFO:tasks.workunit.client.0.trial198.stdout:[ OK ] TestClsRbd.group_snap_set (13 ms) 2026-02-23T10:55:58.916 INFO:tasks.workunit.client.0.trial198.stdout:[ RUN ] TestClsRbd.group_snap_list 2026-02-23T10:55:58.958 INFO:tasks.workunit.client.0.trial198.stdout:[ OK ] TestClsRbd.group_snap_list (41 ms) 2026-02-23T10:55:58.958 INFO:tasks.workunit.client.0.trial198.stdout:[ RUN ] TestClsRbd.group_snap_list_max_return 2026-02-23T10:55:59.058 INFO:tasks.workunit.client.0.trial198.stdout:[ OK ] TestClsRbd.group_snap_list_max_return (100 ms) 2026-02-23T10:55:59.058 INFO:tasks.workunit.client.0.trial198.stdout:[ RUN ] TestClsRbd.group_snap_list_max_read 2026-02-23T10:56:00.064 INFO:tasks.workunit.client.0.trial198.stdout:[ OK ] TestClsRbd.group_snap_list_max_read (1006 ms) 2026-02-23T10:56:00.064 INFO:tasks.workunit.client.0.trial198.stdout:[ RUN ] TestClsRbd.group_snap_remove 2026-02-23T10:56:00.083 INFO:tasks.workunit.client.0.trial198.stdout:[ OK ] TestClsRbd.group_snap_remove (18 ms) 2026-02-23T10:56:00.083 INFO:tasks.workunit.client.0.trial198.stdout:[ RUN ] TestClsRbd.group_snap_remove_without_order 2026-02-23T10:56:00.109 INFO:tasks.workunit.client.0.trial198.stdout:[ OK ] TestClsRbd.group_snap_remove_without_order (25 ms) 2026-02-23T10:56:00.109 INFO:tasks.workunit.client.0.trial198.stdout:[ RUN ] TestClsRbd.group_snap_get_by_id 2026-02-23T10:56:00.121 INFO:tasks.workunit.client.0.trial198.stdout:[ OK ] TestClsRbd.group_snap_get_by_id (13 ms) 2026-02-23T10:56:00.122 INFO:tasks.workunit.client.0.trial198.stdout:[ RUN ] TestClsRbd.trash_methods 2026-02-23T10:56:00.171 INFO:tasks.workunit.client.0.trial198.stdout:[ OK ] TestClsRbd.trash_methods (49 ms) 2026-02-23T10:56:00.171 INFO:tasks.workunit.client.0.trial198.stdout:[ RUN ] TestClsRbd.op_features 2026-02-23T10:56:00.210 INFO:tasks.workunit.client.0.trial198.stdout:[ OK ] TestClsRbd.op_features (38 ms) 2026-02-23T10:56:00.210 INFO:tasks.workunit.client.0.trial198.stdout:[ RUN ] TestClsRbd.clone_parent 2026-02-23T10:56:00.331 INFO:tasks.workunit.client.0.trial198.stdout:[ OK ] TestClsRbd.clone_parent (120 ms) 2026-02-23T10:56:00.331 INFO:tasks.workunit.client.0.trial198.stdout:[ RUN ] TestClsRbd.clone_parent_ns 2026-02-23T10:56:00.381 INFO:tasks.workunit.client.0.trial198.stdout:[ OK ] TestClsRbd.clone_parent_ns (49 ms) 2026-02-23T10:56:00.381 INFO:tasks.workunit.client.0.trial198.stdout:[ RUN ] TestClsRbd.clone_child 2026-02-23T10:56:00.446 INFO:tasks.workunit.client.0.trial198.stdout:[ OK ] TestClsRbd.clone_child (65 ms) 2026-02-23T10:56:00.446 INFO:tasks.workunit.client.0.trial198.stdout:[ RUN ] TestClsRbd.namespace_methods 2026-02-23T10:56:00.483 INFO:tasks.workunit.client.0.trial198.stdout:[ OK ] TestClsRbd.namespace_methods (37 ms) 2026-02-23T10:56:00.484 INFO:tasks.workunit.client.0.trial198.stdout:[ RUN ] TestClsRbd.migration 2026-02-23T10:56:00.547 INFO:tasks.workunit.client.0.trial198.stdout:[ OK ] TestClsRbd.migration (63 ms) 2026-02-23T10:56:00.547 INFO:tasks.workunit.client.0.trial198.stdout:[ RUN ] TestClsRbd.migration_v1 2026-02-23T10:56:00.594 INFO:tasks.workunit.client.0.trial198.stdout:[ OK ] TestClsRbd.migration_v1 (46 ms) 2026-02-23T10:56:00.594 INFO:tasks.workunit.client.0.trial198.stdout:[ RUN ] TestClsRbd.assert_snapc_seq 2026-02-23T10:56:02.227 INFO:tasks.workunit.client.0.trial198.stdout:[ OK ] TestClsRbd.assert_snapc_seq (1633 ms) 2026-02-23T10:56:02.228 INFO:tasks.workunit.client.0.trial198.stdout:[ RUN ] TestClsRbd.sparsify 2026-02-23T10:56:02.367 INFO:tasks.workunit.client.0.trial198.stdout:[ OK ] TestClsRbd.sparsify (139 ms) 2026-02-23T10:56:03.237 INFO:tasks.workunit.client.0.trial198.stdout:[----------] 71 tests from TestClsRbd (22530 ms total) 2026-02-23T10:56:03.237 INFO:tasks.workunit.client.0.trial198.stdout: 2026-02-23T10:56:03.237 INFO:tasks.workunit.client.0.trial198.stdout:[----------] Global test environment tear-down 2026-02-23T10:56:03.237 INFO:tasks.workunit.client.0.trial198.stdout:[==========] 71 tests from 1 test suite ran. (24579 ms total) 2026-02-23T10:56:03.238 INFO:tasks.workunit.client.0.trial198.stdout:[ PASSED ] 71 tests. 2026-02-23T10:56:03.240 INFO:teuthology.orchestra.run:Running command with timeout 3600 2026-02-23T10:56:03.240 DEBUG:teuthology.orchestra.run.trial198:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2026-02-23T10:56:03.308 INFO:tasks.workunit:Running workunits matching cls/test_cls_lock.sh on client.0... 2026-02-23T10:56:03.311 INFO:tasks.workunit:Running workunit cls/test_cls_lock.sh... 2026-02-23T10:56:03.311 DEBUG:teuthology.orchestra.run.trial198:workunit test cls/test_cls_lock.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=e371433fcd1390a0b82cfb18effc422c1f5d8c1d TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 CRIMSON_COMPAT=1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/cls/test_cls_lock.sh 2026-02-23T10:56:03.389 INFO:tasks.workunit.client.0.trial198.stdout:Running main() from gmock_main.cc 2026-02-23T10:56:03.389 INFO:tasks.workunit.client.0.trial198.stdout:[==========] Running 11 tests from 1 test suite. 2026-02-23T10:56:03.389 INFO:tasks.workunit.client.0.trial198.stdout:[----------] Global test environment set-up. 2026-02-23T10:56:03.389 INFO:tasks.workunit.client.0.trial198.stdout:[----------] 11 tests from ClsLock 2026-02-23T10:56:03.389 INFO:tasks.workunit.client.0.trial198.stdout:[ RUN ] ClsLock.TestMultiLocking 2026-02-23T10:56:05.464 INFO:tasks.workunit.client.0.trial198.stdout:lock: mylock 2026-02-23T10:56:05.464 INFO:tasks.workunit.client.0.trial198.stdout: lock_type: exclusive 2026-02-23T10:56:05.464 INFO:tasks.workunit.client.0.trial198.stdout: tag: 2026-02-23T10:56:05.464 INFO:tasks.workunit.client.0.trial198.stdout: lockers: 2026-02-23T10:56:05.464 INFO:tasks.workunit.client.0.trial198.stdout: client.4210 expiration=2026-02-23T10:58:05.439182+0000 addr=v1:10.20.193.198:0/4234914331 cookie= 2026-02-23T10:56:05.484 INFO:tasks.workunit.client.0.trial198.stdout:lock: mylock 2026-02-23T10:56:05.484 INFO:tasks.workunit.client.0.trial198.stdout: lock_type: shared 2026-02-23T10:56:05.484 INFO:tasks.workunit.client.0.trial198.stdout: tag: 2026-02-23T10:56:05.484 INFO:tasks.workunit.client.0.trial198.stdout: lockers: 2026-02-23T10:56:05.484 INFO:tasks.workunit.client.0.trial198.stdout: client.4210 expiration=2026-02-23T10:58:05.477253+0000 addr=v1:10.20.193.198:0/4234914331 cookie= 2026-02-23T10:56:05.484 INFO:tasks.workunit.client.0.trial198.stdout: client.4213 expiration=0.000000 addr=v1:10.20.193.198:0/3866632982 cookie= 2026-02-23T10:56:05.491 INFO:tasks.workunit.client.0.trial198.stdout:lock: mylock 2026-02-23T10:56:05.491 INFO:tasks.workunit.client.0.trial198.stdout: lock_type: shared 2026-02-23T10:56:05.491 INFO:tasks.workunit.client.0.trial198.stdout: tag: 2026-02-23T10:56:05.491 INFO:tasks.workunit.client.0.trial198.stdout: lockers: 2026-02-23T10:56:05.491 INFO:tasks.workunit.client.0.trial198.stdout: client.4213 expiration=0.000000 addr=v1:10.20.193.198:0/3866632982 cookie= 2026-02-23T10:56:06.243 INFO:tasks.workunit.client.0.trial198.stdout:[ OK ] ClsLock.TestMultiLocking (2854 ms) 2026-02-23T10:56:06.243 INFO:tasks.workunit.client.0.trial198.stdout:[ RUN ] ClsLock.TestMeta 2026-02-23T10:56:08.465 INFO:tasks.workunit.client.0.trial198.stdout:lock: mylock 2026-02-23T10:56:08.465 INFO:tasks.workunit.client.0.trial198.stdout: lock_type: shared 2026-02-23T10:56:08.465 INFO:tasks.workunit.client.0.trial198.stdout: tag: 2026-02-23T10:56:08.465 INFO:tasks.workunit.client.0.trial198.stdout: lockers: 2026-02-23T10:56:08.465 INFO:tasks.workunit.client.0.trial198.stdout: client.4219 expiration=0.000000 addr=v1:10.20.193.198:0/4279552792 cookie= 2026-02-23T10:56:08.476 INFO:tasks.workunit.client.0.trial198.stdout:lock: mylock 2026-02-23T10:56:08.476 INFO:tasks.workunit.client.0.trial198.stdout: lock_type: exclusive 2026-02-23T10:56:08.476 INFO:tasks.workunit.client.0.trial198.stdout: tag: new_tag 2026-02-23T10:56:08.476 INFO:tasks.workunit.client.0.trial198.stdout: lockers: 2026-02-23T10:56:08.476 INFO:tasks.workunit.client.0.trial198.stdout: client.4216 expiration=0.000000 addr=v1:10.20.193.198:0/75942136 cookie= 2026-02-23T10:56:09.259 INFO:tasks.workunit.client.0.trial198.stdout:[ OK ] ClsLock.TestMeta (3016 ms) 2026-02-23T10:56:09.260 INFO:tasks.workunit.client.0.trial198.stdout:[ RUN ] ClsLock.TestCookie 2026-02-23T10:56:11.432 INFO:tasks.workunit.client.0.trial198.stdout:lock: mylock 2026-02-23T10:56:11.432 INFO:tasks.workunit.client.0.trial198.stdout: lock_type: exclusive 2026-02-23T10:56:11.432 INFO:tasks.workunit.client.0.trial198.stdout: tag: 2026-02-23T10:56:11.432 INFO:tasks.workunit.client.0.trial198.stdout: lockers: 2026-02-23T10:56:11.446 INFO:tasks.workunit.client.0.trial198.stdout:lock: mylock 2026-02-23T10:56:11.446 INFO:tasks.workunit.client.0.trial198.stdout: lock_type: shared 2026-02-23T10:56:11.446 INFO:tasks.workunit.client.0.trial198.stdout: tag: 2026-02-23T10:56:11.446 INFO:tasks.workunit.client.0.trial198.stdout: lockers: 2026-02-23T10:56:11.446 INFO:tasks.workunit.client.0.trial198.stdout: client.4222 expiration=0.000000 addr=v1:10.20.193.198:0/1529117562 cookie= 2026-02-23T10:56:11.446 INFO:tasks.workunit.client.0.trial198.stdout: client.4222 expiration=0.000000 addr=v1:10.20.193.198:0/1529117562 cookie=new cookie 2026-02-23T10:56:12.267 INFO:tasks.workunit.client.0.trial198.stdout:[ OK ] ClsLock.TestCookie (3007 ms) 2026-02-23T10:56:12.267 INFO:tasks.workunit.client.0.trial198.stdout:[ RUN ] ClsLock.TestMultipleLocks 2026-02-23T10:56:15.279 INFO:tasks.workunit.client.0.trial198.stdout:[ OK ] ClsLock.TestMultipleLocks (3012 ms) 2026-02-23T10:56:15.279 INFO:tasks.workunit.client.0.trial198.stdout:[ RUN ] ClsLock.TestLockDuration 2026-02-23T10:56:23.298 INFO:tasks.workunit.client.0.trial198.stdout:[ OK ] ClsLock.TestLockDuration (8019 ms) 2026-02-23T10:56:23.298 INFO:tasks.workunit.client.0.trial198.stdout:[ RUN ] ClsLock.TestAssertLocked 2026-02-23T10:56:26.306 INFO:tasks.workunit.client.0.trial198.stdout:[ OK ] ClsLock.TestAssertLocked (3007 ms) 2026-02-23T10:56:26.306 INFO:tasks.workunit.client.0.trial198.stdout:[ RUN ] ClsLock.TestSetCookie 2026-02-23T10:56:29.317 INFO:tasks.workunit.client.0.trial198.stdout:[ OK ] ClsLock.TestSetCookie (3010 ms) 2026-02-23T10:56:29.317 INFO:tasks.workunit.client.0.trial198.stdout:[ RUN ] ClsLock.TestRenew 2026-02-23T10:56:50.343 INFO:tasks.workunit.client.0.trial198.stdout:[ OK ] ClsLock.TestRenew (21025 ms) 2026-02-23T10:56:50.343 INFO:tasks.workunit.client.0.trial198.stdout:[ RUN ] ClsLock.TestExclusiveEphemeralBasic 2026-02-23T10:56:57.354 INFO:tasks.workunit.client.0.trial198.stdout:[ OK ] ClsLock.TestExclusiveEphemeralBasic (7011 ms) 2026-02-23T10:56:57.354 INFO:tasks.workunit.client.0.trial198.stdout:[ RUN ] ClsLock.TestExclusiveEphemeralStealEphemeral 2026-02-23T10:57:05.374 INFO:tasks.workunit.client.0.trial198.stdout:[ OK ] ClsLock.TestExclusiveEphemeralStealEphemeral (8020 ms) 2026-02-23T10:57:05.375 INFO:tasks.workunit.client.0.trial198.stdout:[ RUN ] ClsLock.TestExclusiveEphemeralStealExclusive 2026-02-23T10:57:13.395 INFO:tasks.workunit.client.0.trial198.stdout:[ OK ] ClsLock.TestExclusiveEphemeralStealExclusive (8020 ms) 2026-02-23T10:57:13.396 INFO:tasks.workunit.client.0.trial198.stdout:[----------] 11 tests from ClsLock (70006 ms total) 2026-02-23T10:57:13.396 INFO:tasks.workunit.client.0.trial198.stdout: 2026-02-23T10:57:13.396 INFO:tasks.workunit.client.0.trial198.stdout:[----------] Global test environment tear-down 2026-02-23T10:57:13.396 INFO:tasks.workunit.client.0.trial198.stdout:[==========] 11 tests from 1 test suite ran. (70006 ms total) 2026-02-23T10:57:13.396 INFO:tasks.workunit.client.0.trial198.stdout:[ PASSED ] 11 tests. 2026-02-23T10:57:13.398 INFO:teuthology.orchestra.run:Running command with timeout 3600 2026-02-23T10:57:13.399 DEBUG:teuthology.orchestra.run.trial198:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2026-02-23T10:57:13.465 INFO:tasks.workunit:Running workunits matching cls/test_cls_journal.sh on client.0... 2026-02-23T10:57:13.468 INFO:tasks.workunit:Running workunit cls/test_cls_journal.sh... 2026-02-23T10:57:13.468 DEBUG:teuthology.orchestra.run.trial198:workunit test cls/test_cls_journal.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=e371433fcd1390a0b82cfb18effc422c1f5d8c1d TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 CRIMSON_COMPAT=1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/cls/test_cls_journal.sh 2026-02-23T10:57:13.546 INFO:tasks.workunit.client.0.trial198.stdout:Running main() from gmock_main.cc 2026-02-23T10:57:13.546 INFO:tasks.workunit.client.0.trial198.stdout:[==========] Running 27 tests from 1 test suite. 2026-02-23T10:57:13.546 INFO:tasks.workunit.client.0.trial198.stdout:[----------] Global test environment set-up. 2026-02-23T10:57:13.546 INFO:tasks.workunit.client.0.trial198.stdout:[----------] 27 tests from TestClsJournal 2026-02-23T10:57:15.404 INFO:tasks.workunit.client.0.trial198.stdout:[ RUN ] TestClsJournal.Create 2026-02-23T10:57:15.566 INFO:tasks.workunit.client.0.trial198.stdout:[ OK ] TestClsJournal.Create (161 ms) 2026-02-23T10:57:15.566 INFO:tasks.workunit.client.0.trial198.stdout:[ RUN ] TestClsJournal.MinimumSet 2026-02-23T10:57:15.587 INFO:tasks.workunit.client.0.trial198.stdout:[ OK ] TestClsJournal.MinimumSet (21 ms) 2026-02-23T10:57:15.588 INFO:tasks.workunit.client.0.trial198.stdout:[ RUN ] TestClsJournal.MinimumSetStale 2026-02-23T10:57:15.615 INFO:tasks.workunit.client.0.trial198.stdout:[ OK ] TestClsJournal.MinimumSetStale (28 ms) 2026-02-23T10:57:15.615 INFO:tasks.workunit.client.0.trial198.stdout:[ RUN ] TestClsJournal.MinimumSetOrderConstraint 2026-02-23T10:57:15.629 INFO:tasks.workunit.client.0.trial198.stdout:[ OK ] TestClsJournal.MinimumSetOrderConstraint (13 ms) 2026-02-23T10:57:15.629 INFO:tasks.workunit.client.0.trial198.stdout:[ RUN ] TestClsJournal.ActiveSet 2026-02-23T10:57:15.643 INFO:tasks.workunit.client.0.trial198.stdout:[ OK ] TestClsJournal.ActiveSet (14 ms) 2026-02-23T10:57:15.643 INFO:tasks.workunit.client.0.trial198.stdout:[ RUN ] TestClsJournal.ActiveSetStale 2026-02-23T10:57:15.662 INFO:tasks.workunit.client.0.trial198.stdout:[ OK ] TestClsJournal.ActiveSetStale (18 ms) 2026-02-23T10:57:15.662 INFO:tasks.workunit.client.0.trial198.stdout:[ RUN ] TestClsJournal.CreateDuplicate 2026-02-23T10:57:15.673 INFO:tasks.workunit.client.0.trial198.stdout:[ OK ] TestClsJournal.CreateDuplicate (10 ms) 2026-02-23T10:57:15.673 INFO:tasks.workunit.client.0.trial198.stdout:[ RUN ] TestClsJournal.GetClient 2026-02-23T10:57:15.692 INFO:tasks.workunit.client.0.trial198.stdout:[ OK ] TestClsJournal.GetClient (19 ms) 2026-02-23T10:57:15.692 INFO:tasks.workunit.client.0.trial198.stdout:[ RUN ] TestClsJournal.ClientRegister 2026-02-23T10:57:15.705 INFO:tasks.workunit.client.0.trial198.stdout:[ OK ] TestClsJournal.ClientRegister (13 ms) 2026-02-23T10:57:15.705 INFO:tasks.workunit.client.0.trial198.stdout:[ RUN ] TestClsJournal.ClientRegisterDuplicate 2026-02-23T10:57:15.721 INFO:tasks.workunit.client.0.trial198.stdout:[ OK ] TestClsJournal.ClientRegisterDuplicate (15 ms) 2026-02-23T10:57:15.721 INFO:tasks.workunit.client.0.trial198.stdout:[ RUN ] TestClsJournal.ClientUpdateData 2026-02-23T10:57:15.743 INFO:tasks.workunit.client.0.trial198.stdout:[ OK ] TestClsJournal.ClientUpdateData (21 ms) 2026-02-23T10:57:15.743 INFO:tasks.workunit.client.0.trial198.stdout:[ RUN ] TestClsJournal.ClientUpdateState 2026-02-23T10:57:15.765 INFO:tasks.workunit.client.0.trial198.stdout:[ OK ] TestClsJournal.ClientUpdateState (22 ms) 2026-02-23T10:57:15.765 INFO:tasks.workunit.client.0.trial198.stdout:[ RUN ] TestClsJournal.ClientUnregister 2026-02-23T10:57:15.782 INFO:tasks.workunit.client.0.trial198.stdout:[ OK ] TestClsJournal.ClientUnregister (16 ms) 2026-02-23T10:57:15.782 INFO:tasks.workunit.client.0.trial198.stdout:[ RUN ] TestClsJournal.ClientUnregisterDNE 2026-02-23T10:57:15.803 INFO:tasks.workunit.client.0.trial198.stdout:[ OK ] TestClsJournal.ClientUnregisterDNE (21 ms) 2026-02-23T10:57:15.803 INFO:tasks.workunit.client.0.trial198.stdout:[ RUN ] TestClsJournal.ClientUnregisterPruneTags 2026-02-23T10:57:16.360 INFO:tasks.workunit.client.0.trial198.stdout:[ OK ] TestClsJournal.ClientUnregisterPruneTags (557 ms) 2026-02-23T10:57:16.361 INFO:tasks.workunit.client.0.trial198.stdout:[ RUN ] TestClsJournal.ClientCommit 2026-02-23T10:57:16.377 INFO:tasks.workunit.client.0.trial198.stdout:[ OK ] TestClsJournal.ClientCommit (16 ms) 2026-02-23T10:57:16.377 INFO:tasks.workunit.client.0.trial198.stdout:[ RUN ] TestClsJournal.ClientCommitInvalid 2026-02-23T10:57:16.394 INFO:tasks.workunit.client.0.trial198.stdout:[ OK ] TestClsJournal.ClientCommitInvalid (17 ms) 2026-02-23T10:57:16.394 INFO:tasks.workunit.client.0.trial198.stdout:[ RUN ] TestClsJournal.ClientCommitDNE 2026-02-23T10:57:16.399 INFO:tasks.workunit.client.0.trial198.stdout:[ OK ] TestClsJournal.ClientCommitDNE (5 ms) 2026-02-23T10:57:16.399 INFO:tasks.workunit.client.0.trial198.stdout:[ RUN ] TestClsJournal.ClientList 2026-02-23T10:57:16.850 INFO:tasks.workunit.client.0.trial198.stdout:[ OK ] TestClsJournal.ClientList (450 ms) 2026-02-23T10:57:16.850 INFO:tasks.workunit.client.0.trial198.stdout:[ RUN ] TestClsJournal.GetNextTagTid 2026-02-23T10:57:16.875 INFO:tasks.workunit.client.0.trial198.stdout:[ OK ] TestClsJournal.GetNextTagTid (24 ms) 2026-02-23T10:57:16.875 INFO:tasks.workunit.client.0.trial198.stdout:[ RUN ] TestClsJournal.TagCreate 2026-02-23T10:57:16.923 INFO:tasks.workunit.client.0.trial198.stdout:[ OK ] TestClsJournal.TagCreate (47 ms) 2026-02-23T10:57:16.923 INFO:tasks.workunit.client.0.trial198.stdout:[ RUN ] TestClsJournal.TagCreatePrunesTags 2026-02-23T10:57:16.967 INFO:tasks.workunit.client.0.trial198.stdout:[ OK ] TestClsJournal.TagCreatePrunesTags (43 ms) 2026-02-23T10:57:16.967 INFO:tasks.workunit.client.0.trial198.stdout:[ RUN ] TestClsJournal.TagList 2026-02-23T10:57:17.565 INFO:tasks.workunit.client.0.trial198.stdout:[ OK ] TestClsJournal.TagList (598 ms) 2026-02-23T10:57:17.565 INFO:tasks.workunit.client.0.trial198.stdout:[ RUN ] TestClsJournal.GuardAppend 2026-02-23T10:57:17.573 INFO:tasks.workunit.client.0.trial198.stdout:[ OK ] TestClsJournal.GuardAppend (8 ms) 2026-02-23T10:57:17.573 INFO:tasks.workunit.client.0.trial198.stdout:[ RUN ] TestClsJournal.GuardAppendDNE 2026-02-23T10:57:17.575 INFO:tasks.workunit.client.0.trial198.stdout:[ OK ] TestClsJournal.GuardAppendDNE (2 ms) 2026-02-23T10:57:17.576 INFO:tasks.workunit.client.0.trial198.stdout:[ RUN ] TestClsJournal.GuardAppendOverflow 2026-02-23T10:57:17.587 INFO:tasks.workunit.client.0.trial198.stdout:[ OK ] TestClsJournal.GuardAppendOverflow (11 ms) 2026-02-23T10:57:17.587 INFO:tasks.workunit.client.0.trial198.stdout:[ RUN ] TestClsJournal.Append 2026-02-23T10:57:17.623 INFO:tasks.workunit.client.0.trial198.stdout:padding is not enabled 2026-02-23T10:57:17.623 INFO:tasks.workunit.client.0.trial198.stdout:[ OK ] TestClsJournal.Append (35 ms) 2026-02-23T10:57:18.413 INFO:tasks.workunit.client.0.trial198.stdout:[----------] 27 tests from TestClsJournal (2219 ms total) 2026-02-23T10:57:18.414 INFO:tasks.workunit.client.0.trial198.stdout: 2026-02-23T10:57:18.414 INFO:tasks.workunit.client.0.trial198.stdout:[----------] Global test environment tear-down 2026-02-23T10:57:18.414 INFO:tasks.workunit.client.0.trial198.stdout:[==========] 27 tests from 1 test suite ran. (4867 ms total) 2026-02-23T10:57:18.414 INFO:tasks.workunit.client.0.trial198.stdout:[ PASSED ] 27 tests. 2026-02-23T10:57:18.417 INFO:teuthology.orchestra.run:Running command with timeout 3600 2026-02-23T10:57:18.417 DEBUG:teuthology.orchestra.run.trial198:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2026-02-23T10:57:18.484 INFO:tasks.workunit:Stopping ['cls/test_cls_rbd.sh', 'cls/test_cls_lock.sh', 'cls/test_cls_journal.sh'] on client.0... 2026-02-23T10:57:18.484 DEBUG:teuthology.orchestra.run.trial198:> sudo rm -rf -- /home/ubuntu/cephtest/workunits.list.client.0 /home/ubuntu/cephtest/clone.client.0 2026-02-23T10:57:18.874 DEBUG:teuthology.parallel:result is None 2026-02-23T10:57:18.875 DEBUG:teuthology.orchestra.run.trial198:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0 2026-02-23T10:57:18.900 INFO:tasks.workunit:Deleted dir /home/ubuntu/cephtest/mnt.0/client.0 2026-02-23T10:57:18.901 DEBUG:teuthology.orchestra.run.trial198:> rmdir -- /home/ubuntu/cephtest/mnt.0 2026-02-23T10:57:18.957 INFO:tasks.workunit:Deleted artificial mount point /home/ubuntu/cephtest/mnt.0/client.0 2026-02-23T10:57:18.958 DEBUG:teuthology.run_tasks:Unwinding manager ceph 2026-02-23T10:57:18.972 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2026-02-23T10:57:18.972 DEBUG:teuthology.orchestra.run.trial198:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-23T10:57:19.074 INFO:teuthology.orchestra.run.trial198.stderr:2026-02-23T10:57:19.070+0000 7f6a14d0b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T10:57:19.077 INFO:teuthology.orchestra.run.trial198.stderr:2026-02-23T10:57:19.075+0000 7f6a14d0b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T10:57:19.191 INFO:teuthology.orchestra.run.trial198.stdout: 2026-02-23T10:57:19.191 INFO:teuthology.orchestra.run.trial198.stderr:dumped all 2026-02-23T10:57:19.201 INFO:teuthology.orchestra.run.trial198.stdout:{"pg_ready":true,"pg_map":{"version":125,"stamp":"2026-02-23T10:57:18.476601+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":53,"num_read_kb":46,"num_write":81,"num_write_kb":1108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":55,"ondisk_log_size":55,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":3,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":283115520,"kb_used":82052,"kb_used_data":1748,"kb_used_omap":6831,"kb_used_meta":73424,"kb_avail":283033468,"statfs":{"total":289910292480,"available":289826271232,"internally_reserved":0,"allocated":1789952,"data_stored":1752429,"data_compressed":61150,"data_compressed_allocated":81920,"data_compressed_original":1146880,"omap_allocated":6995522,"internal_metadata":75186622},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.079778"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":147,"reported_epoch":61,"state":"active+clean","last_fresh":"2026-02-23T10:57:13.427648+0000","last_change":"2026-02-23T10:55:11.822604+0000","last_active":"2026-02-23T10:57:13.427648+0000","last_peered":"2026-02-23T10:57:13.427648+0000","last_clean":"2026-02-23T10:57:13.427648+0000","last_became_active":"2026-02-23T10:55:09.839991+0000","last_became_peered":"2026-02-23T10:55:09.839991+0000","last_unstale":"2026-02-23T10:57:13.427648+0000","last_undegraded":"2026-02-23T10:57:13.427648+0000","last_fullsized":"2026-02-23T10:57:13.427648+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-23T10:55:08.799034+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-23T10:55:08.799034+0000","last_clean_scrub_stamp":"2026-02-23T10:55:08.799034+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":147,"reported_epoch":61,"state":"active+clean","last_fresh":"2026-02-23T10:57:13.422054+0000","last_change":"2026-02-23T10:55:11.819878+0000","last_active":"2026-02-23T10:57:13.422054+0000","last_peered":"2026-02-23T10:57:13.422054+0000","last_clean":"2026-02-23T10:57:13.422054+0000","last_became_active":"2026-02-23T10:55:09.841672+0000","last_became_peered":"2026-02-23T10:55:09.841672+0000","last_unstale":"2026-02-23T10:57:13.422054+0000","last_undegraded":"2026-02-23T10:57:13.422054+0000","last_fullsized":"2026-02-23T10:57:13.422054+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-23T10:55:08.799034+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-23T10:55:08.799034+0000","last_clean_scrub_stamp":"2026-02-23T10:55:08.799034+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":147,"reported_epoch":61,"state":"active+clean","last_fresh":"2026-02-23T10:57:13.418635+0000","last_change":"2026-02-23T10:55:11.822511+0000","last_active":"2026-02-23T10:57:13.418635+0000","last_peered":"2026-02-23T10:57:13.418635+0000","last_clean":"2026-02-23T10:57:13.418635+0000","last_became_active":"2026-02-23T10:55:09.834419+0000","last_became_peered":"2026-02-23T10:55:09.834419+0000","last_unstale":"2026-02-23T10:57:13.418635+0000","last_undegraded":"2026-02-23T10:57:13.418635+0000","last_fullsized":"2026-02-23T10:57:13.418635+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-23T10:55:08.799034+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-23T10:55:08.799034+0000","last_clean_scrub_stamp":"2026-02-23T10:55:08.799034+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":147,"reported_epoch":61,"state":"active+clean","last_fresh":"2026-02-23T10:57:13.417830+0000","last_change":"2026-02-23T10:55:11.819874+0000","last_active":"2026-02-23T10:57:13.417830+0000","last_peered":"2026-02-23T10:57:13.417830+0000","last_clean":"2026-02-23T10:57:13.417830+0000","last_became_active":"2026-02-23T10:55:09.846966+0000","last_became_peered":"2026-02-23T10:55:09.846966+0000","last_unstale":"2026-02-23T10:57:13.417830+0000","last_undegraded":"2026-02-23T10:57:13.417830+0000","last_fullsized":"2026-02-23T10:57:13.417830+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-23T10:55:08.799034+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-23T10:55:08.799034+0000","last_clean_scrub_stamp":"2026-02-23T10:55:08.799034+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"17'2","reported_seq":158,"reported_epoch":64,"state":"active+clean","last_fresh":"2026-02-23T10:57:16.417483+0000","last_change":"2026-02-23T10:55:11.817943+0000","last_active":"2026-02-23T10:57:16.417483+0000","last_peered":"2026-02-23T10:57:16.417483+0000","last_clean":"2026-02-23T10:57:16.417483+0000","last_became_active":"2026-02-23T10:55:09.839193+0000","last_became_peered":"2026-02-23T10:55:09.839193+0000","last_unstale":"2026-02-23T10:57:16.417483+0000","last_undegraded":"2026-02-23T10:57:16.417483+0000","last_fullsized":"2026-02-23T10:57:16.417483+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-23T10:55:08.799034+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-23T10:55:08.799034+0000","last_clean_scrub_stamp":"2026-02-23T10:55:08.799034+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":147,"reported_epoch":61,"state":"active+clean","last_fresh":"2026-02-23T10:57:13.412172+0000","last_change":"2026-02-23T10:55:11.817133+0000","last_active":"2026-02-23T10:57:13.412172+0000","last_peered":"2026-02-23T10:57:13.412172+0000","last_clean":"2026-02-23T10:57:13.412172+0000","last_became_active":"2026-02-23T10:55:09.839412+0000","last_became_peered":"2026-02-23T10:55:09.839412+0000","last_unstale":"2026-02-23T10:57:13.412172+0000","last_undegraded":"2026-02-23T10:57:13.412172+0000","last_fullsized":"2026-02-23T10:57:13.412172+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-23T10:55:08.799034+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-23T10:55:08.799034+0000","last_clean_scrub_stamp":"2026-02-23T10:55:08.799034+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":147,"reported_epoch":61,"state":"active+clean","last_fresh":"2026-02-23T10:57:13.415182+0000","last_change":"2026-02-23T10:55:11.816017+0000","last_active":"2026-02-23T10:57:13.415182+0000","last_peered":"2026-02-23T10:57:13.415182+0000","last_clean":"2026-02-23T10:57:13.415182+0000","last_became_active":"2026-02-23T10:55:09.834010+0000","last_became_peered":"2026-02-23T10:55:09.834010+0000","last_unstale":"2026-02-23T10:57:13.415182+0000","last_undegraded":"2026-02-23T10:57:13.415182+0000","last_fullsized":"2026-02-23T10:57:13.415182+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-23T10:55:08.799034+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-23T10:55:08.799034+0000","last_clean_scrub_stamp":"2026-02-23T10:55:08.799034+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"15'1","reported_seq":148,"reported_epoch":61,"state":"active+clean","last_fresh":"2026-02-23T10:57:13.415511+0000","last_change":"2026-02-23T10:55:11.817718+0000","last_active":"2026-02-23T10:57:13.415511+0000","last_peered":"2026-02-23T10:57:13.415511+0000","last_clean":"2026-02-23T10:57:13.415511+0000","last_became_active":"2026-02-23T10:55:09.824334+0000","last_became_peered":"2026-02-23T10:55:09.824334+0000","last_unstale":"2026-02-23T10:57:13.415511+0000","last_undegraded":"2026-02-23T10:57:13.415511+0000","last_fullsized":"2026-02-23T10:57:13.415511+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-23T10:55:08.799034+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-23T10:55:08.799034+0000","last_clean_scrub_stamp":"2026-02-23T10:55:08.799034+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"12'52","reported_seq":222,"reported_epoch":61,"state":"active+clean","last_fresh":"2026-02-23T10:57:13.421240+0000","last_change":"2026-02-23T10:55:05.812642+0000","last_active":"2026-02-23T10:57:13.421240+0000","last_peered":"2026-02-23T10:57:13.421240+0000","last_clean":"2026-02-23T10:57:13.421240+0000","last_became_active":"2026-02-23T10:55:05.809738+0000","last_became_peered":"2026-02-23T10:55:05.809738+0000","last_unstale":"2026-02-23T10:57:13.421240+0000","last_undegraded":"2026-02-23T10:57:13.421240+0000","last_fullsized":"2026-02-23T10:57:13.421240+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-23T10:55:04.788039+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-23T10:55:04.788039+0000","last_clean_scrub_stamp":"2026-02-23T10:55:04.788039+0000","objects_scrubbed":0,"log_size":52,"log_dups_size":0,"ondisk_log_size":52,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":53,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":53,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":590368,"data_compressed":30575,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},"log_size":52,"ondisk_log_size":52,"up":2,"acting":2,"num_store_stats":1}],"osd_stats":[{"osd":0,"up_from":9,"seq":38654705798,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27396,"kb_used_data":628,"kb_used_omap":2050,"kb_used_meta":24701,"kb_avail":94344444,"statfs":{"total":96636764160,"available":96608710656,"internally_reserved":0,"allocated":643072,"data_stored":790138,"data_compressed":30575,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":2100148,"internal_metadata":25293900},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":10,"seq":42949673089,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27296,"kb_used_data":528,"kb_used_omap":973,"kb_used_meta":25778,"kb_avail":94344544,"statfs":{"total":96636764160,"available":96608813056,"internally_reserved":0,"allocated":540672,"data_stored":185952,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":996622,"internal_metadata":26397426},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":10,"seq":42949673089,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27360,"kb_used_data":592,"kb_used_omap":3807,"kb_used_meta":22944,"kb_avail":94344480,"statfs":{"total":96636764160,"available":96608747520,"internally_reserved":0,"allocated":606208,"data_stored":776339,"data_compressed":30575,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":3898752,"internal_metadata":23495296},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":590368,"data_compressed":30575,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-23T10:57:19.203 DEBUG:teuthology.orchestra.run.trial198:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-23T10:57:19.281 INFO:teuthology.orchestra.run.trial198.stderr:2026-02-23T10:57:19.276+0000 7f619670b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T10:57:19.283 INFO:teuthology.orchestra.run.trial198.stderr:2026-02-23T10:57:19.281+0000 7f619670b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T10:57:19.396 INFO:teuthology.orchestra.run.trial198.stdout: 2026-02-23T10:57:19.396 INFO:teuthology.orchestra.run.trial198.stderr:dumped all 2026-02-23T10:57:19.406 INFO:teuthology.orchestra.run.trial198.stdout:{"pg_ready":true,"pg_map":{"version":125,"stamp":"2026-02-23T10:57:18.476601+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":53,"num_read_kb":46,"num_write":81,"num_write_kb":1108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":55,"ondisk_log_size":55,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":3,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":283115520,"kb_used":82052,"kb_used_data":1748,"kb_used_omap":6831,"kb_used_meta":73424,"kb_avail":283033468,"statfs":{"total":289910292480,"available":289826271232,"internally_reserved":0,"allocated":1789952,"data_stored":1752429,"data_compressed":61150,"data_compressed_allocated":81920,"data_compressed_original":1146880,"omap_allocated":6995522,"internal_metadata":75186622},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.079778"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":147,"reported_epoch":61,"state":"active+clean","last_fresh":"2026-02-23T10:57:13.427648+0000","last_change":"2026-02-23T10:55:11.822604+0000","last_active":"2026-02-23T10:57:13.427648+0000","last_peered":"2026-02-23T10:57:13.427648+0000","last_clean":"2026-02-23T10:57:13.427648+0000","last_became_active":"2026-02-23T10:55:09.839991+0000","last_became_peered":"2026-02-23T10:55:09.839991+0000","last_unstale":"2026-02-23T10:57:13.427648+0000","last_undegraded":"2026-02-23T10:57:13.427648+0000","last_fullsized":"2026-02-23T10:57:13.427648+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-23T10:55:08.799034+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-23T10:55:08.799034+0000","last_clean_scrub_stamp":"2026-02-23T10:55:08.799034+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":147,"reported_epoch":61,"state":"active+clean","last_fresh":"2026-02-23T10:57:13.422054+0000","last_change":"2026-02-23T10:55:11.819878+0000","last_active":"2026-02-23T10:57:13.422054+0000","last_peered":"2026-02-23T10:57:13.422054+0000","last_clean":"2026-02-23T10:57:13.422054+0000","last_became_active":"2026-02-23T10:55:09.841672+0000","last_became_peered":"2026-02-23T10:55:09.841672+0000","last_unstale":"2026-02-23T10:57:13.422054+0000","last_undegraded":"2026-02-23T10:57:13.422054+0000","last_fullsized":"2026-02-23T10:57:13.422054+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-23T10:55:08.799034+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-23T10:55:08.799034+0000","last_clean_scrub_stamp":"2026-02-23T10:55:08.799034+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":147,"reported_epoch":61,"state":"active+clean","last_fresh":"2026-02-23T10:57:13.418635+0000","last_change":"2026-02-23T10:55:11.822511+0000","last_active":"2026-02-23T10:57:13.418635+0000","last_peered":"2026-02-23T10:57:13.418635+0000","last_clean":"2026-02-23T10:57:13.418635+0000","last_became_active":"2026-02-23T10:55:09.834419+0000","last_became_peered":"2026-02-23T10:55:09.834419+0000","last_unstale":"2026-02-23T10:57:13.418635+0000","last_undegraded":"2026-02-23T10:57:13.418635+0000","last_fullsized":"2026-02-23T10:57:13.418635+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-23T10:55:08.799034+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-23T10:55:08.799034+0000","last_clean_scrub_stamp":"2026-02-23T10:55:08.799034+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":147,"reported_epoch":61,"state":"active+clean","last_fresh":"2026-02-23T10:57:13.417830+0000","last_change":"2026-02-23T10:55:11.819874+0000","last_active":"2026-02-23T10:57:13.417830+0000","last_peered":"2026-02-23T10:57:13.417830+0000","last_clean":"2026-02-23T10:57:13.417830+0000","last_became_active":"2026-02-23T10:55:09.846966+0000","last_became_peered":"2026-02-23T10:55:09.846966+0000","last_unstale":"2026-02-23T10:57:13.417830+0000","last_undegraded":"2026-02-23T10:57:13.417830+0000","last_fullsized":"2026-02-23T10:57:13.417830+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-23T10:55:08.799034+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-23T10:55:08.799034+0000","last_clean_scrub_stamp":"2026-02-23T10:55:08.799034+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"17'2","reported_seq":158,"reported_epoch":64,"state":"active+clean","last_fresh":"2026-02-23T10:57:16.417483+0000","last_change":"2026-02-23T10:55:11.817943+0000","last_active":"2026-02-23T10:57:16.417483+0000","last_peered":"2026-02-23T10:57:16.417483+0000","last_clean":"2026-02-23T10:57:16.417483+0000","last_became_active":"2026-02-23T10:55:09.839193+0000","last_became_peered":"2026-02-23T10:55:09.839193+0000","last_unstale":"2026-02-23T10:57:16.417483+0000","last_undegraded":"2026-02-23T10:57:16.417483+0000","last_fullsized":"2026-02-23T10:57:16.417483+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-23T10:55:08.799034+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-23T10:55:08.799034+0000","last_clean_scrub_stamp":"2026-02-23T10:55:08.799034+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":147,"reported_epoch":61,"state":"active+clean","last_fresh":"2026-02-23T10:57:13.412172+0000","last_change":"2026-02-23T10:55:11.817133+0000","last_active":"2026-02-23T10:57:13.412172+0000","last_peered":"2026-02-23T10:57:13.412172+0000","last_clean":"2026-02-23T10:57:13.412172+0000","last_became_active":"2026-02-23T10:55:09.839412+0000","last_became_peered":"2026-02-23T10:55:09.839412+0000","last_unstale":"2026-02-23T10:57:13.412172+0000","last_undegraded":"2026-02-23T10:57:13.412172+0000","last_fullsized":"2026-02-23T10:57:13.412172+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-23T10:55:08.799034+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-23T10:55:08.799034+0000","last_clean_scrub_stamp":"2026-02-23T10:55:08.799034+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":147,"reported_epoch":61,"state":"active+clean","last_fresh":"2026-02-23T10:57:13.415182+0000","last_change":"2026-02-23T10:55:11.816017+0000","last_active":"2026-02-23T10:57:13.415182+0000","last_peered":"2026-02-23T10:57:13.415182+0000","last_clean":"2026-02-23T10:57:13.415182+0000","last_became_active":"2026-02-23T10:55:09.834010+0000","last_became_peered":"2026-02-23T10:55:09.834010+0000","last_unstale":"2026-02-23T10:57:13.415182+0000","last_undegraded":"2026-02-23T10:57:13.415182+0000","last_fullsized":"2026-02-23T10:57:13.415182+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-23T10:55:08.799034+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-23T10:55:08.799034+0000","last_clean_scrub_stamp":"2026-02-23T10:55:08.799034+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"15'1","reported_seq":148,"reported_epoch":61,"state":"active+clean","last_fresh":"2026-02-23T10:57:13.415511+0000","last_change":"2026-02-23T10:55:11.817718+0000","last_active":"2026-02-23T10:57:13.415511+0000","last_peered":"2026-02-23T10:57:13.415511+0000","last_clean":"2026-02-23T10:57:13.415511+0000","last_became_active":"2026-02-23T10:55:09.824334+0000","last_became_peered":"2026-02-23T10:55:09.824334+0000","last_unstale":"2026-02-23T10:57:13.415511+0000","last_undegraded":"2026-02-23T10:57:13.415511+0000","last_fullsized":"2026-02-23T10:57:13.415511+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-23T10:55:08.799034+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-23T10:55:08.799034+0000","last_clean_scrub_stamp":"2026-02-23T10:55:08.799034+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"12'52","reported_seq":222,"reported_epoch":61,"state":"active+clean","last_fresh":"2026-02-23T10:57:13.421240+0000","last_change":"2026-02-23T10:55:05.812642+0000","last_active":"2026-02-23T10:57:13.421240+0000","last_peered":"2026-02-23T10:57:13.421240+0000","last_clean":"2026-02-23T10:57:13.421240+0000","last_became_active":"2026-02-23T10:55:05.809738+0000","last_became_peered":"2026-02-23T10:55:05.809738+0000","last_unstale":"2026-02-23T10:57:13.421240+0000","last_undegraded":"2026-02-23T10:57:13.421240+0000","last_fullsized":"2026-02-23T10:57:13.421240+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-23T10:55:04.788039+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-23T10:55:04.788039+0000","last_clean_scrub_stamp":"2026-02-23T10:55:04.788039+0000","objects_scrubbed":0,"log_size":52,"log_dups_size":0,"ondisk_log_size":52,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":53,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":53,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":590368,"data_compressed":30575,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},"log_size":52,"ondisk_log_size":52,"up":2,"acting":2,"num_store_stats":1}],"osd_stats":[{"osd":0,"up_from":9,"seq":38654705798,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27396,"kb_used_data":628,"kb_used_omap":2050,"kb_used_meta":24701,"kb_avail":94344444,"statfs":{"total":96636764160,"available":96608710656,"internally_reserved":0,"allocated":643072,"data_stored":790138,"data_compressed":30575,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":2100148,"internal_metadata":25293900},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":10,"seq":42949673089,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27296,"kb_used_data":528,"kb_used_omap":973,"kb_used_meta":25778,"kb_avail":94344544,"statfs":{"total":96636764160,"available":96608813056,"internally_reserved":0,"allocated":540672,"data_stored":185952,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":996622,"internal_metadata":26397426},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":10,"seq":42949673089,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27360,"kb_used_data":592,"kb_used_omap":3807,"kb_used_meta":22944,"kb_avail":94344480,"statfs":{"total":96636764160,"available":96608747520,"internally_reserved":0,"allocated":606208,"data_stored":776339,"data_compressed":30575,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":3898752,"internal_metadata":23495296},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":590368,"data_compressed":30575,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-23T10:57:19.407 INFO:tasks.ceph.ceph_manager.ceph:clean! 2026-02-23T10:57:19.407 DEBUG:teuthology.orchestra.run.trial198:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-23T10:57:19.483 INFO:teuthology.orchestra.run.trial198.stderr:2026-02-23T10:57:19.479+0000 7fab52e1a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T10:57:19.486 INFO:teuthology.orchestra.run.trial198.stderr:2026-02-23T10:57:19.484+0000 7fab52e1a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T10:57:19.601 INFO:teuthology.orchestra.run.trial198.stdout: 2026-02-23T10:57:19.601 INFO:teuthology.orchestra.run.trial198.stderr:dumped all 2026-02-23T10:57:19.610 INFO:teuthology.orchestra.run.trial198.stdout:{"pg_ready":true,"pg_map":{"version":125,"stamp":"2026-02-23T10:57:18.476601+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":53,"num_read_kb":46,"num_write":81,"num_write_kb":1108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":55,"ondisk_log_size":55,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":3,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":283115520,"kb_used":82052,"kb_used_data":1748,"kb_used_omap":6831,"kb_used_meta":73424,"kb_avail":283033468,"statfs":{"total":289910292480,"available":289826271232,"internally_reserved":0,"allocated":1789952,"data_stored":1752429,"data_compressed":61150,"data_compressed_allocated":81920,"data_compressed_original":1146880,"omap_allocated":6995522,"internal_metadata":75186622},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.079778"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":147,"reported_epoch":61,"state":"active+clean","last_fresh":"2026-02-23T10:57:13.427648+0000","last_change":"2026-02-23T10:55:11.822604+0000","last_active":"2026-02-23T10:57:13.427648+0000","last_peered":"2026-02-23T10:57:13.427648+0000","last_clean":"2026-02-23T10:57:13.427648+0000","last_became_active":"2026-02-23T10:55:09.839991+0000","last_became_peered":"2026-02-23T10:55:09.839991+0000","last_unstale":"2026-02-23T10:57:13.427648+0000","last_undegraded":"2026-02-23T10:57:13.427648+0000","last_fullsized":"2026-02-23T10:57:13.427648+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-23T10:55:08.799034+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-23T10:55:08.799034+0000","last_clean_scrub_stamp":"2026-02-23T10:55:08.799034+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":147,"reported_epoch":61,"state":"active+clean","last_fresh":"2026-02-23T10:57:13.422054+0000","last_change":"2026-02-23T10:55:11.819878+0000","last_active":"2026-02-23T10:57:13.422054+0000","last_peered":"2026-02-23T10:57:13.422054+0000","last_clean":"2026-02-23T10:57:13.422054+0000","last_became_active":"2026-02-23T10:55:09.841672+0000","last_became_peered":"2026-02-23T10:55:09.841672+0000","last_unstale":"2026-02-23T10:57:13.422054+0000","last_undegraded":"2026-02-23T10:57:13.422054+0000","last_fullsized":"2026-02-23T10:57:13.422054+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-23T10:55:08.799034+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-23T10:55:08.799034+0000","last_clean_scrub_stamp":"2026-02-23T10:55:08.799034+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":147,"reported_epoch":61,"state":"active+clean","last_fresh":"2026-02-23T10:57:13.418635+0000","last_change":"2026-02-23T10:55:11.822511+0000","last_active":"2026-02-23T10:57:13.418635+0000","last_peered":"2026-02-23T10:57:13.418635+0000","last_clean":"2026-02-23T10:57:13.418635+0000","last_became_active":"2026-02-23T10:55:09.834419+0000","last_became_peered":"2026-02-23T10:55:09.834419+0000","last_unstale":"2026-02-23T10:57:13.418635+0000","last_undegraded":"2026-02-23T10:57:13.418635+0000","last_fullsized":"2026-02-23T10:57:13.418635+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-23T10:55:08.799034+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-23T10:55:08.799034+0000","last_clean_scrub_stamp":"2026-02-23T10:55:08.799034+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":147,"reported_epoch":61,"state":"active+clean","last_fresh":"2026-02-23T10:57:13.417830+0000","last_change":"2026-02-23T10:55:11.819874+0000","last_active":"2026-02-23T10:57:13.417830+0000","last_peered":"2026-02-23T10:57:13.417830+0000","last_clean":"2026-02-23T10:57:13.417830+0000","last_became_active":"2026-02-23T10:55:09.846966+0000","last_became_peered":"2026-02-23T10:55:09.846966+0000","last_unstale":"2026-02-23T10:57:13.417830+0000","last_undegraded":"2026-02-23T10:57:13.417830+0000","last_fullsized":"2026-02-23T10:57:13.417830+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-23T10:55:08.799034+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-23T10:55:08.799034+0000","last_clean_scrub_stamp":"2026-02-23T10:55:08.799034+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"17'2","reported_seq":158,"reported_epoch":64,"state":"active+clean","last_fresh":"2026-02-23T10:57:16.417483+0000","last_change":"2026-02-23T10:55:11.817943+0000","last_active":"2026-02-23T10:57:16.417483+0000","last_peered":"2026-02-23T10:57:16.417483+0000","last_clean":"2026-02-23T10:57:16.417483+0000","last_became_active":"2026-02-23T10:55:09.839193+0000","last_became_peered":"2026-02-23T10:55:09.839193+0000","last_unstale":"2026-02-23T10:57:16.417483+0000","last_undegraded":"2026-02-23T10:57:16.417483+0000","last_fullsized":"2026-02-23T10:57:16.417483+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-23T10:55:08.799034+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-23T10:55:08.799034+0000","last_clean_scrub_stamp":"2026-02-23T10:55:08.799034+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":147,"reported_epoch":61,"state":"active+clean","last_fresh":"2026-02-23T10:57:13.412172+0000","last_change":"2026-02-23T10:55:11.817133+0000","last_active":"2026-02-23T10:57:13.412172+0000","last_peered":"2026-02-23T10:57:13.412172+0000","last_clean":"2026-02-23T10:57:13.412172+0000","last_became_active":"2026-02-23T10:55:09.839412+0000","last_became_peered":"2026-02-23T10:55:09.839412+0000","last_unstale":"2026-02-23T10:57:13.412172+0000","last_undegraded":"2026-02-23T10:57:13.412172+0000","last_fullsized":"2026-02-23T10:57:13.412172+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-23T10:55:08.799034+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-23T10:55:08.799034+0000","last_clean_scrub_stamp":"2026-02-23T10:55:08.799034+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":147,"reported_epoch":61,"state":"active+clean","last_fresh":"2026-02-23T10:57:13.415182+0000","last_change":"2026-02-23T10:55:11.816017+0000","last_active":"2026-02-23T10:57:13.415182+0000","last_peered":"2026-02-23T10:57:13.415182+0000","last_clean":"2026-02-23T10:57:13.415182+0000","last_became_active":"2026-02-23T10:55:09.834010+0000","last_became_peered":"2026-02-23T10:55:09.834010+0000","last_unstale":"2026-02-23T10:57:13.415182+0000","last_undegraded":"2026-02-23T10:57:13.415182+0000","last_fullsized":"2026-02-23T10:57:13.415182+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-23T10:55:08.799034+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-23T10:55:08.799034+0000","last_clean_scrub_stamp":"2026-02-23T10:55:08.799034+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"15'1","reported_seq":148,"reported_epoch":61,"state":"active+clean","last_fresh":"2026-02-23T10:57:13.415511+0000","last_change":"2026-02-23T10:55:11.817718+0000","last_active":"2026-02-23T10:57:13.415511+0000","last_peered":"2026-02-23T10:57:13.415511+0000","last_clean":"2026-02-23T10:57:13.415511+0000","last_became_active":"2026-02-23T10:55:09.824334+0000","last_became_peered":"2026-02-23T10:55:09.824334+0000","last_unstale":"2026-02-23T10:57:13.415511+0000","last_undegraded":"2026-02-23T10:57:13.415511+0000","last_fullsized":"2026-02-23T10:57:13.415511+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-23T10:55:08.799034+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-23T10:55:08.799034+0000","last_clean_scrub_stamp":"2026-02-23T10:55:08.799034+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"12'52","reported_seq":222,"reported_epoch":61,"state":"active+clean","last_fresh":"2026-02-23T10:57:13.421240+0000","last_change":"2026-02-23T10:55:05.812642+0000","last_active":"2026-02-23T10:57:13.421240+0000","last_peered":"2026-02-23T10:57:13.421240+0000","last_clean":"2026-02-23T10:57:13.421240+0000","last_became_active":"2026-02-23T10:55:05.809738+0000","last_became_peered":"2026-02-23T10:55:05.809738+0000","last_unstale":"2026-02-23T10:57:13.421240+0000","last_undegraded":"2026-02-23T10:57:13.421240+0000","last_fullsized":"2026-02-23T10:57:13.421240+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-23T10:55:04.788039+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-23T10:55:04.788039+0000","last_clean_scrub_stamp":"2026-02-23T10:55:04.788039+0000","objects_scrubbed":0,"log_size":52,"log_dups_size":0,"ondisk_log_size":52,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":53,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":53,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":590368,"data_compressed":30575,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},"log_size":52,"ondisk_log_size":52,"up":2,"acting":2,"num_store_stats":1}],"osd_stats":[{"osd":0,"up_from":9,"seq":38654705798,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27396,"kb_used_data":628,"kb_used_omap":2050,"kb_used_meta":24701,"kb_avail":94344444,"statfs":{"total":96636764160,"available":96608710656,"internally_reserved":0,"allocated":643072,"data_stored":790138,"data_compressed":30575,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":2100148,"internal_metadata":25293900},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":10,"seq":42949673089,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27296,"kb_used_data":528,"kb_used_omap":973,"kb_used_meta":25778,"kb_avail":94344544,"statfs":{"total":96636764160,"available":96608813056,"internally_reserved":0,"allocated":540672,"data_stored":185952,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":996622,"internal_metadata":26397426},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":10,"seq":42949673089,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27360,"kb_used_data":592,"kb_used_omap":3807,"kb_used_meta":22944,"kb_avail":94344480,"statfs":{"total":96636764160,"available":96608747520,"internally_reserved":0,"allocated":606208,"data_stored":776339,"data_compressed":30575,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":3898752,"internal_metadata":23495296},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":590368,"data_compressed":30575,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-23T10:57:19.612 DEBUG:teuthology.orchestra.run.trial198:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2026-02-23T10:57:19.689 INFO:teuthology.orchestra.run.trial198.stderr:2026-02-23T10:57:19.685+0000 7f0d69e5a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T10:57:19.691 INFO:teuthology.orchestra.run.trial198.stderr:2026-02-23T10:57:19.689+0000 7f0d69e5a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T10:57:19.803 INFO:teuthology.orchestra.run.trial198.stdout: 2026-02-23T10:57:19.803 INFO:teuthology.orchestra.run.trial198.stdout:{"epoch":65,"fsid":"8ebafdf4-9523-4bab-a465-a90635e02d85","created":"2026-02-23T10:54:52.211292+0000","modified":"2026-02-23T10:57:18.406099+0000","last_up_change":"2026-02-23T10:55:03.782459+0000","last_in_change":"2026-02-23T10:54:53.980646+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":15,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T10:55:04.470198+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"13","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.9900000095367432,"score_stable":2.9900000095367432,"optimal_score":0.67000001668930054,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2026-02-23T10:55:07.934221+0000","flags":270337,"flags_names":"hashpspool,selfmanaged_snaps,crimson","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":2,"snap_epoch":17,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"4acd148c-ad73-48ca-abb1-37def718455b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":62,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.198:6803","nonce":1261899012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.198:6802","nonce":1261899012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.198:6807","nonce":1261899012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.198:6804","nonce":1261899012}]},"public_addr":"10.20.193.198:6803/1261899012","cluster_addr":"10.20.193.198:6802/1261899012","heartbeat_back_addr":"10.20.193.198:6807/1261899012","heartbeat_front_addr":"10.20.193.198:6804/1261899012","state":["exists","up"]},{"osd":1,"uuid":"f6b3e06e-2b8b-4194-ba73-c0edaaff09dc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":10,"up_thru":62,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.198:6805","nonce":3805786617}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.198:6806","nonce":3805786617}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.198:6811","nonce":3805786617}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.198:6808","nonce":3805786617}]},"public_addr":"10.20.193.198:6805/3805786617","cluster_addr":"10.20.193.198:6806/3805786617","heartbeat_back_addr":"10.20.193.198:6811/3805786617","heartbeat_front_addr":"10.20.193.198:6808/3805786617","state":["exists","up"]},{"osd":2,"uuid":"d28833e3-00d5-418c-8ba8-81a1c18e33e2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":10,"up_thru":62,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.198:6809","nonce":677970331}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.198:6810","nonce":677970331}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.198:6813","nonce":677970331}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.198:6812","nonce":677970331}]},"public_addr":"10.20.193.198:6809/677970331","cluster_addr":"10.20.193.198:6810/677970331","heartbeat_back_addr":"10.20.193.198:6813/677970331","heartbeat_front_addr":"10.20.193.198:6812/677970331","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":2,"snaps":[{"begin":2,"length":1}]},{"pool":3,"snaps":[{"begin":2,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T10:57:20.814 INFO:tasks.ceph:Scrubbing osd.0 2026-02-23T10:57:20.814 DEBUG:teuthology.orchestra.run.trial198:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 config set osd_debug_deep_scrub_sleep 0 2026-02-23T10:57:20.892 INFO:teuthology.orchestra.run.trial198.stderr:2026-02-23T10:57:20.886+0000 7f8c7f85a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T10:57:20.895 INFO:teuthology.orchestra.run.trial198.stderr:2026-02-23T10:57:20.893+0000 7f8c7f85a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T10:57:20.903 INFO:teuthology.orchestra.run.trial198.stdout:{ 2026-02-23T10:57:20.903 INFO:teuthology.orchestra.run.trial198.stdout: "success": "" 2026-02-23T10:57:20.903 INFO:teuthology.orchestra.run.trial198.stdout:} 2026-02-23T10:57:20.910 DEBUG:teuthology.orchestra.run.trial198:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 0 2026-02-23T10:57:20.987 INFO:teuthology.orchestra.run.trial198.stderr:2026-02-23T10:57:20.982+0000 7fe318f0b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T10:57:20.989 INFO:teuthology.orchestra.run.trial198.stderr:2026-02-23T10:57:20.987+0000 7fe318f0b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T10:57:21.103 INFO:teuthology.orchestra.run.trial198.stderr:instructed osd(s) 0 to deep-scrub 2026-02-23T10:57:21.114 INFO:tasks.ceph:Scrubbing osd.1 2026-02-23T10:57:21.114 DEBUG:teuthology.orchestra.run.trial198:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 config set osd_debug_deep_scrub_sleep 0 2026-02-23T10:57:21.193 INFO:teuthology.orchestra.run.trial198.stderr:2026-02-23T10:57:21.186+0000 7f9428783640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T10:57:21.195 INFO:teuthology.orchestra.run.trial198.stderr:2026-02-23T10:57:21.193+0000 7f9428783640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T10:57:21.204 INFO:teuthology.orchestra.run.trial198.stdout:{ 2026-02-23T10:57:21.205 INFO:teuthology.orchestra.run.trial198.stdout: "success": "" 2026-02-23T10:57:21.205 INFO:teuthology.orchestra.run.trial198.stdout:} 2026-02-23T10:57:21.213 DEBUG:teuthology.orchestra.run.trial198:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 1 2026-02-23T10:57:21.291 INFO:teuthology.orchestra.run.trial198.stderr:2026-02-23T10:57:21.286+0000 7f6721ddc640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T10:57:21.293 INFO:teuthology.orchestra.run.trial198.stderr:2026-02-23T10:57:21.291+0000 7f6721ddc640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T10:57:21.406 INFO:teuthology.orchestra.run.trial198.stderr:instructed osd(s) 1 to deep-scrub 2026-02-23T10:57:21.419 INFO:tasks.ceph:Scrubbing osd.2 2026-02-23T10:57:21.419 DEBUG:teuthology.orchestra.run.trial198:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 config set osd_debug_deep_scrub_sleep 0 2026-02-23T10:57:21.500 INFO:teuthology.orchestra.run.trial198.stderr:2026-02-23T10:57:21.493+0000 7f7182411640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T10:57:21.502 INFO:teuthology.orchestra.run.trial198.stderr:2026-02-23T10:57:21.499+0000 7f7182411640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T10:57:21.510 INFO:teuthology.orchestra.run.trial198.stdout:{ 2026-02-23T10:57:21.511 INFO:teuthology.orchestra.run.trial198.stdout: "success": "" 2026-02-23T10:57:21.511 INFO:teuthology.orchestra.run.trial198.stdout:} 2026-02-23T10:57:21.518 DEBUG:teuthology.orchestra.run.trial198:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 2 2026-02-23T10:57:21.594 INFO:teuthology.orchestra.run.trial198.stderr:2026-02-23T10:57:21.590+0000 7f1f0dfa4640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T10:57:21.596 INFO:teuthology.orchestra.run.trial198.stderr:2026-02-23T10:57:21.594+0000 7f1f0dfa4640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T10:57:21.714 INFO:teuthology.orchestra.run.trial198.stderr:instructed osd(s) 2 to deep-scrub 2026-02-23T10:57:21.727 DEBUG:teuthology.orchestra.run.trial198:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-23T10:57:21.802 INFO:teuthology.orchestra.run.trial198.stderr:2026-02-23T10:57:21.798+0000 7f4497acb640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T10:57:21.804 INFO:teuthology.orchestra.run.trial198.stderr:2026-02-23T10:57:21.802+0000 7f4497acb640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T10:57:21.916 INFO:teuthology.orchestra.run.trial198.stdout: 2026-02-23T10:57:21.916 INFO:teuthology.orchestra.run.trial198.stderr:dumped all 2026-02-23T10:57:21.925 INFO:teuthology.orchestra.run.trial198.stdout:{"pg_ready":true,"pg_map":{"version":126,"stamp":"2026-02-23T10:57:20.476879+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":53,"num_read_kb":46,"num_write":81,"num_write_kb":1108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":55,"ondisk_log_size":55,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":3,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":283115520,"kb_used":82144,"kb_used_data":1836,"kb_used_omap":7919,"kb_used_meta":72336,"kb_avail":283033376,"statfs":{"total":289910292480,"available":289826177024,"internally_reserved":0,"allocated":1880064,"data_stored":1780083,"data_compressed":61150,"data_compressed_allocated":81920,"data_compressed_original":1146880,"omap_allocated":8109622,"internal_metadata":74072522},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.000861"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":159,"reported_epoch":65,"state":"active+clean","last_fresh":"2026-02-23T10:57:18.421000+0000","last_change":"2026-02-23T10:55:11.822604+0000","last_active":"2026-02-23T10:57:18.421000+0000","last_peered":"2026-02-23T10:57:18.421000+0000","last_clean":"2026-02-23T10:57:18.421000+0000","last_became_active":"2026-02-23T10:55:09.839991+0000","last_became_peered":"2026-02-23T10:55:09.839991+0000","last_unstale":"2026-02-23T10:57:18.421000+0000","last_undegraded":"2026-02-23T10:57:18.421000+0000","last_fullsized":"2026-02-23T10:57:18.421000+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-23T10:55:08.799034+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-23T10:55:08.799034+0000","last_clean_scrub_stamp":"2026-02-23T10:55:08.799034+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":159,"reported_epoch":65,"state":"active+clean","last_fresh":"2026-02-23T10:57:18.427483+0000","last_change":"2026-02-23T10:55:11.819878+0000","last_active":"2026-02-23T10:57:18.427483+0000","last_peered":"2026-02-23T10:57:18.427483+0000","last_clean":"2026-02-23T10:57:18.427483+0000","last_became_active":"2026-02-23T10:55:09.841672+0000","last_became_peered":"2026-02-23T10:55:09.841672+0000","last_unstale":"2026-02-23T10:57:18.427483+0000","last_undegraded":"2026-02-23T10:57:18.427483+0000","last_fullsized":"2026-02-23T10:57:18.427483+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-23T10:55:08.799034+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-23T10:55:08.799034+0000","last_clean_scrub_stamp":"2026-02-23T10:55:08.799034+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":159,"reported_epoch":65,"state":"active+clean","last_fresh":"2026-02-23T10:57:18.417737+0000","last_change":"2026-02-23T10:55:11.822511+0000","last_active":"2026-02-23T10:57:18.417737+0000","last_peered":"2026-02-23T10:57:18.417737+0000","last_clean":"2026-02-23T10:57:18.417737+0000","last_became_active":"2026-02-23T10:55:09.834419+0000","last_became_peered":"2026-02-23T10:55:09.834419+0000","last_unstale":"2026-02-23T10:57:18.417737+0000","last_undegraded":"2026-02-23T10:57:18.417737+0000","last_fullsized":"2026-02-23T10:57:18.417737+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-23T10:55:08.799034+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-23T10:55:08.799034+0000","last_clean_scrub_stamp":"2026-02-23T10:55:08.799034+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":159,"reported_epoch":65,"state":"active+clean","last_fresh":"2026-02-23T10:57:18.425672+0000","last_change":"2026-02-23T10:55:11.819874+0000","last_active":"2026-02-23T10:57:18.425672+0000","last_peered":"2026-02-23T10:57:18.425672+0000","last_clean":"2026-02-23T10:57:18.425672+0000","last_became_active":"2026-02-23T10:55:09.846966+0000","last_became_peered":"2026-02-23T10:55:09.846966+0000","last_unstale":"2026-02-23T10:57:18.425672+0000","last_undegraded":"2026-02-23T10:57:18.425672+0000","last_fullsized":"2026-02-23T10:57:18.425672+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-23T10:55:08.799034+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-23T10:55:08.799034+0000","last_clean_scrub_stamp":"2026-02-23T10:55:08.799034+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"17'2","reported_seq":158,"reported_epoch":64,"state":"active+clean","last_fresh":"2026-02-23T10:57:16.417483+0000","last_change":"2026-02-23T10:55:11.817943+0000","last_active":"2026-02-23T10:57:16.417483+0000","last_peered":"2026-02-23T10:57:16.417483+0000","last_clean":"2026-02-23T10:57:16.417483+0000","last_became_active":"2026-02-23T10:55:09.839193+0000","last_became_peered":"2026-02-23T10:55:09.839193+0000","last_unstale":"2026-02-23T10:57:16.417483+0000","last_undegraded":"2026-02-23T10:57:16.417483+0000","last_fullsized":"2026-02-23T10:57:16.417483+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-23T10:55:08.799034+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-23T10:55:08.799034+0000","last_clean_scrub_stamp":"2026-02-23T10:55:08.799034+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":159,"reported_epoch":65,"state":"active+clean","last_fresh":"2026-02-23T10:57:18.427431+0000","last_change":"2026-02-23T10:55:11.817133+0000","last_active":"2026-02-23T10:57:18.427431+0000","last_peered":"2026-02-23T10:57:18.427431+0000","last_clean":"2026-02-23T10:57:18.427431+0000","last_became_active":"2026-02-23T10:55:09.839412+0000","last_became_peered":"2026-02-23T10:55:09.839412+0000","last_unstale":"2026-02-23T10:57:18.427431+0000","last_undegraded":"2026-02-23T10:57:18.427431+0000","last_fullsized":"2026-02-23T10:57:18.427431+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-23T10:55:08.799034+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-23T10:55:08.799034+0000","last_clean_scrub_stamp":"2026-02-23T10:55:08.799034+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":159,"reported_epoch":65,"state":"active+clean","last_fresh":"2026-02-23T10:57:18.431557+0000","last_change":"2026-02-23T10:55:11.816017+0000","last_active":"2026-02-23T10:57:18.431557+0000","last_peered":"2026-02-23T10:57:18.431557+0000","last_clean":"2026-02-23T10:57:18.431557+0000","last_became_active":"2026-02-23T10:55:09.834010+0000","last_became_peered":"2026-02-23T10:55:09.834010+0000","last_unstale":"2026-02-23T10:57:18.431557+0000","last_undegraded":"2026-02-23T10:57:18.431557+0000","last_fullsized":"2026-02-23T10:57:18.431557+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-23T10:55:08.799034+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-23T10:55:08.799034+0000","last_clean_scrub_stamp":"2026-02-23T10:55:08.799034+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"15'1","reported_seq":160,"reported_epoch":65,"state":"active+clean","last_fresh":"2026-02-23T10:57:18.425825+0000","last_change":"2026-02-23T10:55:11.817718+0000","last_active":"2026-02-23T10:57:18.425825+0000","last_peered":"2026-02-23T10:57:18.425825+0000","last_clean":"2026-02-23T10:57:18.425825+0000","last_became_active":"2026-02-23T10:55:09.824334+0000","last_became_peered":"2026-02-23T10:55:09.824334+0000","last_unstale":"2026-02-23T10:57:18.425825+0000","last_undegraded":"2026-02-23T10:57:18.425825+0000","last_fullsized":"2026-02-23T10:57:18.425825+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-23T10:55:08.799034+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-23T10:55:08.799034+0000","last_clean_scrub_stamp":"2026-02-23T10:55:08.799034+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"12'52","reported_seq":234,"reported_epoch":65,"state":"active+clean","last_fresh":"2026-02-23T10:57:18.432893+0000","last_change":"2026-02-23T10:55:05.812642+0000","last_active":"2026-02-23T10:57:18.432893+0000","last_peered":"2026-02-23T10:57:18.432893+0000","last_clean":"2026-02-23T10:57:18.432893+0000","last_became_active":"2026-02-23T10:55:05.809738+0000","last_became_peered":"2026-02-23T10:55:05.809738+0000","last_unstale":"2026-02-23T10:57:18.432893+0000","last_undegraded":"2026-02-23T10:57:18.432893+0000","last_fullsized":"2026-02-23T10:57:18.432893+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-23T10:55:04.788039+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-23T10:55:04.788039+0000","last_clean_scrub_stamp":"2026-02-23T10:55:04.788039+0000","objects_scrubbed":0,"log_size":52,"log_dups_size":0,"ondisk_log_size":52,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":53,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":53,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":590368,"data_compressed":30575,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},"log_size":52,"ondisk_log_size":52,"up":2,"acting":2,"num_store_stats":1}],"osd_stats":[{"osd":0,"up_from":9,"seq":38654705798,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27396,"kb_used_data":628,"kb_used_omap":2050,"kb_used_meta":24701,"kb_avail":94344444,"statfs":{"total":96636764160,"available":96608710656,"internally_reserved":0,"allocated":643072,"data_stored":790138,"data_compressed":30575,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":2100148,"internal_metadata":25293900},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":10,"seq":42949673094,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27340,"kb_used_data":572,"kb_used_omap":2300,"kb_used_meta":24451,"kb_avail":94344500,"statfs":{"total":96636764160,"available":96608768000,"internally_reserved":0,"allocated":585728,"data_stored":199779,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2356177,"internal_metadata":25037871},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":10,"seq":42949673094,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27408,"kb_used_data":636,"kb_used_omap":3567,"kb_used_meta":23184,"kb_avail":94344432,"statfs":{"total":96636764160,"available":96608698368,"internally_reserved":0,"allocated":651264,"data_stored":790166,"data_compressed":30575,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":3653297,"internal_metadata":23740751},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":590368,"data_compressed":30575,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-23T10:57:21.930 INFO:tasks.ceph:pgid 2.7 last_scrub_stamp 2026-02-23T10:55:08.799034+0000 time.struct_time(tm_year=2026, tm_mon=2, tm_mday=23, tm_hour=10, tm_min=55, tm_sec=8, tm_wday=0, tm_yday=54, tm_isdst=-1) <= time.struct_time(tm_year=2026, tm_mon=2, tm_mday=23, tm_hour=10, tm_min=57, tm_sec=19, tm_wday=0, tm_yday=54, tm_isdst=0) 2026-02-23T10:57:21.930 INFO:tasks.ceph:pgid 2.6 last_scrub_stamp 2026-02-23T10:55:08.799034+0000 time.struct_time(tm_year=2026, tm_mon=2, tm_mday=23, tm_hour=10, tm_min=55, tm_sec=8, tm_wday=0, tm_yday=54, tm_isdst=-1) <= time.struct_time(tm_year=2026, tm_mon=2, tm_mday=23, tm_hour=10, tm_min=57, tm_sec=19, tm_wday=0, tm_yday=54, tm_isdst=0) 2026-02-23T10:57:21.930 INFO:tasks.ceph:pgid 2.5 last_scrub_stamp 2026-02-23T10:55:08.799034+0000 time.struct_time(tm_year=2026, tm_mon=2, tm_mday=23, tm_hour=10, tm_min=55, tm_sec=8, tm_wday=0, tm_yday=54, tm_isdst=-1) <= time.struct_time(tm_year=2026, tm_mon=2, tm_mday=23, tm_hour=10, tm_min=57, tm_sec=19, tm_wday=0, tm_yday=54, tm_isdst=0) 2026-02-23T10:57:21.930 INFO:tasks.ceph:pgid 2.4 last_scrub_stamp 2026-02-23T10:55:08.799034+0000 time.struct_time(tm_year=2026, tm_mon=2, tm_mday=23, tm_hour=10, tm_min=55, tm_sec=8, tm_wday=0, tm_yday=54, tm_isdst=-1) <= time.struct_time(tm_year=2026, tm_mon=2, tm_mday=23, tm_hour=10, tm_min=57, tm_sec=19, tm_wday=0, tm_yday=54, tm_isdst=0) 2026-02-23T10:57:21.931 INFO:tasks.ceph:pgid 2.2 last_scrub_stamp 2026-02-23T10:55:08.799034+0000 time.struct_time(tm_year=2026, tm_mon=2, tm_mday=23, tm_hour=10, tm_min=55, tm_sec=8, tm_wday=0, tm_yday=54, tm_isdst=-1) <= time.struct_time(tm_year=2026, tm_mon=2, tm_mday=23, tm_hour=10, tm_min=57, tm_sec=19, tm_wday=0, tm_yday=54, tm_isdst=0) 2026-02-23T10:57:21.931 INFO:tasks.ceph:pgid 2.1 last_scrub_stamp 2026-02-23T10:55:08.799034+0000 time.struct_time(tm_year=2026, tm_mon=2, tm_mday=23, tm_hour=10, tm_min=55, tm_sec=8, tm_wday=0, tm_yday=54, tm_isdst=-1) <= time.struct_time(tm_year=2026, tm_mon=2, tm_mday=23, tm_hour=10, tm_min=57, tm_sec=19, tm_wday=0, tm_yday=54, tm_isdst=0) 2026-02-23T10:57:21.931 INFO:tasks.ceph:pgid 2.0 last_scrub_stamp 2026-02-23T10:55:08.799034+0000 time.struct_time(tm_year=2026, tm_mon=2, tm_mday=23, tm_hour=10, tm_min=55, tm_sec=8, tm_wday=0, tm_yday=54, tm_isdst=-1) <= time.struct_time(tm_year=2026, tm_mon=2, tm_mday=23, tm_hour=10, tm_min=57, tm_sec=19, tm_wday=0, tm_yday=54, tm_isdst=0) 2026-02-23T10:57:21.931 INFO:tasks.ceph:pgid 2.3 last_scrub_stamp 2026-02-23T10:55:08.799034+0000 time.struct_time(tm_year=2026, tm_mon=2, tm_mday=23, tm_hour=10, tm_min=55, tm_sec=8, tm_wday=0, tm_yday=54, tm_isdst=-1) <= time.struct_time(tm_year=2026, tm_mon=2, tm_mday=23, tm_hour=10, tm_min=57, tm_sec=19, tm_wday=0, tm_yday=54, tm_isdst=0) 2026-02-23T10:57:21.931 INFO:tasks.ceph:pgid 1.0 last_scrub_stamp 2026-02-23T10:55:04.788039+0000 time.struct_time(tm_year=2026, tm_mon=2, tm_mday=23, tm_hour=10, tm_min=55, tm_sec=4, tm_wday=0, tm_yday=54, tm_isdst=-1) <= time.struct_time(tm_year=2026, tm_mon=2, tm_mday=23, tm_hour=10, tm_min=57, tm_sec=19, tm_wday=0, tm_yday=54, tm_isdst=0) 2026-02-23T10:57:21.932 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2026-02-23T10:57:41.932 DEBUG:teuthology.orchestra.run.trial198:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-23T10:57:42.011 INFO:teuthology.orchestra.run.trial198.stderr:2026-02-23T10:57:42.007+0000 7f9f4570b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T10:57:42.013 INFO:teuthology.orchestra.run.trial198.stderr:2026-02-23T10:57:42.012+0000 7f9f4570b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T10:57:42.123 INFO:teuthology.orchestra.run.trial198.stdout: 2026-02-23T10:57:42.123 INFO:teuthology.orchestra.run.trial198.stderr:dumped all 2026-02-23T10:57:42.132 INFO:teuthology.orchestra.run.trial198.stdout:{"pg_ready":true,"pg_map":{"version":136,"stamp":"2026-02-23T10:57:40.479745+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":53,"num_read_kb":46,"num_write":81,"num_write_kb":1108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":55,"ondisk_log_size":55,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":3,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":283115520,"kb_used":82160,"kb_used_data":1856,"kb_used_omap":7897,"kb_used_meta":72358,"kb_avail":283033360,"statfs":{"total":289910292480,"available":289826160640,"internally_reserved":0,"allocated":1900544,"data_stored":1789564,"data_compressed":61150,"data_compressed_allocated":81920,"data_compressed_original":1146880,"omap_allocated":8086773,"internal_metadata":74095371},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001806"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":161,"reported_epoch":65,"state":"active+clean","last_fresh":"2026-02-23T10:57:21.410906+0000","last_change":"2026-02-23T10:57:21.417678+0000","last_active":"2026-02-23T10:57:21.410906+0000","last_peered":"2026-02-23T10:57:21.410906+0000","last_clean":"2026-02-23T10:57:21.410906+0000","last_became_active":"2026-02-23T10:55:09.839991+0000","last_became_peered":"2026-02-23T10:55:09.839991+0000","last_unstale":"2026-02-23T10:57:21.410906+0000","last_undegraded":"2026-02-23T10:57:21.410906+0000","last_fullsized":"2026-02-23T10:57:21.410906+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-23T10:57:21.417607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-23T10:57:21.417607+0000","last_clean_scrub_stamp":"2026-02-23T10:55:08.799034+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":161,"reported_epoch":65,"state":"active+clean","last_fresh":"2026-02-23T10:57:21.408698+0000","last_change":"2026-02-23T10:57:21.415974+0000","last_active":"2026-02-23T10:57:21.408698+0000","last_peered":"2026-02-23T10:57:21.408698+0000","last_clean":"2026-02-23T10:57:21.408698+0000","last_became_active":"2026-02-23T10:55:09.841672+0000","last_became_peered":"2026-02-23T10:55:09.841672+0000","last_unstale":"2026-02-23T10:57:21.408698+0000","last_undegraded":"2026-02-23T10:57:21.408698+0000","last_fullsized":"2026-02-23T10:57:21.408698+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-23T10:57:21.415915+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-23T10:57:21.415915+0000","last_clean_scrub_stamp":"2026-02-23T10:55:08.799034+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":161,"reported_epoch":65,"state":"active+clean","last_fresh":"2026-02-23T10:57:21.408284+0000","last_change":"2026-02-23T10:57:21.418211+0000","last_active":"2026-02-23T10:57:21.408284+0000","last_peered":"2026-02-23T10:57:21.408284+0000","last_clean":"2026-02-23T10:57:21.408284+0000","last_became_active":"2026-02-23T10:55:09.834419+0000","last_became_peered":"2026-02-23T10:55:09.834419+0000","last_unstale":"2026-02-23T10:57:21.408284+0000","last_undegraded":"2026-02-23T10:57:21.408284+0000","last_fullsized":"2026-02-23T10:57:21.408284+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-23T10:57:21.418155+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-23T10:57:21.418155+0000","last_clean_scrub_stamp":"2026-02-23T10:55:08.799034+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":161,"reported_epoch":65,"state":"active+clean","last_fresh":"2026-02-23T10:57:21.408737+0000","last_change":"2026-02-23T10:57:21.417472+0000","last_active":"2026-02-23T10:57:21.408737+0000","last_peered":"2026-02-23T10:57:21.408737+0000","last_clean":"2026-02-23T10:57:21.408737+0000","last_became_active":"2026-02-23T10:55:09.846966+0000","last_became_peered":"2026-02-23T10:55:09.846966+0000","last_unstale":"2026-02-23T10:57:21.408737+0000","last_undegraded":"2026-02-23T10:57:21.408737+0000","last_fullsized":"2026-02-23T10:57:21.408737+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-23T10:57:21.417415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-23T10:57:21.417415+0000","last_clean_scrub_stamp":"2026-02-23T10:55:08.799034+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"17'2","reported_seq":163,"reported_epoch":65,"state":"active+clean","last_fresh":"2026-02-23T10:57:21.103583+0000","last_change":"2026-02-23T10:57:21.108404+0000","last_active":"2026-02-23T10:57:21.103583+0000","last_peered":"2026-02-23T10:57:21.103583+0000","last_clean":"2026-02-23T10:57:21.103583+0000","last_became_active":"2026-02-23T10:55:09.839193+0000","last_became_peered":"2026-02-23T10:55:09.839193+0000","last_unstale":"2026-02-23T10:57:21.103583+0000","last_undegraded":"2026-02-23T10:57:21.103583+0000","last_fullsized":"2026-02-23T10:57:21.103583+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"17'2","last_scrub_stamp":"2026-02-23T10:57:21.108305+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-23T10:57:21.108305+0000","last_clean_scrub_stamp":"2026-02-23T10:55:08.799034+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":161,"reported_epoch":65,"state":"active+clean","last_fresh":"2026-02-23T10:57:21.715489+0000","last_change":"2026-02-23T10:57:21.719021+0000","last_active":"2026-02-23T10:57:21.715489+0000","last_peered":"2026-02-23T10:57:21.715489+0000","last_clean":"2026-02-23T10:57:21.715489+0000","last_became_active":"2026-02-23T10:55:09.839412+0000","last_became_peered":"2026-02-23T10:55:09.839412+0000","last_unstale":"2026-02-23T10:57:21.715489+0000","last_undegraded":"2026-02-23T10:57:21.715489+0000","last_fullsized":"2026-02-23T10:57:21.715489+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-23T10:57:21.718954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-23T10:57:21.718954+0000","last_clean_scrub_stamp":"2026-02-23T10:55:08.799034+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":161,"reported_epoch":65,"state":"active+clean","last_fresh":"2026-02-23T10:57:21.714822+0000","last_change":"2026-02-23T10:57:21.719254+0000","last_active":"2026-02-23T10:57:21.714822+0000","last_peered":"2026-02-23T10:57:21.714822+0000","last_clean":"2026-02-23T10:57:21.714822+0000","last_became_active":"2026-02-23T10:55:09.834010+0000","last_became_peered":"2026-02-23T10:55:09.834010+0000","last_unstale":"2026-02-23T10:57:21.714822+0000","last_undegraded":"2026-02-23T10:57:21.714822+0000","last_fullsized":"2026-02-23T10:57:21.714822+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-23T10:57:21.719154+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-23T10:57:21.719154+0000","last_clean_scrub_stamp":"2026-02-23T10:55:08.799034+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"15'1","reported_seq":162,"reported_epoch":65,"state":"active+clean","last_fresh":"2026-02-23T10:57:21.408085+0000","last_change":"2026-02-23T10:57:21.415155+0000","last_active":"2026-02-23T10:57:21.408085+0000","last_peered":"2026-02-23T10:57:21.408085+0000","last_clean":"2026-02-23T10:57:21.408085+0000","last_became_active":"2026-02-23T10:55:09.824334+0000","last_became_peered":"2026-02-23T10:55:09.824334+0000","last_unstale":"2026-02-23T10:57:21.408085+0000","last_undegraded":"2026-02-23T10:57:21.408085+0000","last_fullsized":"2026-02-23T10:57:21.408085+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"15'1","last_scrub_stamp":"2026-02-23T10:57:21.415068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-23T10:57:21.415068+0000","last_clean_scrub_stamp":"2026-02-23T10:55:08.799034+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"12'52","reported_seq":236,"reported_epoch":65,"state":"active+clean","last_fresh":"2026-02-23T10:57:21.407374+0000","last_change":"2026-02-23T10:57:21.419605+0000","last_active":"2026-02-23T10:57:21.407374+0000","last_peered":"2026-02-23T10:57:21.407374+0000","last_clean":"2026-02-23T10:57:21.407374+0000","last_became_active":"2026-02-23T10:55:05.809738+0000","last_became_peered":"2026-02-23T10:55:05.809738+0000","last_unstale":"2026-02-23T10:57:21.407374+0000","last_undegraded":"2026-02-23T10:57:21.407374+0000","last_fullsized":"2026-02-23T10:57:21.407374+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"12'52","last_scrub_stamp":"2026-02-23T10:57:21.419531+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-23T10:57:21.419531+0000","last_clean_scrub_stamp":"2026-02-23T10:55:04.788039+0000","objects_scrubbed":0,"log_size":52,"log_dups_size":0,"ondisk_log_size":52,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":53,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":53,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":590368,"data_compressed":30575,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},"log_size":52,"ondisk_log_size":52,"up":2,"acting":2,"num_store_stats":1}],"osd_stats":[{"osd":0,"up_from":9,"seq":38654705818,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27408,"kb_used_data":640,"kb_used_omap":2026,"kb_used_meta":24725,"kb_avail":94344432,"statfs":{"total":96636764160,"available":96608698368,"internally_reserved":0,"allocated":655360,"data_stored":793317,"data_compressed":30575,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":2075403,"internal_metadata":25318645},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":10,"seq":42949673114,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27344,"kb_used_data":576,"kb_used_omap":2372,"kb_used_meta":24379,"kb_avail":94344496,"statfs":{"total":96636764160,"available":96608763904,"internally_reserved":0,"allocated":589824,"data_stored":202930,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2429226,"internal_metadata":24964822},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":10,"seq":42949673114,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27408,"kb_used_data":640,"kb_used_omap":3498,"kb_used_meta":23253,"kb_avail":94344432,"statfs":{"total":96636764160,"available":96608698368,"internally_reserved":0,"allocated":655360,"data_stored":793317,"data_compressed":30575,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":3582144,"internal_metadata":23811904},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":590368,"data_compressed":30575,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-23T10:57:42.134 DEBUG:teuthology.orchestra.run.trial198:> sudo ceph --cluster ceph config set global mon_health_to_clog false 2026-02-23T10:57:42.211 INFO:teuthology.orchestra.run.trial198.stderr:2026-02-23T10:57:42.206+0000 7eff35a1a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T10:57:42.213 INFO:teuthology.orchestra.run.trial198.stderr:2026-02-23T10:57:42.212+0000 7eff35a1a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T10:57:42.339 INFO:teuthology.misc:Shutting down mds daemons... 2026-02-23T10:57:42.340 INFO:teuthology.misc:Shutting down osd daemons... 2026-02-23T10:57:42.340 DEBUG:tasks.ceph.osd.0:waiting for process to exit 2026-02-23T10:57:42.340 INFO:teuthology.orchestra.run:waiting for 300 2026-02-23T10:57:42.408 INFO:tasks.ceph.osd.0:Stopped 2026-02-23T10:57:42.408 DEBUG:tasks.ceph.osd.1:waiting for process to exit 2026-02-23T10:57:42.408 INFO:teuthology.orchestra.run:waiting for 300 2026-02-23T10:57:42.484 INFO:tasks.ceph.osd.1:Stopped 2026-02-23T10:57:42.484 DEBUG:tasks.ceph.osd.2:waiting for process to exit 2026-02-23T10:57:42.484 INFO:teuthology.orchestra.run:waiting for 300 2026-02-23T10:57:42.572 INFO:tasks.ceph.osd.2:Stopped 2026-02-23T10:57:42.572 INFO:teuthology.misc:Shutting down mgr daemons... 2026-02-23T10:57:42.572 DEBUG:tasks.ceph.mgr.x:waiting for process to exit 2026-02-23T10:57:42.572 INFO:teuthology.orchestra.run:waiting for 300 2026-02-23T10:57:42.603 INFO:tasks.ceph.mgr.x:Stopped 2026-02-23T10:57:42.603 INFO:teuthology.misc:Shutting down mon daemons... 2026-02-23T10:57:42.604 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2026-02-23T10:57:42.604 INFO:teuthology.orchestra.run:waiting for 300 2026-02-23T10:57:42.654 INFO:tasks.ceph.mon.a:Stopped 2026-02-23T10:57:42.654 INFO:tasks.ceph:Checking cluster log for badness... 2026-02-23T10:57:42.654 DEBUG:teuthology.orchestra.run.trial198:> sudo grep -E '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/ceph.log | grep -E -v '\(MDS_ALL_DOWN\)' | grep -E -v '\(MDS_UP_LESS_THAN_MAX\)' | grep -E -v '\(POOL_APP_NOT_ENABLED\)' | head -n 1 2026-02-23T10:57:42.720 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-0 on ubuntu@trial198.front.sepia.ceph.com 2026-02-23T10:57:42.720 DEBUG:teuthology.orchestra.run.trial198:> sync && sudo umount -f /var/lib/ceph/osd/ceph-0 2026-02-23T10:57:42.788 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-1 on ubuntu@trial198.front.sepia.ceph.com 2026-02-23T10:57:42.788 DEBUG:teuthology.orchestra.run.trial198:> sync && sudo umount -f /var/lib/ceph/osd/ceph-1 2026-02-23T10:57:42.855 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-2 on ubuntu@trial198.front.sepia.ceph.com 2026-02-23T10:57:42.855 DEBUG:teuthology.orchestra.run.trial198:> sync && sudo umount -f /var/lib/ceph/osd/ceph-2 2026-02-23T10:57:42.922 INFO:tasks.ceph:Archiving mon data... 2026-02-23T10:57:42.924 DEBUG:teuthology.misc:Transferring archived files from trial198:/var/lib/ceph/mon/ceph-a to /home/teuthworker/mnt/teuthology/shraddhaag-2026-02-23_09:58:48-crimson-rados-main-distro-debug-trial/66138/data/mon.a.tgz 2026-02-23T10:57:42.924 DEBUG:teuthology.orchestra.run.trial198:> mktemp 2026-02-23T10:57:42.978 INFO:teuthology.orchestra.run.trial198.stdout:/tmp/tmp.NQwNAA1uhS 2026-02-23T10:57:42.979 DEBUG:teuthology.orchestra.run.trial198:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-a -- . > /tmp/tmp.NQwNAA1uhS 2026-02-23T10:57:43.068 DEBUG:teuthology.orchestra.run.trial198:> sudo chmod 0666 /tmp/tmp.NQwNAA1uhS 2026-02-23T10:57:43.149 DEBUG:teuthology.orchestra.remote:trial198:/tmp/tmp.NQwNAA1uhS is 221KB 2026-02-23T10:57:43.242 DEBUG:teuthology.orchestra.run.trial198:> rm -fr /tmp/tmp.NQwNAA1uhS 2026-02-23T10:57:43.260 INFO:tasks.ceph:Cleaning ceph cluster... 2026-02-23T10:57:43.260 DEBUG:teuthology.orchestra.run.trial198:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2026-02-23T10:57:43.380 INFO:teuthology.util.scanner:summary_data or yaml_file is empty! 2026-02-23T10:57:43.380 INFO:tasks.ceph:Archiving crash dumps... 2026-02-23T10:57:43.383 DEBUG:teuthology.misc:Transferring archived files from trial198:/var/lib/ceph/crash to /home/teuthworker/mnt/teuthology/shraddhaag-2026-02-23_09:58:48-crimson-rados-main-distro-debug-trial/66138/remote/trial198/crash 2026-02-23T10:57:43.384 DEBUG:teuthology.orchestra.run.trial198:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2026-02-23T10:57:43.444 INFO:tasks.ceph:Compressing logs... 2026-02-23T10:57:43.444 DEBUG:teuthology.orchestra.run.trial198:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2026-02-23T10:57:43.526 INFO:teuthology.orchestra.run.trial198.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.2.log 2026-02-23T10:57:43.526 INFO:teuthology.orchestra.run.trial198.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40742.log 2026-02-23T10:57:43.526 INFO:teuthology.orchestra.run.trial198.stderr:/var/log/ceph/ceph-osd.2.log: gzip -5 --verbose -- /var/log/ceph/ceph-mon.a.log 2026-02-23T10:57:43.526 INFO:teuthology.orchestra.run.trial198.stderr:/var/log/ceph/ceph-client.admin.40742.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42065.log 2026-02-23T10:57:43.527 INFO:teuthology.orchestra.run.trial198.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.40742.log.gz 2026-02-23T10:57:43.527 INFO:teuthology.orchestra.run.trial198.stderr:/var/log/ceph/ceph-mon.a.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.0.log 2026-02-23T10:57:43.527 INFO:teuthology.orchestra.run.trial198.stderr:/var/log/ceph/ceph-client.admin.42065.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42454.log 2026-02-23T10:57:43.527 INFO:teuthology.orchestra.run.trial198.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.42065.log.gz 2026-02-23T10:57:43.527 INFO:teuthology.orchestra.run.trial198.stderr:/var/log/ceph/ceph-osd.0.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41817.log 2026-02-23T10:57:43.527 INFO:teuthology.orchestra.run.trial198.stderr:/var/log/ceph/ceph-client.admin.42454.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43467.log 2026-02-23T10:57:43.527 INFO:teuthology.orchestra.run.trial198.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.42454.log.gz 2026-02-23T10:57:43.527 INFO:teuthology.orchestra.run.trial198.stderr:/var/log/ceph/ceph-client.admin.41817.log: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.41817.log.gzgzip 2026-02-23T10:57:43.527 INFO:teuthology.orchestra.run.trial198.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.43271.log 2026-02-23T10:57:43.527 INFO:teuthology.orchestra.run.trial198.stderr:/var/log/ceph/ceph-client.admin.43467.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40511.log 2026-02-23T10:57:43.527 INFO:teuthology.orchestra.run.trial198.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.43467.log.gz 2026-02-23T10:57:43.527 INFO:teuthology.orchestra.run.trial198.stderr:/var/log/ceph/ceph-client.admin.43271.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41918.log 2026-02-23T10:57:43.528 INFO:teuthology.orchestra.run.trial198.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.43271.log.gz 2026-02-23T10:57:43.528 INFO:teuthology.orchestra.run.trial198.stderr:/var/log/ceph/ceph-client.admin.40511.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43810.log 2026-02-23T10:57:43.528 INFO:teuthology.orchestra.run.trial198.stderr: 48.0% -- replaced with /var/log/ceph/ceph-client.admin.40511.log.gz 2026-02-23T10:57:43.528 INFO:teuthology.orchestra.run.trial198.stderr:/var/log/ceph/ceph-client.admin.41918.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.41918.log.gz 2026-02-23T10:57:43.528 INFO:teuthology.orchestra.run.trial198.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43079.log 2026-02-23T10:57:43.528 INFO:teuthology.orchestra.run.trial198.stderr:/var/log/ceph/ceph-client.admin.43810.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41768.log 2026-02-23T10:57:43.528 INFO:teuthology.orchestra.run.trial198.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.43810.log.gz 2026-02-23T10:57:43.528 INFO:teuthology.orchestra.run.trial198.stderr:/var/log/ceph/ceph-client.admin.43079.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42187.log 2026-02-23T10:57:43.528 INFO:teuthology.orchestra.run.trial198.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.43079.log.gz 2026-02-23T10:57:43.528 INFO:teuthology.orchestra.run.trial198.stderr:/var/log/ceph/ceph-client.admin.41768.log: gzip -5 --verbose -- /var/log/ceph/ceph.tmp-client.admin.38717.log 2026-02-23T10:57:43.528 INFO:teuthology.orchestra.run.trial198.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.41768.log.gz 2026-02-23T10:57:43.528 INFO:teuthology.orchestra.run.trial198.stderr:/var/log/ceph/ceph-client.admin.42187.log: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.42187.log.gz 2026-02-23T10:57:43.528 INFO:teuthology.orchestra.run.trial198.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43320.log 2026-02-23T10:57:43.529 INFO:teuthology.orchestra.run.trial198.stderr:/var/log/ceph/ceph.tmp-client.admin.38717.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42405.log 2026-02-23T10:57:43.529 INFO:teuthology.orchestra.run.trial198.stderr: 66.9% -- replaced with /var/log/ceph/ceph.tmp-client.admin.38717.log.gz 2026-02-23T10:57:43.529 INFO:teuthology.orchestra.run.trial198.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41580.log 2026-02-23T10:57:43.529 INFO:teuthology.orchestra.run.trial198.stderr:/var/log/ceph/ceph-client.admin.43320.log: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.43320.log.gz 2026-02-23T10:57:43.529 INFO:teuthology.orchestra.run.trial198.stderr:/var/log/ceph/ceph-client.admin.42405.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42728.log 2026-02-23T10:57:43.529 INFO:teuthology.orchestra.run.trial198.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.42405.log.gz 2026-02-23T10:57:43.529 INFO:teuthology.orchestra.run.trial198.stderr:/var/log/ceph/ceph-client.admin.41580.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.41580.log.gz 2026-02-23T10:57:43.529 INFO:teuthology.orchestra.run.trial198.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43712.log 2026-02-23T10:57:43.529 INFO:teuthology.orchestra.run.trial198.stderr:/var/log/ceph/ceph-client.admin.42728.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43614.log 2026-02-23T10:57:43.529 INFO:teuthology.orchestra.run.trial198.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.42728.log.gz 2026-02-23T10:57:43.529 INFO:teuthology.orchestra.run.trial198.stderr:/var/log/ceph/ceph-client.admin.43712.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40416.log 2026-02-23T10:57:43.529 INFO:teuthology.orchestra.run.trial198.stderr: 49.2% -- replaced with /var/log/ceph/ceph-client.admin.43712.log.gz 2026-02-23T10:57:43.529 INFO:teuthology.orchestra.run.trial198.stderr:/var/log/ceph/ceph-client.admin.43614.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40558.log 2026-02-23T10:57:43.530 INFO:teuthology.orchestra.run.trial198.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.43614.log.gz 2026-02-23T10:57:43.530 INFO:teuthology.orchestra.run.trial198.stderr:/var/log/ceph/ceph-client.admin.40416.log: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.40416.log.gz 2026-02-23T10:57:43.530 INFO:teuthology.orchestra.run.trial198.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42799.log 2026-02-23T10:57:43.530 INFO:teuthology.orchestra.run.trial198.stderr:/var/log/ceph/ceph-client.admin.40558.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40695.log 2026-02-23T10:57:43.530 INFO:teuthology.orchestra.run.trial198.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.40558.log.gz 2026-02-23T10:57:43.530 INFO:teuthology.orchestra.run.trial198.stderr:/var/log/ceph/ceph-client.admin.42799.log: gzip -5 --verbose 89.7% -- -- replaced with /var/log/ceph/ceph-client.admin.42799.log.gz /var/log/ceph/ceph-client.admin.42160.log 2026-02-23T10:57:43.530 INFO:teuthology.orchestra.run.trial198.stderr: 2026-02-23T10:57:43.530 INFO:teuthology.orchestra.run.trial198.stderr:/var/log/ceph/ceph-client.admin.40695.log: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.40695.log.gz 2026-02-23T10:57:43.530 INFO:teuthology.orchestra.run.trial198.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41920.log 2026-02-23T10:57:43.530 INFO:teuthology.orchestra.run.trial198.stderr:/var/log/ceph/ceph-client.admin.42160.log: gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2026-02-23T10:57:43.530 INFO:teuthology.orchestra.run.trial198.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.42160.log.gz 2026-02-23T10:57:43.530 INFO:teuthology.orchestra.run.trial198.stderr:/var/log/ceph/ceph-client.admin.41920.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.41920.log.gz 2026-02-23T10:57:43.531 INFO:teuthology.orchestra.run.trial198.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42066.log 2026-02-23T10:57:43.531 INFO:teuthology.orchestra.run.trial198.stderr:gzip/var/log/ceph/ceph.audit.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.43663.log 2026-02-23T10:57:43.531 INFO:teuthology.orchestra.run.trial198.stderr:/var/log/ceph/ceph-client.admin.42066.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41919.log 2026-02-23T10:57:43.531 INFO:teuthology.orchestra.run.trial198.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.42066.log.gz 2026-02-23T10:57:43.531 INFO:teuthology.orchestra.run.trial198.stderr: 87.8% -- replaced with /var/log/ceph/ceph.audit.log.gz 2026-02-23T10:57:43.531 INFO:teuthology.orchestra.run.trial198.stderr:/var/log/ceph/ceph-client.admin.43663.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.43663.log.gz 2026-02-23T10:57:43.531 INFO:teuthology.orchestra.run.trial198.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.1.log 2026-02-23T10:57:43.531 INFO:teuthology.orchestra.run.trial198.stderr:/var/log/ceph/ceph-client.admin.41919.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40789.log 2026-02-23T10:57:43.531 INFO:teuthology.orchestra.run.trial198.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.41919.log.gz 2026-02-23T10:57:43.531 INFO:teuthology.orchestra.run.trial198.stderr:/var/log/ceph/ceph-osd.1.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41008.log 2026-02-23T10:57:43.531 INFO:teuthology.orchestra.run.trial198.stderr:/var/log/ceph/ceph-client.admin.40789.log: gzip -5 50.0% --verbose -- -- replaced with /var/log/ceph/ceph-client.admin.40789.log.gz /var/log/ceph/ceph-client.admin.42356.log 2026-02-23T10:57:43.531 INFO:teuthology.orchestra.run.trial198.stderr: 2026-02-23T10:57:43.531 INFO:teuthology.orchestra.run.trial198.stderr:/var/log/ceph/ceph-client.admin.41008.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43763.log 2026-02-23T10:57:43.532 INFO:teuthology.orchestra.run.trial198.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.41008.log.gz 2026-02-23T10:57:43.532 INFO:teuthology.orchestra.run.trial198.stderr:/var/log/ceph/ceph-client.admin.42356.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43516.log 2026-02-23T10:57:43.532 INFO:teuthology.orchestra.run.trial198.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.42356.log.gz 2026-02-23T10:57:43.532 INFO:teuthology.orchestra.run.trial198.stderr:/var/log/ceph/ceph-client.admin.43763.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.43763.log.gz 2026-02-23T10:57:43.532 INFO:teuthology.orchestra.run.trial198.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42258.log 2026-02-23T10:57:43.532 INFO:teuthology.orchestra.run.trial198.stderr:/var/log/ceph/ceph-client.admin.43516.log: gzip -5 --verbose -- /var/log/ceph/ceph-mgr.x.log 2026-02-23T10:57:43.532 INFO:teuthology.orchestra.run.trial198.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.43516.log.gz 2026-02-23T10:57:43.532 INFO:teuthology.orchestra.run.trial198.stderr:/var/log/ceph/ceph-client.admin.42258.log: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.42258.log.gz 2026-02-23T10:57:43.532 INFO:teuthology.orchestra.run.trial198.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42286.log 2026-02-23T10:57:43.532 INFO:teuthology.orchestra.run.trial198.stderr:/var/log/ceph/ceph-mgr.x.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41670.log 2026-02-23T10:57:43.532 INFO:teuthology.orchestra.run.trial198.stderr:/var/log/ceph/ceph-client.admin.42286.log: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.42286.log.gz 2026-02-23T10:57:43.532 INFO:teuthology.orchestra.run.trial198.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43565.log 2026-02-23T10:57:43.532 INFO:teuthology.orchestra.run.trial198.stderr:/var/log/ceph/ceph-client.admin.41670.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41510.log 2026-02-23T10:57:43.533 INFO:teuthology.orchestra.run.trial198.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.41670.log.gz 2026-02-23T10:57:43.533 INFO:teuthology.orchestra.run.trial198.stderr:/var/log/ceph/ceph-client.admin.43565.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.43565.log.gz 2026-02-23T10:57:43.533 INFO:teuthology.orchestra.run.trial198.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41719.log 2026-02-23T10:57:43.533 INFO:teuthology.orchestra.run.trial198.stderr:/var/log/ceph/ceph-client.admin.41510.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41605.log 2026-02-23T10:57:43.533 INFO:teuthology.orchestra.run.trial198.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.41510.log.gz 2026-02-23T10:57:43.533 INFO:teuthology.orchestra.run.trial198.stderr:/var/log/ceph/ceph-client.admin.41719.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.41719.log.gz 2026-02-23T10:57:43.533 INFO:teuthology.orchestra.run.trial198.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42067.log 2026-02-23T10:57:43.533 INFO:teuthology.orchestra.run.trial198.stderr:/var/log/ceph/ceph-client.admin.41605.log: 66.7% -- replaced with /var/log/ceph/ceph-client.admin.41605.log.gz 2026-02-23T10:57:43.533 INFO:teuthology.orchestra.run.trial198.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.log 2026-02-23T10:57:43.533 INFO:teuthology.orchestra.run.trial198.stderr:/var/log/ceph/ceph-client.admin.42067.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43369.log 2026-02-23T10:57:43.533 INFO:teuthology.orchestra.run.trial198.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.42067.log.gz 2026-02-23T10:57:43.533 INFO:teuthology.orchestra.run.trial198.stderr:/var/log/ceph/ceph.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43418.log 2026-02-23T10:57:43.533 INFO:teuthology.orchestra.run.trial198.stderr: 85.6% -- replaced with /var/log/ceph/ceph.log.gz 2026-02-23T10:57:43.533 INFO:teuthology.orchestra.run.trial198.stderr:/var/log/ceph/ceph-client.admin.43369.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43222.log 2026-02-23T10:57:43.534 INFO:teuthology.orchestra.run.trial198.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.43369.log.gz 2026-02-23T10:57:43.534 INFO:teuthology.orchestra.run.trial198.stderr:/var/log/ceph/ceph-client.admin.43418.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41404.log 2026-02-23T10:57:43.534 INFO:teuthology.orchestra.run.trial198.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.43418.log.gz 2026-02-23T10:57:43.534 INFO:teuthology.orchestra.run.trial198.stderr:/var/log/ceph/ceph-client.admin.43222.log: gzip -5 --verbose 50.0% -- -- replaced with /var/log/ceph/ceph-client.admin.43222.log.gz /var/log/ceph/ceph-client.admin.41116.log 2026-02-23T10:57:43.534 INFO:teuthology.orchestra.run.trial198.stderr: 2026-02-23T10:57:43.534 INFO:teuthology.orchestra.run.trial198.stderr:/var/log/ceph/ceph-client.admin.41404.log: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.41404.log.gz 2026-02-23T10:57:43.534 INFO:teuthology.orchestra.run.trial198.stderr:/var/log/ceph/ceph-client.admin.41116.log: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.41116.log.gz 2026-02-23T10:57:43.535 INFO:teuthology.orchestra.run.trial198.stderr: 91.0% -- replaced with /var/log/ceph/ceph-mgr.x.log.gz 2026-02-23T10:57:43.568 INFO:teuthology.orchestra.run.trial198.stderr: 90.8% -- replaced with /var/log/ceph/ceph-mon.a.log.gz 2026-02-23T10:57:44.005 INFO:teuthology.orchestra.run.trial198.stderr: 93.1% -- replaced with /var/log/ceph/ceph-osd.2.log.gz 2026-02-23T10:57:44.158 INFO:teuthology.orchestra.run.trial198.stderr: 93.7% -- replaced with /var/log/ceph/ceph-osd.1.log.gz 2026-02-23T10:57:44.451 INFO:teuthology.orchestra.run.trial198.stderr: 92.5% -- replaced with /var/log/ceph/ceph-osd.0.log.gz 2026-02-23T10:57:44.453 INFO:teuthology.orchestra.run.trial198.stderr: 2026-02-23T10:57:44.453 INFO:teuthology.orchestra.run.trial198.stderr:real 0m0.951s 2026-02-23T10:57:44.453 INFO:teuthology.orchestra.run.trial198.stderr:user 0m2.023s 2026-02-23T10:57:44.453 INFO:teuthology.orchestra.run.trial198.stderr:sys 0m0.167s 2026-02-23T10:57:44.453 INFO:tasks.ceph:Archiving logs... 2026-02-23T10:57:44.453 DEBUG:teuthology.misc:Transferring archived files from trial198:/var/log/ceph to /home/teuthworker/mnt/teuthology/shraddhaag-2026-02-23_09:58:48-crimson-rados-main-distro-debug-trial/66138/remote/trial198/log 2026-02-23T10:57:44.455 DEBUG:teuthology.orchestra.run.trial198:> sudo tar c -f - -C /var/log/ceph -- . 2026-02-23T10:57:44.802 DEBUG:teuthology.run_tasks:Unwinding manager install 2026-02-23T10:57:44.877 DEBUG:teuthology.run_tasks:Unwinding manager install 2026-02-23T10:57:44.910 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2026-02-23T10:57:44.910 DEBUG:teuthology.orchestra.run.trial198:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2026-02-23T10:57:44.936 DEBUG:teuthology.run_tasks:Unwinding manager clock 2026-02-23T10:57:44.947 INFO:teuthology.task.clock:Checking final clock skew... 2026-02-23T10:57:44.947 DEBUG:teuthology.orchestra.run.trial198:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2026-02-23T10:57:44.991 INFO:teuthology.orchestra.run.trial198.stderr:bash: line 1: ntpq: command not found 2026-02-23T10:57:45.006 INFO:teuthology.orchestra.run.trial198.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2026-02-23T10:57:45.006 INFO:teuthology.orchestra.run.trial198.stdout:=============================================================================== 2026-02-23T10:57:45.006 INFO:teuthology.orchestra.run.trial198.stdout:^+ 23-95-35-34-host.colocro> 2 6 377 11 +227us[ +64us] +/- 36ms 2026-02-23T10:57:45.006 INFO:teuthology.orchestra.run.trial198.stdout:^+ 149.248.12.167.vultruser> 3 6 377 9 -303us[ -303us] +/- 34ms 2026-02-23T10:57:45.006 INFO:teuthology.orchestra.run.trial198.stdout:^* 144.202.62.209.vultruser> 3 6 377 10 -329us[ -492us] +/- 20ms 2026-02-23T10:57:45.007 INFO:teuthology.orchestra.run.trial198.stdout:^- 23-93-251-54.dedicated.s> 1 6 377 7 +1057us[+1057us] +/- 135ms 2026-02-23T10:57:45.008 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2026-02-23T10:57:45.023 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2026-02-23T10:57:45.023 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2026-02-23T10:57:45.038 DEBUG:teuthology.orchestra.run.trial198:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2026-02-23T10:57:45.104 DEBUG:teuthology.orchestra.run.trial198:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 2026-02-23T10:57:45.128 DEBUG:teuthology.orchestra.run:got remote process result: 1 2026-02-23T10:57:45.129 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2026-02-23T10:57:45.145 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2026-02-23T10:57:45.162 INFO:teuthology.task.internal:Duration was 631.699606 seconds 2026-02-23T10:57:45.162 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2026-02-23T10:57:45.177 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2026-02-23T10:57:45.177 DEBUG:teuthology.orchestra.run.trial198:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2026-02-23T10:57:45.214 INFO:teuthology.orchestra.run.trial198.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2026-02-23T10:57:45.453 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2026-02-23T10:57:45.453 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@trial198.front.sepia.ceph.com 2026-02-23T10:57:45.453 DEBUG:teuthology.orchestra.run.trial198:> grep -E --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | grep -E -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | grep -E -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2026-02-23T10:57:45.511 INFO:teuthology.task.internal.syslog:Gathering journactl... 2026-02-23T10:57:45.512 DEBUG:teuthology.orchestra.run.trial198:> sudo journalctl > /home/ubuntu/cephtest/archive/syslog/journalctl.log 2026-02-23T10:57:45.614 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2026-02-23T10:57:45.614 DEBUG:teuthology.orchestra.run.trial198:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --max-args=1 --max-procs=0 --verbose --no-run-if-empty -- gzip -5 --verbose -- 2026-02-23T10:57:45.637 INFO:teuthology.orchestra.run.trial198.stderr:gzip -5 --verbose -- /home/ubuntu/cephtest/archive/syslog/misc.log 2026-02-23T10:57:45.639 INFO:teuthology.orchestra.run.trial198.stderr:gzip -5 --verbose -- /home/ubuntu/cephtest/archive/syslog/kern.log 2026-02-23T10:57:45.639 INFO:teuthology.orchestra.run.trial198.stderr:/home/ubuntu/cephtest/archive/syslog/misc.log: 0.0% -- replaced with /home/ubuntu/cephtest/archive/syslog/misc.log.gz 2026-02-23T10:57:45.639 INFO:teuthology.orchestra.run.trial198.stderr:gzip -5 --verbose -- /home/ubuntu/cephtest/archive/syslog/journalctl.log 2026-02-23T10:57:45.639 INFO:teuthology.orchestra.run.trial198.stderr:/home/ubuntu/cephtest/archive/syslog/kern.log: 0.0% -- replaced with /home/ubuntu/cephtest/archive/syslog/kern.log.gz 2026-02-23T10:57:45.649 INFO:teuthology.orchestra.run.trial198.stderr:/home/ubuntu/cephtest/archive/syslog/journalctl.log: 82.0% -- replaced with /home/ubuntu/cephtest/archive/syslog/journalctl.log.gz 2026-02-23T10:57:45.651 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2026-02-23T10:57:45.668 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2026-02-23T10:57:45.668 DEBUG:teuthology.orchestra.run.trial198:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2026-02-23T10:57:45.715 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2026-02-23T10:57:45.732 DEBUG:teuthology.orchestra.run.trial198:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2026-02-23T10:57:45.778 INFO:teuthology.orchestra.run.trial198.stdout:kernel.core_pattern = core 2026-02-23T10:57:45.796 DEBUG:teuthology.orchestra.run.trial198:> test -e /home/ubuntu/cephtest/archive/coredump 2026-02-23T10:57:45.851 DEBUG:teuthology.orchestra.run:got remote process result: 1 2026-02-23T10:57:45.851 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2026-02-23T10:57:45.869 INFO:teuthology.task.internal:Transferring archived files... 2026-02-23T10:57:45.870 DEBUG:teuthology.misc:Transferring archived files from trial198:/home/ubuntu/cephtest/archive to /home/teuthworker/mnt/teuthology/shraddhaag-2026-02-23_09:58:48-crimson-rados-main-distro-debug-trial/66138/remote/trial198 2026-02-23T10:57:45.871 DEBUG:teuthology.orchestra.run.trial198:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2026-02-23T10:57:45.927 INFO:teuthology.task.internal:Removing archive directory... 2026-02-23T10:57:45.928 DEBUG:teuthology.orchestra.run.trial198:> rm -rf -- /home/ubuntu/cephtest/archive 2026-02-23T10:57:45.976 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2026-02-23T10:57:45.994 INFO:teuthology.task.internal:Not uploading archives. 2026-02-23T10:57:45.994 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2026-02-23T10:57:46.011 INFO:teuthology.task.internal:Tidying up after the test... 2026-02-23T10:57:46.011 DEBUG:teuthology.orchestra.run.trial198:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2026-02-23T10:57:46.034 INFO:teuthology.orchestra.run.trial198.stdout: 917576 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Feb 23 10:57 /home/ubuntu/cephtest 2026-02-23T10:57:46.035 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2026-02-23T10:57:46.053 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2026-02-23T10:57:46.091 INFO:teuthology.run:Summary data: description: crimson-rados/rbd/{0-crimson_install clusters/crimson-fixed-1 crimson-supported-all-distro/centos_latest crimson_qa_overrides deploy/ceph objectstore/crimson_bluestore tasks/rbd_cls_tests} duration: 631.699606180191 flavor: debug owner: scheduled_shraddhaag@soko04.front.sepia.ceph.com success: true 2026-02-23T10:57:46.092 DEBUG:teuthology.report:Pushing job info to https://paddles-paddles.apps.pok.os.sepia.ceph.com 2026-02-23T10:57:46.208 INFO:teuthology.run:pass