2026-02-23T11:09:33.287 INFO:root:teuthology version: 1.2.4.dev9+gfa17720d0 2026-02-23T11:09:33.300 DEBUG:teuthology.report:Pushing job info to https://paddles-paddles.apps.pok.os.sepia.ceph.com 2026-02-23T11:09:33.379 INFO:teuthology.run:Config: archive_path: /home/teuthworker/mnt/teuthology/shraddhaag-2026-02-23_09:58:48-crimson-rados-main-distro-debug-trial/66207 branch: main description: crimson-rados/rbd/{0-crimson_install clusters/crimson-fixed-1 crimson-supported-all-distro/centos_latest crimson_qa_overrides deploy/ceph objectstore/seastore/segmented$/{crimson_seastore_segmented_2q} tasks/rbd_python_api_tests_old_format} email: null first_in_suite: false flavor: debug job_id: '66207' kernel: branch: distro kdb: true sha1: distro last_in_suite: false machine_type: trial name: shraddhaag-2026-02-23_09:58:48-crimson-rados-main-distro-debug-trial no_nested_subset: false os_type: centos os_version: 9.stream overrides: admin_socket: branch: main ceph: conf: global: enable experimental unrecoverable data corrupting features: crimson ms client mode: crc ms cluster mode: crc ms mon client mode: crc ms mon cluster mode: crc ms mon service mode: crc ms service mode: crc mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 osd pool default crimson: true osd: crimson cpu num: 4 crimson osd obc lru size: 10 debug ms: 20 debug osd: 20 debug seastore: 20 debug seastore backref: 20 debug seastore cache: 20 debug seastore cleaner: 20 debug seastore device: 20 debug seastore epm: 20 debug seastore fixedkv tree: 20 debug seastore journal: 20 debug seastore lba: 20 debug seastore odata: 20 debug seastore omap: 20 debug seastore onode: 20 debug seastore t: 20 debug seastore tm: 20 osd objectstore: seastore osd shutdown pgref assert: true seastore cachepin size pershard: 64M seastore cachepin type: 2Q seastore max concurrent transactions: 8 crimson_compat: true flavor: debug log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - \(POOL_APP_NOT_ENABLED\) - \(SLOW_OPS\) - slow request sha1: e371433fcd1390a0b82cfb18effc422c1f5d8c1d ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log global: osd crush chooseleaf type: 0 osd pool default pg num: 128 osd pool default pgp num: 128 osd pool default size: 2 mon: {} install: ceph: extra_system_packages: - python3-pytest flavor: debug sha1: e371433fcd1390a0b82cfb18effc422c1f5d8c1d workunit: branch: wip-shraddhaag-enable-cephadm-tests env: CRIMSON_COMPAT: '1' sha1: e371433fcd1390a0b82cfb18effc422c1f5d8c1d owner: scheduled_shraddhaag@soko04.front.sepia.ceph.com priority: 49 repo: https://git.ceph.com/ceph-ci.git roles: - - mon.a - mgr.x - osd.0 - osd.1 - osd.2 - client.0 seed: 66 sha1: e371433fcd1390a0b82cfb18effc422c1f5d8c1d sleep_before_teardown: 0 suite: crimson-rados suite_branch: wip-shraddhaag-enable-cephadm-tests suite_path: /home/teuthworker/src/github.com_shraddhaag_ceph_e371433fcd1390a0b82cfb18effc422c1f5d8c1d/qa suite_relpath: qa suite_repo: https://github.com/shraddhaag/ceph.git suite_sha1: e371433fcd1390a0b82cfb18effc422c1f5d8c1d targets: trial030.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBJTQTYanJEsMxlFPAGNwdTzaNEO9mmBJiTGJf/qdsjP35Ic4gX1XnBG/0B2NzQslEsCQApuXOnrtX5njotcds4Q= tasks: - install: extra_packages: deb: - ceph-osd-crimson - ceph-osd-crimson-dbg rpm: - ceph-osd-crimson - ceph-osd-crimson-debuginfo - pexec: all: - sudo update-alternatives --set ceph-osd /usr/bin/ceph-osd-crimson - install: null - ceph: conf: mon: mon min osdmap epochs: 50 mon osdmap full prune interval: 2 mon osdmap full prune min: 15 mon osdmap full prune txsize: 2 paxos service trim min: 10 osd: debug monc: 20 - workunit: clients: client.0: - rbd/test_librbd_python.sh teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: fa17720d0088c3ac28e473468bfc79eeaff5cd38 timestamp: 2026-02-23_09:58:48 tube: trial user: shraddhaag verbose: false worker_log: /home/teuthworker/mnt/teuthology/worker_logs/dispatcher.trial.3318653 2026-02-23T11:09:33.379 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/github.com_shraddhaag_ceph_e371433fcd1390a0b82cfb18effc422c1f5d8c1d/qa; will attempt to use it 2026-02-23T11:09:33.380 INFO:teuthology.run:Found tasks at /home/teuthworker/src/github.com_shraddhaag_ceph_e371433fcd1390a0b82cfb18effc422c1f5d8c1d/qa/tasks 2026-02-23T11:09:33.380 INFO:teuthology.run_tasks:Running task internal.check_packages... 2026-02-23T11:09:33.381 INFO:teuthology.task.internal:Checking packages... 2026-02-23T11:09:33.381 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'debug' and ceph hash 'e371433fcd1390a0b82cfb18effc422c1f5d8c1d' 2026-02-23T11:09:33.381 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2026-02-23T11:09:33.381 INFO:teuthology.packaging:ref: None 2026-02-23T11:09:33.381 INFO:teuthology.packaging:tag: None 2026-02-23T11:09:33.381 INFO:teuthology.packaging:branch: main 2026-02-23T11:09:33.381 INFO:teuthology.packaging:sha1: e371433fcd1390a0b82cfb18effc422c1f5d8c1d 2026-02-23T11:09:33.382 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=debug&distros=centos%2F9%2Fx86_64&ref=main 2026-02-23T11:09:33.506 INFO:teuthology.task.internal:Found packages for ceph version 20.3.0-5253.g5ccdd6ad 2026-02-23T11:09:33.507 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2026-02-23T11:09:33.519 INFO:teuthology.task.internal:no buildpackages task found 2026-02-23T11:09:33.519 INFO:teuthology.run_tasks:Running task internal.save_config... 2026-02-23T11:09:33.534 INFO:teuthology.task.internal:Saving configuration 2026-02-23T11:09:33.546 INFO:teuthology.run_tasks:Running task internal.check_lock... 2026-02-23T11:09:33.559 INFO:teuthology.task.internal.check_lock:Checking locks... 2026-02-23T11:09:33.632 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'trial030.front.sepia.ceph.com', 'description': '/home/teuthworker/mnt/teuthology/shraddhaag-2026-02-23_09:58:48-crimson-rados-main-distro-debug-trial/66207', 'up': True, 'machine_type': 'trial', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9.stream', 'arch': 'x86_64', 'locked': True, 'locked_since': '2026-02-23 11:06:10.612653', 'locked_by': 'scheduled_shraddhaag@soko04.front.sepia.ceph.com', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBJTQTYanJEsMxlFPAGNwdTzaNEO9mmBJiTGJf/qdsjP35Ic4gX1XnBG/0B2NzQslEsCQApuXOnrtX5njotcds4Q='} 2026-02-23T11:09:33.632 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2026-02-23T11:09:33.643 INFO:teuthology.task.internal:roles: ubuntu@trial030.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'osd.0', 'osd.1', 'osd.2', 'client.0'] 2026-02-23T11:09:33.643 INFO:teuthology.run_tasks:Running task console_log... 2026-02-23T11:09:33.721 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7781fa036200>, signals=[15]) 2026-02-23T11:09:33.721 INFO:teuthology.run_tasks:Running task internal.connect... 2026-02-23T11:09:33.737 INFO:teuthology.task.internal:Opening connections... 2026-02-23T11:09:33.737 DEBUG:teuthology.task.internal:connecting to ubuntu@trial030.front.sepia.ceph.com 2026-02-23T11:09:33.738 DEBUG:teuthology.orchestra.connection:{'hostname': 'trial030.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2026-02-23T11:09:33.817 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2026-02-23T11:09:33.829 DEBUG:teuthology.orchestra.run.trial030:> uname -m 2026-02-23T11:09:33.909 INFO:teuthology.orchestra.run.trial030.stdout:x86_64 2026-02-23T11:09:33.910 DEBUG:teuthology.orchestra.run.trial030:> cat /etc/os-release 2026-02-23T11:09:33.968 INFO:teuthology.orchestra.run.trial030.stdout:NAME="CentOS Stream" 2026-02-23T11:09:33.968 INFO:teuthology.orchestra.run.trial030.stdout:VERSION="9" 2026-02-23T11:09:33.968 INFO:teuthology.orchestra.run.trial030.stdout:ID="centos" 2026-02-23T11:09:33.968 INFO:teuthology.orchestra.run.trial030.stdout:ID_LIKE="rhel fedora" 2026-02-23T11:09:33.968 INFO:teuthology.orchestra.run.trial030.stdout:VERSION_ID="9" 2026-02-23T11:09:33.968 INFO:teuthology.orchestra.run.trial030.stdout:PLATFORM_ID="platform:el9" 2026-02-23T11:09:33.968 INFO:teuthology.orchestra.run.trial030.stdout:PRETTY_NAME="CentOS Stream 9" 2026-02-23T11:09:33.968 INFO:teuthology.orchestra.run.trial030.stdout:ANSI_COLOR="0;31" 2026-02-23T11:09:33.968 INFO:teuthology.orchestra.run.trial030.stdout:LOGO="fedora-logo-icon" 2026-02-23T11:09:33.968 INFO:teuthology.orchestra.run.trial030.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2026-02-23T11:09:33.968 INFO:teuthology.orchestra.run.trial030.stdout:HOME_URL="https://centos.org/" 2026-02-23T11:09:33.968 INFO:teuthology.orchestra.run.trial030.stdout:BUG_REPORT_URL="https://issues.redhat.com/" 2026-02-23T11:09:33.968 INFO:teuthology.orchestra.run.trial030.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2026-02-23T11:09:33.968 INFO:teuthology.orchestra.run.trial030.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2026-02-23T11:09:33.969 INFO:teuthology.lock.ops:Updating trial030.front.sepia.ceph.com on lock server 2026-02-23T11:09:34.030 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2026-02-23T11:09:34.042 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2026-02-23T11:09:34.054 INFO:teuthology.task.internal:Checking for old test directory... 2026-02-23T11:09:34.054 DEBUG:teuthology.orchestra.run.trial030:> test '!' -e /home/ubuntu/cephtest 2026-02-23T11:09:34.069 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2026-02-23T11:09:34.080 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2026-02-23T11:09:34.080 DEBUG:teuthology.orchestra.run.trial030:> test -z $(ls -A /var/lib/ceph) 2026-02-23T11:09:34.125 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2026-02-23T11:09:34.211 INFO:teuthology.run_tasks:Running task kernel... 2026-02-23T11:09:34.230 INFO:teuthology.task.kernel:normalize config orig: {'branch': 'distro', 'kdb': True, 'sha1': 'distro'} 2026-02-23T11:09:34.230 INFO:teuthology.task.kernel:config {'mon.a': {'branch': 'distro', 'kdb': True, 'sha1': 'distro'}}, timeout 300 2026-02-23T11:09:34.230 DEBUG:teuthology.orchestra.run.trial030:> test -f /run/.containerenv -o -f /.dockerenv 2026-02-23T11:09:34.244 DEBUG:teuthology.orchestra.run:got remote process result: 1 2026-02-23T11:09:34.245 DEBUG:teuthology.orchestra.run.trial030:> uname -r 2026-02-23T11:09:34.301 INFO:teuthology.orchestra.run.trial030.stdout:5.14.0-681.el9.x86_64 2026-02-23T11:09:34.301 INFO:teuthology.task.kernel:Running kernel on trial030: 5.14.0-681.el9.x86_64 2026-02-23T11:09:34.301 DEBUG:teuthology.orchestra.run.trial030:> sudo yum install -y kernel 2026-02-23T11:09:35.720 INFO:teuthology.orchestra.run.trial030.stdout:CentOS Stream 9 - BaseOS 8.7 MB/s | 8.9 MB 00:01 2026-02-23T11:09:40.625 INFO:teuthology.orchestra.run.trial030.stdout:CentOS Stream 9 - AppStream 6.1 MB/s | 27 MB 00:04 2026-02-23T11:09:43.316 INFO:teuthology.orchestra.run.trial030.stdout:CentOS Stream 9 - CRB 23 MB/s | 7.8 MB 00:00 2026-02-23T11:09:44.222 INFO:teuthology.orchestra.run.trial030.stdout:CentOS Stream 9 - Extras packages 104 kB/s | 20 kB 00:00 2026-02-23T11:09:44.648 INFO:teuthology.orchestra.run.trial030.stdout:Extra Packages for Enterprise Linux 57 MB/s | 20 MB 00:00 2026-02-23T11:09:47.820 INFO:teuthology.orchestra.run.trial030.stdout:lab-extras 2.7 MB/s | 50 kB 00:00 2026-02-23T11:09:48.827 INFO:teuthology.orchestra.run.trial030.stdout:Package kernel-5.14.0-645.el9.x86_64 is already installed. 2026-02-23T11:09:48.828 INFO:teuthology.orchestra.run.trial030.stdout:Package kernel-5.14.0-681.el9.x86_64 is already installed. 2026-02-23T11:09:48.846 INFO:teuthology.orchestra.run.trial030.stdout:Dependencies resolved. 2026-02-23T11:09:48.847 INFO:teuthology.orchestra.run.trial030.stdout:Nothing to do. 2026-02-23T11:09:48.847 INFO:teuthology.orchestra.run.trial030.stdout:Complete! 2026-02-23T11:09:48.870 DEBUG:teuthology.orchestra.run.trial030:> echo no | sudo yum reinstall kernel || true 2026-02-23T11:09:49.170 INFO:teuthology.orchestra.run.trial030.stdout:Last metadata expiration check: 0:00:02 ago on Mon 23 Feb 2026 11:09:47 AM UTC. 2026-02-23T11:09:49.250 INFO:teuthology.orchestra.run.trial030.stdout:Dependencies resolved. 2026-02-23T11:09:49.251 INFO:teuthology.orchestra.run.trial030.stdout:================================================================================ 2026-02-23T11:09:49.251 INFO:teuthology.orchestra.run.trial030.stdout: Package Architecture Version Repository Size 2026-02-23T11:09:49.251 INFO:teuthology.orchestra.run.trial030.stdout:================================================================================ 2026-02-23T11:09:49.251 INFO:teuthology.orchestra.run.trial030.stdout:Reinstalling: 2026-02-23T11:09:49.251 INFO:teuthology.orchestra.run.trial030.stdout: kernel x86_64 5.14.0-681.el9 baseos 803 k 2026-02-23T11:09:49.251 INFO:teuthology.orchestra.run.trial030.stdout: 2026-02-23T11:09:49.251 INFO:teuthology.orchestra.run.trial030.stdout:Transaction Summary 2026-02-23T11:09:49.251 INFO:teuthology.orchestra.run.trial030.stdout:================================================================================ 2026-02-23T11:09:49.251 INFO:teuthology.orchestra.run.trial030.stdout: 2026-02-23T11:09:49.251 INFO:teuthology.orchestra.run.trial030.stdout:Total download size: 803 k 2026-02-23T11:09:49.252 INFO:teuthology.orchestra.run.trial030.stdout:Installed size: 0 2026-02-23T11:09:49.252 INFO:teuthology.orchestra.run.trial030.stderr:Operation aborted. 2026-02-23T11:09:49.271 INFO:teuthology.orchestra.run.trial030.stdout:Is this ok [y/N]: 2026-02-23T11:09:49.271 DEBUG:teuthology.orchestra.run.trial030:> sudo yum reinstall -y kernel || true 2026-02-23T11:09:49.570 INFO:teuthology.orchestra.run.trial030.stdout:Last metadata expiration check: 0:00:02 ago on Mon 23 Feb 2026 11:09:47 AM UTC. 2026-02-23T11:09:49.651 INFO:teuthology.orchestra.run.trial030.stdout:Dependencies resolved. 2026-02-23T11:09:49.651 INFO:teuthology.orchestra.run.trial030.stdout:================================================================================ 2026-02-23T11:09:49.651 INFO:teuthology.orchestra.run.trial030.stdout: Package Architecture Version Repository Size 2026-02-23T11:09:49.651 INFO:teuthology.orchestra.run.trial030.stdout:================================================================================ 2026-02-23T11:09:49.651 INFO:teuthology.orchestra.run.trial030.stdout:Reinstalling: 2026-02-23T11:09:49.651 INFO:teuthology.orchestra.run.trial030.stdout: kernel x86_64 5.14.0-681.el9 baseos 803 k 2026-02-23T11:09:49.651 INFO:teuthology.orchestra.run.trial030.stdout: 2026-02-23T11:09:49.651 INFO:teuthology.orchestra.run.trial030.stdout:Transaction Summary 2026-02-23T11:09:49.651 INFO:teuthology.orchestra.run.trial030.stdout:================================================================================ 2026-02-23T11:09:49.651 INFO:teuthology.orchestra.run.trial030.stdout: 2026-02-23T11:09:49.651 INFO:teuthology.orchestra.run.trial030.stdout:Total download size: 803 k 2026-02-23T11:09:49.651 INFO:teuthology.orchestra.run.trial030.stdout:Installed size: 0 2026-02-23T11:09:49.652 INFO:teuthology.orchestra.run.trial030.stdout:Downloading Packages: 2026-02-23T11:09:49.907 INFO:teuthology.orchestra.run.trial030.stdout:kernel-5.14.0-681.el9.x86_64.rpm 4.0 MB/s | 803 kB 00:00 2026-02-23T11:09:49.907 INFO:teuthology.orchestra.run.trial030.stdout:-------------------------------------------------------------------------------- 2026-02-23T11:09:49.907 INFO:teuthology.orchestra.run.trial030.stdout:Total 3.1 MB/s | 803 kB 00:00 2026-02-23T11:09:49.917 INFO:teuthology.orchestra.run.trial030.stdout:Running transaction check 2026-02-23T11:09:49.950 INFO:teuthology.orchestra.run.trial030.stdout:Transaction check succeeded. 2026-02-23T11:09:49.951 INFO:teuthology.orchestra.run.trial030.stdout:Running transaction test 2026-02-23T11:09:49.955 INFO:teuthology.orchestra.run.trial030.stdout:Transaction test succeeded. 2026-02-23T11:09:49.955 INFO:teuthology.orchestra.run.trial030.stdout:Running transaction 2026-02-23T11:09:49.979 INFO:teuthology.orchestra.run.trial030.stdout: Preparing : 1/1 2026-02-23T11:09:49.983 INFO:teuthology.orchestra.run.trial030.stdout: Reinstalling : kernel-5.14.0-681.el9.x86_64 1/2 2026-02-23T11:09:49.983 INFO:teuthology.orchestra.run.trial030.stdout: Cleanup : kernel-5.14.0-681.el9.x86_64 2/2 2026-02-23T11:09:50.091 INFO:teuthology.orchestra.run.trial030.stdout: Running scriptlet: kernel-5.14.0-681.el9.x86_64 2/2 2026-02-23T11:09:50.091 INFO:teuthology.orchestra.run.trial030.stdout: Verifying : kernel-5.14.0-681.el9.x86_64 1/2 2026-02-23T11:09:50.148 INFO:teuthology.orchestra.run.trial030.stdout: Verifying : kernel-5.14.0-681.el9.x86_64 2/2 2026-02-23T11:09:50.148 INFO:teuthology.orchestra.run.trial030.stdout: 2026-02-23T11:09:50.148 INFO:teuthology.orchestra.run.trial030.stdout:Reinstalled: 2026-02-23T11:09:50.148 INFO:teuthology.orchestra.run.trial030.stdout: kernel-5.14.0-681.el9.x86_64 2026-02-23T11:09:50.148 INFO:teuthology.orchestra.run.trial030.stdout: 2026-02-23T11:09:50.149 INFO:teuthology.orchestra.run.trial030.stdout:Complete! 2026-02-23T11:09:50.174 DEBUG:teuthology.orchestra.run.trial030:> rpm -q kernel | sort -rV | head -n 1 2026-02-23T11:09:50.197 INFO:teuthology.orchestra.run.trial030.stdout:kernel-5.14.0-681.el9.x86_64 2026-02-23T11:09:50.198 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-681.el9.x86_64 2026-02-23T11:09:50.198 INFO:teuthology.task.kernel:Newest distro kernel installed and running 2026-02-23T11:09:50.198 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2026-02-23T11:09:50.198 DEBUG:teuthology.orchestra.run.trial030:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2026-02-23T11:09:50.262 INFO:teuthology.orchestra.run.trial030.stdout:ttyS1 2026-02-23T11:09:50.285 DEBUG:teuthology.parallel:result is None 2026-02-23T11:09:50.285 INFO:teuthology.run_tasks:Running task internal.base... 2026-02-23T11:09:50.298 INFO:teuthology.task.internal:Creating test directory... 2026-02-23T11:09:50.298 DEBUG:teuthology.orchestra.run.trial030:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2026-02-23T11:09:50.341 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2026-02-23T11:09:50.354 INFO:teuthology.run_tasks:Running task internal.archive... 2026-02-23T11:09:50.367 INFO:teuthology.task.internal:Creating archive directory... 2026-02-23T11:09:50.367 DEBUG:teuthology.orchestra.run.trial030:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2026-02-23T11:09:50.403 INFO:teuthology.run_tasks:Running task internal.coredump... 2026-02-23T11:09:50.415 INFO:teuthology.task.internal:Enabling coredump saving... 2026-02-23T11:09:50.415 DEBUG:teuthology.orchestra.run.trial030:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2026-02-23T11:09:50.462 INFO:teuthology.orchestra.run.trial030.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-23T11:09:50.471 INFO:teuthology.orchestra.run.trial030.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-23T11:09:50.473 INFO:teuthology.run_tasks:Running task internal.sudo... 2026-02-23T11:09:50.491 INFO:teuthology.task.internal:Configuring sudo... 2026-02-23T11:09:50.491 DEBUG:teuthology.orchestra.run.trial030:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2026-02-23T11:09:50.538 INFO:teuthology.run_tasks:Running task internal.syslog... 2026-02-23T11:09:50.554 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2026-02-23T11:09:50.555 DEBUG:teuthology.orchestra.run.trial030:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2026-02-23T11:09:50.594 DEBUG:teuthology.orchestra.run.trial030:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2026-02-23T11:09:50.711 DEBUG:teuthology.orchestra.run.trial030:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2026-02-23T11:09:50.734 DEBUG:teuthology.orchestra.run.trial030:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2026-02-23T11:09:50.858 DEBUG:teuthology.orchestra.run.trial030:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2026-02-23T11:09:50.881 DEBUG:teuthology.orchestra.run.trial030:> set -ex 2026-02-23T11:09:50.881 DEBUG:teuthology.orchestra.run.trial030:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2026-02-23T11:09:50.945 DEBUG:teuthology.orchestra.run.trial030:> sudo service rsyslog restart 2026-02-23T11:09:51.013 INFO:teuthology.orchestra.run.trial030.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2026-02-23T11:09:51.528 INFO:teuthology.run_tasks:Running task internal.timer... 2026-02-23T11:09:51.542 INFO:teuthology.task.internal:Starting timer... 2026-02-23T11:09:51.542 INFO:teuthology.run_tasks:Running task pcp... 2026-02-23T11:09:51.563 INFO:teuthology.run_tasks:Running task selinux... 2026-02-23T11:09:51.582 DEBUG:teuthology.orchestra.run.trial030:> sudo service auditd rotate 2026-02-23T11:09:51.611 INFO:teuthology.orchestra.run.trial030.stdout:Rotating logs: 2026-02-23T11:09:51.612 DEBUG:teuthology.task.selinux:Getting current SELinux state 2026-02-23T11:09:51.612 DEBUG:teuthology.orchestra.run.trial030:> /usr/sbin/getenforce 2026-02-23T11:09:51.669 INFO:teuthology.orchestra.run.trial030.stdout:Permissive 2026-02-23T11:09:51.669 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@trial030.front.sepia.ceph.com': 'permissive'} 2026-02-23T11:09:51.670 DEBUG:teuthology.orchestra.run.trial030:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 2026-02-23T11:09:51.733 DEBUG:teuthology.orchestra.run:got remote process result: 1 2026-02-23T11:09:51.734 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2026-02-23T11:09:51.734 DEBUG:teuthology.orchestra.run.trial030:> sudo /usr/sbin/setenforce permissive 2026-02-23T11:09:51.798 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2026-02-23T11:09:51.811 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2026-02-23T11:09:51.825 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2026-02-23T11:09:51.827 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit trial030.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2026-02-23T11:12:32.310 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@trial030.front.sepia.ceph.com')] 2026-02-23T11:12:32.311 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@trial030.front.sepia.ceph.com' 2026-02-23T11:12:32.312 DEBUG:teuthology.orchestra.connection:{'hostname': 'trial030.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2026-02-23T11:12:32.422 DEBUG:teuthology.orchestra.run.trial030:> true 2026-02-23T11:12:32.503 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@trial030.front.sepia.ceph.com' 2026-02-23T11:12:32.503 INFO:teuthology.run_tasks:Running task clock... 2026-02-23T11:12:32.518 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2026-02-23T11:12:32.518 INFO:teuthology.orchestra.run:Running command with timeout 360 2026-02-23T11:12:32.519 DEBUG:teuthology.orchestra.run.trial030:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2026-02-23T11:12:32.570 INFO:teuthology.orchestra.run.trial030.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2026-02-23T11:12:32.582 INFO:teuthology.orchestra.run.trial030.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2026-02-23T11:12:32.603 INFO:teuthology.orchestra.run.trial030.stderr:sudo: ntpd: command not found 2026-02-23T11:12:32.612 INFO:teuthology.orchestra.run.trial030.stdout:506 Cannot talk to daemon 2026-02-23T11:12:32.624 INFO:teuthology.orchestra.run.trial030.stderr:Failed to start ntp.service: Unit ntp.service not found. 2026-02-23T11:12:32.635 INFO:teuthology.orchestra.run.trial030.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2026-02-23T11:12:32.681 INFO:teuthology.orchestra.run.trial030.stderr:bash: line 1: ntpq: command not found 2026-02-23T11:12:32.960 INFO:teuthology.orchestra.run.trial030.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2026-02-23T11:12:32.960 INFO:teuthology.orchestra.run.trial030.stdout:=============================================================================== 2026-02-23T11:12:32.960 INFO:teuthology.orchestra.run.trial030.stdout:^? ntp2.zeus.frumentum.media 0 6 0 - +0ns[ +0ns] +/- 0ns 2026-02-23T11:12:32.960 INFO:teuthology.orchestra.run.trial030.stdout:^? nyc2.us.ntp.li 0 6 0 - +0ns[ +0ns] +/- 0ns 2026-02-23T11:12:32.960 INFO:teuthology.orchestra.run.trial030.stdout:^? ntp6.kernfusion.at 2 6 1 0 +9673us[+9673us] +/- 47ms 2026-02-23T11:12:32.961 INFO:teuthology.orchestra.run.trial030.stdout:^? s2-a.time.mci1.us.rozint> 0 6 0 - +0ns[ +0ns] +/- 0ns 2026-02-23T11:12:32.962 INFO:teuthology.run_tasks:Running task install... 2026-02-23T11:12:32.979 DEBUG:teuthology.task.install:project ceph 2026-02-23T11:12:32.980 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'extra_system_packages': ['python3-pytest'], 'flavor': 'debug', 'sha1': 'e371433fcd1390a0b82cfb18effc422c1f5d8c1d'}} 2026-02-23T11:12:32.980 DEBUG:teuthology.task.install:config {'extra_packages': {'deb': ['ceph-osd-crimson', 'ceph-osd-crimson-dbg'], 'rpm': ['ceph-osd-crimson', 'ceph-osd-crimson-debuginfo']}, 'extra_system_packages': ['python3-pytest'], 'flavor': 'debug', 'sha1': 'e371433fcd1390a0b82cfb18effc422c1f5d8c1d'} 2026-02-23T11:12:32.980 INFO:teuthology.task.install:Using flavor: debug 2026-02-23T11:12:32.988 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-osd', 'ceph-osd-classic', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-osd', 'ceph-osd-classic', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2026-02-23T11:12:32.989 INFO:teuthology.task.install:extra packages: {'deb': ['ceph-osd-crimson', 'ceph-osd-crimson-dbg'], 'rpm': ['ceph-osd-crimson', 'ceph-osd-crimson-debuginfo']} 2026-02-23T11:12:32.989 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['ceph-osd-crimson', 'ceph-osd-crimson-dbg'], 'rpm': ['ceph-osd-crimson', 'ceph-osd-crimson-debuginfo']}, 'extra_system_packages': ['python3-pytest'], 'extras': None, 'enable_coprs': [], 'flavor': 'debug', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': 'e371433fcd1390a0b82cfb18effc422c1f5d8c1d', 'tag': None, 'wait_for_package': False} 2026-02-23T11:12:32.990 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=debug&distros=centos%2F9%2Fx86_64&sha1=e371433fcd1390a0b82cfb18effc422c1f5d8c1d 2026-02-23T11:12:33.160 INFO:teuthology.task.install.rpm:Pulling from https://3.chacra.ceph.com/r/ceph/wip-shraddhaag-enable-cephadm-tests/e371433fcd1390a0b82cfb18effc422c1f5d8c1d/centos/9/flavors/debug/ 2026-02-23T11:12:33.160 INFO:teuthology.task.install.rpm:Package version is 20.3.0-5439.ge371433f 2026-02-23T11:12:33.549 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://3.chacra.ceph.com/r/ceph/wip-shraddhaag-enable-cephadm-tests/e371433fcd1390a0b82cfb18effc422c1f5d8c1d/centos/9/flavors/debug/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://3.chacra.ceph.com/r/ceph/wip-shraddhaag-enable-cephadm-tests/e371433fcd1390a0b82cfb18effc422c1f5d8c1d/centos/9/flavors/debug/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://3.chacra.ceph.com/r/ceph/wip-shraddhaag-enable-cephadm-tests/e371433fcd1390a0b82cfb18effc422c1f5d8c1d/centos/9/flavors/debug/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2026-02-23T11:12:33.549 DEBUG:teuthology.orchestra.run.trial030:> set -ex 2026-02-23T11:12:33.549 DEBUG:teuthology.orchestra.run.trial030:> sudo dd of=/etc/yum.repos.d/ceph.repo 2026-02-23T11:12:33.577 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-osd, ceph-osd-classic, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, ceph-osd-crimson, ceph-osd-crimson-debuginfo, python3-pytest on remote rpm x86_64 2026-02-23T11:12:33.577 DEBUG:teuthology.orchestra.run.trial030:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/e371433fcd1390a0b82cfb18effc422c1f5d8c1d/;g' /etc/yum.repos.d/ceph.repo ; fi 2026-02-23T11:12:33.641 DEBUG:teuthology.orchestra.run.trial030:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2026-02-23T11:12:33.716 DEBUG:teuthology.orchestra.run.trial030:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2026-02-23T11:12:33.781 INFO:teuthology.orchestra.run.trial030.stdout:check_obsoletes = 1 2026-02-23T11:12:33.783 DEBUG:teuthology.orchestra.run.trial030:> sudo yum clean all 2026-02-23T11:12:33.944 INFO:teuthology.orchestra.run.trial030.stdout:41 files removed 2026-02-23T11:12:33.959 DEBUG:teuthology.orchestra.run.trial030:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-osd ceph-osd-classic ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd ceph-osd-crimson ceph-osd-crimson-debuginfo python3-pytest 2026-02-23T11:12:34.880 INFO:teuthology.orchestra.run.trial030.stdout:ceph packages for x86_64 118 kB/s | 92 kB 00:00 2026-02-23T11:12:35.483 INFO:teuthology.orchestra.run.trial030.stdout:ceph noarch packages 41 kB/s | 24 kB 00:00 2026-02-23T11:12:36.041 INFO:teuthology.orchestra.run.trial030.stdout:ceph source packages 3.8 kB/s | 2.1 kB 00:00 2026-02-23T11:12:36.767 INFO:teuthology.orchestra.run.trial030.stdout:CentOS Stream 9 - BaseOS 13 MB/s | 8.9 MB 00:00 2026-02-23T11:12:40.007 INFO:teuthology.orchestra.run.trial030.stdout:CentOS Stream 9 - AppStream 9.9 MB/s | 27 MB 00:02 2026-02-23T11:12:44.084 INFO:teuthology.orchestra.run.trial030.stdout:CentOS Stream 9 - CRB 4.7 MB/s | 7.8 MB 00:01 2026-02-23T11:12:45.605 INFO:teuthology.orchestra.run.trial030.stdout:CentOS Stream 9 - Extras packages 25 kB/s | 20 kB 00:00 2026-02-23T11:12:46.309 INFO:teuthology.orchestra.run.trial030.stdout:Extra Packages for Enterprise Linux 32 MB/s | 20 MB 00:00 2026-02-23T11:12:49.482 INFO:teuthology.orchestra.run.trial030.stdout:lab-extras 2.0 MB/s | 50 kB 00:00 2026-02-23T11:12:50.671 INFO:teuthology.orchestra.run.trial030.stdout:Package librados2-2:16.2.4-5.el9.x86_64 is already installed. 2026-02-23T11:12:50.671 INFO:teuthology.orchestra.run.trial030.stdout:Package librbd1-2:16.2.4-5.el9.x86_64 is already installed. 2026-02-23T11:12:50.698 INFO:teuthology.orchestra.run.trial030.stdout:Dependencies resolved. 2026-02-23T11:12:50.702 INFO:teuthology.orchestra.run.trial030.stdout:======================================================================================= 2026-02-23T11:12:50.702 INFO:teuthology.orchestra.run.trial030.stdout: Package Arch Version Repository Size 2026-02-23T11:12:50.702 INFO:teuthology.orchestra.run.trial030.stdout:======================================================================================= 2026-02-23T11:12:50.702 INFO:teuthology.orchestra.run.trial030.stdout:Installing: 2026-02-23T11:12:50.702 INFO:teuthology.orchestra.run.trial030.stdout: ceph x86_64 2:20.3.0-5439.ge371433f.el9 ceph 6.5 k 2026-02-23T11:12:50.702 INFO:teuthology.orchestra.run.trial030.stdout: ceph-base x86_64 2:20.3.0-5439.ge371433f.el9 ceph 5.1 M 2026-02-23T11:12:50.702 INFO:teuthology.orchestra.run.trial030.stdout: ceph-fuse x86_64 2:20.3.0-5439.ge371433f.el9 ceph 914 k 2026-02-23T11:12:50.702 INFO:teuthology.orchestra.run.trial030.stdout: ceph-immutable-object-cache x86_64 2:20.3.0-5439.ge371433f.el9 ceph 149 k 2026-02-23T11:12:50.703 INFO:teuthology.orchestra.run.trial030.stdout: ceph-mgr x86_64 2:20.3.0-5439.ge371433f.el9 ceph 937 k 2026-02-23T11:12:50.703 INFO:teuthology.orchestra.run.trial030.stdout: ceph-mgr-cephadm noarch 2:20.3.0-5439.ge371433f.el9 ceph-noarch 184 k 2026-02-23T11:12:50.703 INFO:teuthology.orchestra.run.trial030.stdout: ceph-mgr-dashboard noarch 2:20.3.0-5439.ge371433f.el9 ceph-noarch 51 M 2026-02-23T11:12:50.703 INFO:teuthology.orchestra.run.trial030.stdout: ceph-mgr-diskprediction-local noarch 2:20.3.0-5439.ge371433f.el9 ceph-noarch 7.4 M 2026-02-23T11:12:50.703 INFO:teuthology.orchestra.run.trial030.stdout: ceph-mgr-rook noarch 2:20.3.0-5439.ge371433f.el9 ceph-noarch 50 k 2026-02-23T11:12:50.703 INFO:teuthology.orchestra.run.trial030.stdout: ceph-osd x86_64 2:20.3.0-5439.ge371433f.el9 ceph 164 k 2026-02-23T11:12:50.703 INFO:teuthology.orchestra.run.trial030.stdout: ceph-osd-classic x86_64 2:20.3.0-5439.ge371433f.el9 ceph 15 M 2026-02-23T11:12:50.703 INFO:teuthology.orchestra.run.trial030.stdout: ceph-osd-crimson x86_64 2:20.3.0-5439.ge371433f.el9 ceph 87 M 2026-02-23T11:12:50.703 INFO:teuthology.orchestra.run.trial030.stdout: ceph-osd-crimson-debuginfo x86_64 2:20.3.0-5439.ge371433f.el9 ceph 429 M 2026-02-23T11:12:50.703 INFO:teuthology.orchestra.run.trial030.stdout: ceph-radosgw x86_64 2:20.3.0-5439.ge371433f.el9 ceph 11 M 2026-02-23T11:12:50.703 INFO:teuthology.orchestra.run.trial030.stdout: ceph-test x86_64 2:20.3.0-5439.ge371433f.el9 ceph 126 M 2026-02-23T11:12:50.703 INFO:teuthology.orchestra.run.trial030.stdout: ceph-volume noarch 2:20.3.0-5439.ge371433f.el9 ceph-noarch 301 k 2026-02-23T11:12:50.704 INFO:teuthology.orchestra.run.trial030.stdout: cephadm noarch 2:20.3.0-5439.ge371433f.el9 ceph-noarch 1.0 M 2026-02-23T11:12:50.704 INFO:teuthology.orchestra.run.trial030.stdout: libcephfs-devel x86_64 2:20.3.0-5439.ge371433f.el9 ceph 37 k 2026-02-23T11:12:50.704 INFO:teuthology.orchestra.run.trial030.stdout: libcephfs2 x86_64 2:20.3.0-5439.ge371433f.el9 ceph 860 k 2026-02-23T11:12:50.704 INFO:teuthology.orchestra.run.trial030.stdout: librados-devel x86_64 2:20.3.0-5439.ge371433f.el9 ceph 124 k 2026-02-23T11:12:50.704 INFO:teuthology.orchestra.run.trial030.stdout: python3-cephfs x86_64 2:20.3.0-5439.ge371433f.el9 ceph 204 k 2026-02-23T11:12:50.704 INFO:teuthology.orchestra.run.trial030.stdout: python3-pytest noarch 6.2.2-7.el9 appstream 519 k 2026-02-23T11:12:50.704 INFO:teuthology.orchestra.run.trial030.stdout: python3-rados x86_64 2:20.3.0-5439.ge371433f.el9 ceph 324 k 2026-02-23T11:12:50.704 INFO:teuthology.orchestra.run.trial030.stdout: python3-rbd x86_64 2:20.3.0-5439.ge371433f.el9 ceph 305 k 2026-02-23T11:12:50.704 INFO:teuthology.orchestra.run.trial030.stdout: python3-rgw x86_64 2:20.3.0-5439.ge371433f.el9 ceph 100 k 2026-02-23T11:12:50.704 INFO:teuthology.orchestra.run.trial030.stdout: rbd-fuse x86_64 2:20.3.0-5439.ge371433f.el9 ceph 86 k 2026-02-23T11:12:50.704 INFO:teuthology.orchestra.run.trial030.stdout: rbd-mirror x86_64 2:20.3.0-5439.ge371433f.el9 ceph 2.6 M 2026-02-23T11:12:50.705 INFO:teuthology.orchestra.run.trial030.stdout: rbd-nbd x86_64 2:20.3.0-5439.ge371433f.el9 ceph 166 k 2026-02-23T11:12:50.705 INFO:teuthology.orchestra.run.trial030.stdout:Upgrading: 2026-02-23T11:12:50.705 INFO:teuthology.orchestra.run.trial030.stdout: librados2 x86_64 2:20.3.0-5439.ge371433f.el9 ceph 3.3 M 2026-02-23T11:12:50.705 INFO:teuthology.orchestra.run.trial030.stdout: librbd1 x86_64 2:20.3.0-5439.ge371433f.el9 ceph 2.7 M 2026-02-23T11:12:50.705 INFO:teuthology.orchestra.run.trial030.stdout:Installing dependencies: 2026-02-23T11:12:50.705 INFO:teuthology.orchestra.run.trial030.stdout: abseil-cpp x86_64 20211102.0-4.el9 epel 551 k 2026-02-23T11:12:50.705 INFO:teuthology.orchestra.run.trial030.stdout: boost-program-options x86_64 1.75.0-13.el9 appstream 104 k 2026-02-23T11:12:50.705 INFO:teuthology.orchestra.run.trial030.stdout: ceph-common x86_64 2:20.3.0-5439.ge371433f.el9 ceph 21 M 2026-02-23T11:12:50.706 INFO:teuthology.orchestra.run.trial030.stdout: ceph-grafana-dashboards noarch 2:20.3.0-5439.ge371433f.el9 ceph-noarch 57 k 2026-02-23T11:12:50.706 INFO:teuthology.orchestra.run.trial030.stdout: ceph-mds x86_64 2:20.3.0-5439.ge371433f.el9 ceph 2.0 M 2026-02-23T11:12:50.706 INFO:teuthology.orchestra.run.trial030.stdout: ceph-mgr-modules-core noarch 2:20.3.0-5439.ge371433f.el9 ceph-noarch 298 k 2026-02-23T11:12:50.706 INFO:teuthology.orchestra.run.trial030.stdout: ceph-mon x86_64 2:20.3.0-5439.ge371433f.el9 ceph 4.9 M 2026-02-23T11:12:50.706 INFO:teuthology.orchestra.run.trial030.stdout: ceph-prometheus-alerts noarch 2:20.3.0-5439.ge371433f.el9 ceph-noarch 17 k 2026-02-23T11:12:50.706 INFO:teuthology.orchestra.run.trial030.stdout: ceph-selinux x86_64 2:20.3.0-5439.ge371433f.el9 ceph 25 k 2026-02-23T11:12:50.706 INFO:teuthology.orchestra.run.trial030.stdout: cryptsetup x86_64 2.8.1-3.el9 baseos 351 k 2026-02-23T11:12:50.706 INFO:teuthology.orchestra.run.trial030.stdout: flexiblas x86_64 3.0.4-9.el9 appstream 30 k 2026-02-23T11:12:50.706 INFO:teuthology.orchestra.run.trial030.stdout: flexiblas-netlib x86_64 3.0.4-9.el9 appstream 3.0 M 2026-02-23T11:12:50.707 INFO:teuthology.orchestra.run.trial030.stdout: flexiblas-openblas-openmp x86_64 3.0.4-9.el9 appstream 15 k 2026-02-23T11:12:50.707 INFO:teuthology.orchestra.run.trial030.stdout: gperftools-libs x86_64 2.9.1-3.el9 epel 308 k 2026-02-23T11:12:50.707 INFO:teuthology.orchestra.run.trial030.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2026-02-23T11:12:50.707 INFO:teuthology.orchestra.run.trial030.stdout: hwloc-libs x86_64 2.4.1-6.el9 baseos 2.1 M 2026-02-23T11:12:50.707 INFO:teuthology.orchestra.run.trial030.stdout: ledmon-libs x86_64 1.1.0-3.el9 baseos 40 k 2026-02-23T11:12:50.707 INFO:teuthology.orchestra.run.trial030.stdout: libarrow x86_64 9.0.0-15.el9 epel 4.4 M 2026-02-23T11:12:50.707 INFO:teuthology.orchestra.run.trial030.stdout: libarrow-doc noarch 9.0.0-15.el9 epel 25 k 2026-02-23T11:12:50.707 INFO:teuthology.orchestra.run.trial030.stdout: libasan8 x86_64 15.2.1-7.el9 appstream 525 k 2026-02-23T11:12:50.708 INFO:teuthology.orchestra.run.trial030.stdout: libatomic x86_64 11.5.0-14.el9 baseos 23 k 2026-02-23T11:12:50.708 INFO:teuthology.orchestra.run.trial030.stdout: libcephfs-proxy2 x86_64 2:20.3.0-5439.ge371433f.el9 ceph 26 k 2026-02-23T11:12:50.708 INFO:teuthology.orchestra.run.trial030.stdout: libcephsqlite x86_64 2:20.3.0-5439.ge371433f.el9 ceph 153 k 2026-02-23T11:12:50.708 INFO:teuthology.orchestra.run.trial030.stdout: libconfig x86_64 1.7.2-9.el9 baseos 72 k 2026-02-23T11:12:50.708 INFO:teuthology.orchestra.run.trial030.stdout: libgfortran x86_64 11.5.0-14.el9 baseos 794 k 2026-02-23T11:12:50.708 INFO:teuthology.orchestra.run.trial030.stdout: libnbd x86_64 1.20.3-4.el9 appstream 164 k 2026-02-23T11:12:50.708 INFO:teuthology.orchestra.run.trial030.stdout: liboath x86_64 2.6.12-1.el9 epel 49 k 2026-02-23T11:12:50.708 INFO:teuthology.orchestra.run.trial030.stdout: libpmemobj x86_64 1.12.1-1.el9 appstream 160 k 2026-02-23T11:12:50.708 INFO:teuthology.orchestra.run.trial030.stdout: libquadmath x86_64 11.5.0-14.el9 baseos 184 k 2026-02-23T11:12:50.709 INFO:teuthology.orchestra.run.trial030.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2026-02-23T11:12:50.709 INFO:teuthology.orchestra.run.trial030.stdout: libradosstriper1 x86_64 2:20.3.0-5439.ge371433f.el9 ceph 263 k 2026-02-23T11:12:50.709 INFO:teuthology.orchestra.run.trial030.stdout: librdkafka x86_64 1.6.1-102.el9 appstream 662 k 2026-02-23T11:12:50.709 INFO:teuthology.orchestra.run.trial030.stdout: librgw2 x86_64 2:20.3.0-5439.ge371433f.el9 ceph 5.6 M 2026-02-23T11:12:50.709 INFO:teuthology.orchestra.run.trial030.stdout: libstoragemgmt x86_64 1.10.1-1.el9 appstream 246 k 2026-02-23T11:12:50.709 INFO:teuthology.orchestra.run.trial030.stdout: libubsan x86_64 11.5.0-14.el9 appstream 176 k 2026-02-23T11:12:50.709 INFO:teuthology.orchestra.run.trial030.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2026-02-23T11:12:50.709 INFO:teuthology.orchestra.run.trial030.stdout: libxslt x86_64 1.1.34-12.el9 appstream 233 k 2026-02-23T11:12:50.710 INFO:teuthology.orchestra.run.trial030.stdout: lksctp-tools x86_64 1.0.19-2.el9 baseos 94 k 2026-02-23T11:12:50.710 INFO:teuthology.orchestra.run.trial030.stdout: lttng-ust x86_64 2.12.0-6.el9 appstream 292 k 2026-02-23T11:12:50.710 INFO:teuthology.orchestra.run.trial030.stdout: lua x86_64 5.4.4-4.el9 appstream 188 k 2026-02-23T11:12:50.710 INFO:teuthology.orchestra.run.trial030.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2026-02-23T11:12:50.710 INFO:teuthology.orchestra.run.trial030.stdout: luarocks noarch 3.9.2-5.el9 epel 151 k 2026-02-23T11:12:50.710 INFO:teuthology.orchestra.run.trial030.stdout: mailcap noarch 2.1.49-5.el9 baseos 33 k 2026-02-23T11:12:50.710 INFO:teuthology.orchestra.run.trial030.stdout: openblas x86_64 0.3.29-1.el9 appstream 42 k 2026-02-23T11:12:50.710 INFO:teuthology.orchestra.run.trial030.stdout: openblas-openmp x86_64 0.3.29-1.el9 appstream 5.3 M 2026-02-23T11:12:50.710 INFO:teuthology.orchestra.run.trial030.stdout: parquet-libs x86_64 9.0.0-15.el9 epel 838 k 2026-02-23T11:12:50.711 INFO:teuthology.orchestra.run.trial030.stdout: pciutils x86_64 3.7.0-7.el9 baseos 93 k 2026-02-23T11:12:50.711 INFO:teuthology.orchestra.run.trial030.stdout: protobuf x86_64 3.14.0-17.el9 appstream 1.0 M 2026-02-23T11:12:50.711 INFO:teuthology.orchestra.run.trial030.stdout: protobuf-compiler x86_64 3.14.0-17.el9 crb 862 k 2026-02-23T11:12:50.711 INFO:teuthology.orchestra.run.trial030.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2026-02-23T11:12:50.711 INFO:teuthology.orchestra.run.trial030.stdout: python3-autocommand noarch 2.2.2-8.el9 epel 29 k 2026-02-23T11:12:50.711 INFO:teuthology.orchestra.run.trial030.stdout: python3-babel noarch 2.9.1-2.el9 appstream 6.0 M 2026-02-23T11:12:50.711 INFO:teuthology.orchestra.run.trial030.stdout: python3-backports-tarfile noarch 1.2.0-1.el9 epel 60 k 2026-02-23T11:12:50.711 INFO:teuthology.orchestra.run.trial030.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2026-02-23T11:12:50.712 INFO:teuthology.orchestra.run.trial030.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2026-02-23T11:12:50.712 INFO:teuthology.orchestra.run.trial030.stdout: python3-ceph-argparse x86_64 2:20.3.0-5439.ge371433f.el9 ceph 45 k 2026-02-23T11:12:50.712 INFO:teuthology.orchestra.run.trial030.stdout: python3-ceph-common x86_64 2:20.3.0-5439.ge371433f.el9 ceph 186 k 2026-02-23T11:12:50.712 INFO:teuthology.orchestra.run.trial030.stdout: python3-certifi noarch 2023.05.07-4.el9 epel 14 k 2026-02-23T11:12:50.712 INFO:teuthology.orchestra.run.trial030.stdout: python3-cffi x86_64 1.14.5-5.el9 baseos 253 k 2026-02-23T11:12:50.712 INFO:teuthology.orchestra.run.trial030.stdout: python3-cheroot noarch 10.0.1-4.el9 epel 173 k 2026-02-23T11:12:50.712 INFO:teuthology.orchestra.run.trial030.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2026-02-23T11:12:50.712 INFO:teuthology.orchestra.run.trial030.stdout: python3-cryptography x86_64 36.0.1-5.el9 baseos 1.2 M 2026-02-23T11:12:50.713 INFO:teuthology.orchestra.run.trial030.stdout: python3-devel x86_64 3.9.25-3.el9 appstream 244 k 2026-02-23T11:12:50.713 INFO:teuthology.orchestra.run.trial030.stdout: python3-google-auth noarch 1:2.45.0-1.el9 epel 254 k 2026-02-23T11:12:50.713 INFO:teuthology.orchestra.run.trial030.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2026-02-23T11:12:50.713 INFO:teuthology.orchestra.run.trial030.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2026-02-23T11:12:50.713 INFO:teuthology.orchestra.run.trial030.stdout: python3-iniconfig noarch 1.1.1-7.el9 appstream 17 k 2026-02-23T11:12:50.713 INFO:teuthology.orchestra.run.trial030.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2026-02-23T11:12:50.713 INFO:teuthology.orchestra.run.trial030.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2026-02-23T11:12:50.713 INFO:teuthology.orchestra.run.trial030.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2026-02-23T11:12:50.714 INFO:teuthology.orchestra.run.trial030.stdout: python3-jaraco-context noarch 6.0.1-3.el9 epel 20 k 2026-02-23T11:12:50.714 INFO:teuthology.orchestra.run.trial030.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2026-02-23T11:12:50.714 INFO:teuthology.orchestra.run.trial030.stdout: python3-jaraco-text noarch 4.0.0-2.el9 epel 26 k 2026-02-23T11:12:50.714 INFO:teuthology.orchestra.run.trial030.stdout: python3-jinja2 noarch 2.11.3-8.el9 appstream 249 k 2026-02-23T11:12:50.714 INFO:teuthology.orchestra.run.trial030.stdout: python3-jmespath noarch 1.0.1-1.el9 appstream 48 k 2026-02-23T11:12:50.714 INFO:teuthology.orchestra.run.trial030.stdout: python3-kubernetes noarch 1:26.1.0-3.el9 epel 1.0 M 2026-02-23T11:12:50.714 INFO:teuthology.orchestra.run.trial030.stdout: python3-libstoragemgmt x86_64 1.10.1-1.el9 appstream 177 k 2026-02-23T11:12:50.714 INFO:teuthology.orchestra.run.trial030.stdout: python3-markupsafe x86_64 1.1.1-12.el9 appstream 35 k 2026-02-23T11:12:50.715 INFO:teuthology.orchestra.run.trial030.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2026-02-23T11:12:50.715 INFO:teuthology.orchestra.run.trial030.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2026-02-23T11:12:50.715 INFO:teuthology.orchestra.run.trial030.stdout: python3-numpy x86_64 1:1.23.5-2.el9 appstream 6.1 M 2026-02-23T11:12:50.715 INFO:teuthology.orchestra.run.trial030.stdout: python3-numpy-f2py x86_64 1:1.23.5-2.el9 appstream 442 k 2026-02-23T11:12:50.715 INFO:teuthology.orchestra.run.trial030.stdout: python3-packaging noarch 20.9-5.el9 appstream 77 k 2026-02-23T11:12:50.715 INFO:teuthology.orchestra.run.trial030.stdout: python3-pluggy noarch 0.13.1-7.el9 appstream 41 k 2026-02-23T11:12:50.715 INFO:teuthology.orchestra.run.trial030.stdout: python3-ply noarch 3.11-14.el9 baseos 106 k 2026-02-23T11:12:50.715 INFO:teuthology.orchestra.run.trial030.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2026-02-23T11:12:50.715 INFO:teuthology.orchestra.run.trial030.stdout: python3-protobuf noarch 3.14.0-17.el9 appstream 267 k 2026-02-23T11:12:50.716 INFO:teuthology.orchestra.run.trial030.stdout: python3-py noarch 1.10.0-6.el9 appstream 477 k 2026-02-23T11:12:50.716 INFO:teuthology.orchestra.run.trial030.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2026-02-23T11:12:50.716 INFO:teuthology.orchestra.run.trial030.stdout: python3-pyasn1 noarch 0.4.8-6.el9 appstream 159 k 2026-02-23T11:12:50.716 INFO:teuthology.orchestra.run.trial030.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 appstream 279 k 2026-02-23T11:12:50.716 INFO:teuthology.orchestra.run.trial030.stdout: python3-pycparser noarch 2.20-6.el9 baseos 135 k 2026-02-23T11:12:50.716 INFO:teuthology.orchestra.run.trial030.stdout: python3-pyparsing noarch 2.4.7-9.el9 baseos 150 k 2026-02-23T11:12:50.716 INFO:teuthology.orchestra.run.trial030.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2026-02-23T11:12:50.716 INFO:teuthology.orchestra.run.trial030.stdout: python3-requests noarch 2.25.1-10.el9 baseos 126 k 2026-02-23T11:12:50.716 INFO:teuthology.orchestra.run.trial030.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 appstream 54 k 2026-02-23T11:12:50.717 INFO:teuthology.orchestra.run.trial030.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2026-02-23T11:12:50.717 INFO:teuthology.orchestra.run.trial030.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2026-02-23T11:12:50.717 INFO:teuthology.orchestra.run.trial030.stdout: python3-scipy x86_64 1.9.3-2.el9 appstream 19 M 2026-02-23T11:12:50.717 INFO:teuthology.orchestra.run.trial030.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2026-02-23T11:12:50.717 INFO:teuthology.orchestra.run.trial030.stdout: python3-toml noarch 0.10.2-6.el9 appstream 42 k 2026-02-23T11:12:50.717 INFO:teuthology.orchestra.run.trial030.stdout: python3-typing-extensions noarch 4.15.0-1.el9 epel 86 k 2026-02-23T11:12:50.717 INFO:teuthology.orchestra.run.trial030.stdout: python3-urllib3 noarch 1.26.5-7.el9 baseos 218 k 2026-02-23T11:12:50.717 INFO:teuthology.orchestra.run.trial030.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2026-02-23T11:12:50.718 INFO:teuthology.orchestra.run.trial030.stdout: python3-xmltodict noarch 0.12.0-15.el9 epel 22 k 2026-02-23T11:12:50.718 INFO:teuthology.orchestra.run.trial030.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2026-02-23T11:12:50.718 INFO:teuthology.orchestra.run.trial030.stdout: qatlib x86_64 25.08.0-2.el9 appstream 240 k 2026-02-23T11:12:50.718 INFO:teuthology.orchestra.run.trial030.stdout: qatzip-libs x86_64 1.3.1-1.el9 appstream 66 k 2026-02-23T11:12:50.718 INFO:teuthology.orchestra.run.trial030.stdout: re2 x86_64 1:20211101-20.el9 epel 191 k 2026-02-23T11:12:50.718 INFO:teuthology.orchestra.run.trial030.stdout: socat x86_64 1.7.4.1-8.el9 appstream 303 k 2026-02-23T11:12:50.718 INFO:teuthology.orchestra.run.trial030.stdout: thrift x86_64 0.15.0-4.el9 epel 1.6 M 2026-02-23T11:12:50.718 INFO:teuthology.orchestra.run.trial030.stdout: unzip x86_64 6.0-59.el9 baseos 182 k 2026-02-23T11:12:50.719 INFO:teuthology.orchestra.run.trial030.stdout: xmlstarlet x86_64 1.6.1-20.el9 appstream 64 k 2026-02-23T11:12:50.719 INFO:teuthology.orchestra.run.trial030.stdout: yaml-cpp x86_64 0.6.3-5.el9 epel 122 k 2026-02-23T11:12:50.719 INFO:teuthology.orchestra.run.trial030.stdout: zip x86_64 3.0-35.el9 baseos 266 k 2026-02-23T11:12:50.719 INFO:teuthology.orchestra.run.trial030.stdout:Installing weak dependencies: 2026-02-23T11:12:50.719 INFO:teuthology.orchestra.run.trial030.stdout: qatlib-service x86_64 25.08.0-2.el9 appstream 37 k 2026-02-23T11:12:50.719 INFO:teuthology.orchestra.run.trial030.stdout: 2026-02-23T11:12:50.719 INFO:teuthology.orchestra.run.trial030.stdout:Transaction Summary 2026-02-23T11:12:50.719 INFO:teuthology.orchestra.run.trial030.stdout:======================================================================================= 2026-02-23T11:12:50.720 INFO:teuthology.orchestra.run.trial030.stdout:Install 144 Packages 2026-02-23T11:12:50.720 INFO:teuthology.orchestra.run.trial030.stdout:Upgrade 2 Packages 2026-02-23T11:12:50.720 INFO:teuthology.orchestra.run.trial030.stdout: 2026-02-23T11:12:50.721 INFO:teuthology.orchestra.run.trial030.stdout:Total download size: 848 M 2026-02-23T11:12:50.721 INFO:teuthology.orchestra.run.trial030.stdout:Downloading Packages: 2026-02-23T11:12:51.816 INFO:teuthology.orchestra.run.trial030.stdout:(1/146): ceph-20.3.0-5439.ge371433f.el9.x86_64. 23 kB/s | 6.5 kB 00:00 2026-02-23T11:12:52.438 INFO:teuthology.orchestra.run.trial030.stdout:(2/146): ceph-fuse-20.3.0-5439.ge371433f.el9.x8 1.4 MB/s | 914 kB 00:00 2026-02-23T11:12:52.508 INFO:teuthology.orchestra.run.trial030.stdout:(3/146): ceph-immutable-object-cache-20.3.0-543 2.1 MB/s | 149 kB 00:00 2026-02-23T11:12:53.663 INFO:teuthology.orchestra.run.trial030.stdout:(4/146): ceph-mds-20.3.0-5439.ge371433f.el9.x86 1.8 MB/s | 2.0 MB 00:01 2026-02-23T11:12:54.188 INFO:teuthology.orchestra.run.trial030.stdout:(5/146): ceph-mgr-20.3.0-5439.ge371433f.el9.x86 1.7 MB/s | 937 kB 00:00 2026-02-23T11:12:55.985 INFO:teuthology.orchestra.run.trial030.stdout:(6/146): ceph-base-20.3.0-5439.ge371433f.el9.x8 1.1 MB/s | 5.1 MB 00:04 2026-02-23T11:12:56.116 INFO:teuthology.orchestra.run.trial030.stdout:(7/146): ceph-osd-20.3.0-5439.ge371433f.el9.x86 1.2 MB/s | 164 kB 00:00 2026-02-23T11:12:57.821 INFO:teuthology.orchestra.run.trial030.stdout:(8/146): ceph-mon-20.3.0-5439.ge371433f.el9.x86 1.3 MB/s | 4.9 MB 00:03 2026-02-23T11:13:01.228 INFO:teuthology.orchestra.run.trial030.stdout:(9/146): ceph-common-20.3.0-5439.ge371433f.el9. 2.1 MB/s | 21 MB 00:09 2026-02-23T11:13:03.807 INFO:teuthology.orchestra.run.trial030.stdout:(10/146): ceph-osd-classic-20.3.0-5439.ge371433 1.9 MB/s | 15 MB 00:07 2026-02-23T11:13:09.067 INFO:teuthology.orchestra.run.trial030.stdout:(11/146): ceph-radosgw-20.3.0-5439.ge371433f.el 2.1 MB/s | 11 MB 00:05 2026-02-23T11:13:09.132 INFO:teuthology.orchestra.run.trial030.stdout:(12/146): ceph-selinux-20.3.0-5439.ge371433f.el 392 kB/s | 25 kB 00:00 2026-02-23T11:14:12.549 INFO:teuthology.orchestra.run.trial030.stdout:(13/146): ceph-osd-crimson-20.3.0-5439.ge371433 1.2 MB/s | 87 MB 01:14 2026-02-23T11:14:12.619 INFO:teuthology.orchestra.run.trial030.stdout:(14/146): libcephfs-devel-20.3.0-5439.ge371433f 536 kB/s | 37 kB 00:00 2026-02-23T11:14:12.689 INFO:teuthology.orchestra.run.trial030.stdout:(15/146): libcephfs-proxy2-20.3.0-5439.ge371433 376 kB/s | 26 kB 00:00 2026-02-23T11:14:13.627 INFO:teuthology.orchestra.run.trial030.stdout:(16/146): libcephfs2-20.3.0-5439.ge371433f.el9. 917 kB/s | 860 kB 00:00 2026-02-23T11:14:13.905 INFO:teuthology.orchestra.run.trial030.stdout:(17/146): libcephsqlite-20.3.0-5439.ge371433f.e 551 kB/s | 153 kB 00:00 2026-02-23T11:14:14.109 INFO:teuthology.orchestra.run.trial030.stdout:(18/146): librados-devel-20.3.0-5439.ge371433f. 605 kB/s | 124 kB 00:00 2026-02-23T11:14:14.447 INFO:teuthology.orchestra.run.trial030.stdout:(19/146): libradosstriper1-20.3.0-5439.ge371433 779 kB/s | 263 kB 00:00 2026-02-23T11:14:18.986 INFO:teuthology.orchestra.run.trial030.stdout:(20/146): ceph-test-20.3.0-5439.ge371433f.el9.x 1.8 MB/s | 126 MB 01:09 2026-02-23T11:14:19.054 INFO:teuthology.orchestra.run.trial030.stdout:(21/146): python3-ceph-argparse-20.3.0-5439.ge3 667 kB/s | 45 kB 00:00 2026-02-23T11:14:19.194 INFO:teuthology.orchestra.run.trial030.stdout:(22/146): python3-ceph-common-20.3.0-5439.ge371 1.3 MB/s | 186 kB 00:00 2026-02-23T11:14:19.337 INFO:teuthology.orchestra.run.trial030.stdout:(23/146): python3-cephfs-20.3.0-5439.ge371433f. 1.4 MB/s | 204 kB 00:00 2026-02-23T11:14:19.540 INFO:teuthology.orchestra.run.trial030.stdout:(24/146): python3-rados-20.3.0-5439.ge371433f.e 1.6 MB/s | 324 kB 00:00 2026-02-23T11:14:19.749 INFO:teuthology.orchestra.run.trial030.stdout:(25/146): python3-rbd-20.3.0-5439.ge371433f.el9 1.4 MB/s | 305 kB 00:00 2026-02-23T11:14:20.084 INFO:teuthology.orchestra.run.trial030.stdout:(26/146): python3-rgw-20.3.0-5439.ge371433f.el9 299 kB/s | 100 kB 00:00 2026-02-23T11:14:20.223 INFO:teuthology.orchestra.run.trial030.stdout:(27/146): rbd-fuse-20.3.0-5439.ge371433f.el9.x8 622 kB/s | 86 kB 00:00 2026-02-23T11:14:21.894 INFO:teuthology.orchestra.run.trial030.stdout:(28/146): librgw2-20.3.0-5439.ge371433f.el9.x86 774 kB/s | 5.6 MB 00:07 2026-02-23T11:14:22.101 INFO:teuthology.orchestra.run.trial030.stdout:(29/146): rbd-nbd-20.3.0-5439.ge371433f.el9.x86 803 kB/s | 166 kB 00:00 2026-02-23T11:14:22.172 INFO:teuthology.orchestra.run.trial030.stdout:(30/146): ceph-grafana-dashboards-20.3.0-5439.g 810 kB/s | 57 kB 00:00 2026-02-23T11:14:22.450 INFO:teuthology.orchestra.run.trial030.stdout:(31/146): ceph-mgr-cephadm-20.3.0-5439.ge371433 663 kB/s | 184 kB 00:00 2026-02-23T11:14:23.428 INFO:teuthology.orchestra.run.trial030.stdout:(32/146): rbd-mirror-20.3.0-5439.ge371433f.el9. 829 kB/s | 2.6 MB 00:03 2026-02-23T11:14:31.593 INFO:teuthology.orchestra.run.trial030.stdout:(33/146): ceph-mgr-diskprediction-local-20.3.0- 929 kB/s | 7.4 MB 00:08 2026-02-23T11:14:31.865 INFO:teuthology.orchestra.run.trial030.stdout:(34/146): ceph-mgr-modules-core-20.3.0-5439.ge3 1.1 MB/s | 298 kB 00:00 2026-02-23T11:14:31.935 INFO:teuthology.orchestra.run.trial030.stdout:(35/146): ceph-mgr-rook-20.3.0-5439.ge371433f.e 719 kB/s | 50 kB 00:00 2026-02-23T11:14:32.005 INFO:teuthology.orchestra.run.trial030.stdout:(36/146): ceph-prometheus-alerts-20.3.0-5439.ge 248 kB/s | 17 kB 00:00 2026-02-23T11:14:32.410 INFO:teuthology.orchestra.run.trial030.stdout:(37/146): ceph-volume-20.3.0-5439.ge371433f.el9 746 kB/s | 301 kB 00:00 2026-02-23T11:14:33.434 INFO:teuthology.orchestra.run.trial030.stdout:(38/146): cephadm-20.3.0-5439.ge371433f.el9.noa 1.0 MB/s | 1.0 MB 00:01 2026-02-23T11:14:34.250 INFO:teuthology.orchestra.run.trial030.stdout:(39/146): cryptsetup-2.8.1-3.el9.x86_64.rpm 430 kB/s | 351 kB 00:00 2026-02-23T11:14:35.258 INFO:teuthology.orchestra.run.trial030.stdout:(40/146): hwloc-libs-2.4.1-6.el9.x86_64.rpm 2.1 MB/s | 2.1 MB 00:01 2026-02-23T11:14:35.837 INFO:teuthology.orchestra.run.trial030.stdout:(41/146): ledmon-libs-1.1.0-3.el9.x86_64.rpm 70 kB/s | 40 kB 00:00 2026-02-23T11:14:36.378 INFO:teuthology.orchestra.run.trial030.stdout:(42/146): libatomic-11.5.0-14.el9.x86_64.rpm 42 kB/s | 23 kB 00:00 2026-02-23T11:14:37.013 INFO:teuthology.orchestra.run.trial030.stdout:(43/146): libconfig-1.7.2-9.el9.x86_64.rpm 113 kB/s | 72 kB 00:00 2026-02-23T11:14:37.908 INFO:teuthology.orchestra.run.trial030.stdout:(44/146): libgfortran-11.5.0-14.el9.x86_64.rpm 887 kB/s | 794 kB 00:00 2026-02-23T11:14:38.638 INFO:teuthology.orchestra.run.trial030.stdout:(45/146): libquadmath-11.5.0-14.el9.x86_64.rpm 253 kB/s | 184 kB 00:00 2026-02-23T11:14:39.294 INFO:teuthology.orchestra.run.trial030.stdout:(46/146): lksctp-tools-1.0.19-2.el9.x86_64.rpm 143 kB/s | 94 kB 00:00 2026-02-23T11:14:39.858 INFO:teuthology.orchestra.run.trial030.stdout:(47/146): mailcap-2.1.49-5.el9.noarch.rpm 59 kB/s | 33 kB 00:00 2026-02-23T11:14:40.513 INFO:teuthology.orchestra.run.trial030.stdout:(48/146): pciutils-3.7.0-7.el9.x86_64.rpm 142 kB/s | 93 kB 00:00 2026-02-23T11:14:41.278 INFO:teuthology.orchestra.run.trial030.stdout:(49/146): python3-cffi-1.14.5-5.el9.x86_64.rpm 331 kB/s | 253 kB 00:00 2026-02-23T11:14:42.233 INFO:teuthology.orchestra.run.trial030.stdout:(50/146): python3-cryptography-36.0.1-5.el9.x86 1.3 MB/s | 1.2 MB 00:00 2026-02-23T11:14:42.908 INFO:teuthology.orchestra.run.trial030.stdout:(51/146): python3-ply-3.11-14.el9.noarch.rpm 158 kB/s | 106 kB 00:00 2026-02-23T11:14:43.611 INFO:teuthology.orchestra.run.trial030.stdout:(52/146): python3-pycparser-2.20-6.el9.noarch.r 192 kB/s | 135 kB 00:00 2026-02-23T11:14:44.323 INFO:teuthology.orchestra.run.trial030.stdout:(53/146): python3-pyparsing-2.4.7-9.el9.noarch. 211 kB/s | 150 kB 00:00 2026-02-23T11:14:45.017 INFO:teuthology.orchestra.run.trial030.stdout:(54/146): python3-requests-2.25.1-10.el9.noarch 182 kB/s | 126 kB 00:00 2026-02-23T11:14:45.768 INFO:teuthology.orchestra.run.trial030.stdout:(55/146): python3-urllib3-1.26.5-7.el9.noarch.r 290 kB/s | 218 kB 00:00 2026-02-23T11:14:46.502 INFO:teuthology.orchestra.run.trial030.stdout:(56/146): unzip-6.0-59.el9.x86_64.rpm 247 kB/s | 182 kB 00:00 2026-02-23T11:14:47.278 INFO:teuthology.orchestra.run.trial030.stdout:(57/146): zip-3.0-35.el9.x86_64.rpm 342 kB/s | 266 kB 00:00 2026-02-23T11:14:47.988 INFO:teuthology.orchestra.run.trial030.stdout:(58/146): boost-program-options-1.75.0-13.el9.x 147 kB/s | 104 kB 00:00 2026-02-23T11:14:48.146 INFO:teuthology.orchestra.run.trial030.stdout:(59/146): flexiblas-3.0.4-9.el9.x86_64.rpm 188 kB/s | 30 kB 00:00 2026-02-23T11:14:50.954 INFO:teuthology.orchestra.run.trial030.stdout:(60/146): flexiblas-netlib-3.0.4-9.el9.x86_64.r 1.1 MB/s | 3.0 MB 00:02 2026-02-23T11:14:51.111 INFO:teuthology.orchestra.run.trial030.stdout:(61/146): flexiblas-openblas-openmp-3.0.4-9.el9 94 kB/s | 15 kB 00:00 2026-02-23T11:14:51.271 INFO:teuthology.orchestra.run.trial030.stdout:(62/146): libasan8-15.2.1-7.el9.x86_64.rpm 3.2 MB/s | 525 kB 00:00 2026-02-23T11:14:51.430 INFO:teuthology.orchestra.run.trial030.stdout:(63/146): libnbd-1.20.3-4.el9.x86_64.rpm 1.0 MB/s | 164 kB 00:00 2026-02-23T11:14:51.590 INFO:teuthology.orchestra.run.trial030.stdout:(64/146): libpmemobj-1.12.1-1.el9.x86_64.rpm 1.0 MB/s | 160 kB 00:00 2026-02-23T11:14:51.748 INFO:teuthology.orchestra.run.trial030.stdout:(65/146): librabbitmq-0.11.0-7.el9.x86_64.rpm 286 kB/s | 45 kB 00:00 2026-02-23T11:14:51.909 INFO:teuthology.orchestra.run.trial030.stdout:(66/146): librdkafka-1.6.1-102.el9.x86_64.rpm 4.0 MB/s | 662 kB 00:00 2026-02-23T11:14:52.068 INFO:teuthology.orchestra.run.trial030.stdout:(67/146): libstoragemgmt-1.10.1-1.el9.x86_64.rp 1.5 MB/s | 246 kB 00:00 2026-02-23T11:14:52.226 INFO:teuthology.orchestra.run.trial030.stdout:(68/146): libubsan-11.5.0-14.el9.x86_64.rpm 1.1 MB/s | 176 kB 00:00 2026-02-23T11:14:52.385 INFO:teuthology.orchestra.run.trial030.stdout:(69/146): libxslt-1.1.34-12.el9.x86_64.rpm 1.4 MB/s | 233 kB 00:00 2026-02-23T11:14:52.544 INFO:teuthology.orchestra.run.trial030.stdout:(70/146): lttng-ust-2.12.0-6.el9.x86_64.rpm 1.8 MB/s | 292 kB 00:00 2026-02-23T11:14:52.702 INFO:teuthology.orchestra.run.trial030.stdout:(71/146): lua-5.4.4-4.el9.x86_64.rpm 1.2 MB/s | 188 kB 00:00 2026-02-23T11:14:52.861 INFO:teuthology.orchestra.run.trial030.stdout:(72/146): openblas-0.3.29-1.el9.x86_64.rpm 266 kB/s | 42 kB 00:00 2026-02-23T11:14:53.342 INFO:teuthology.orchestra.run.trial030.stdout:(73/146): openblas-openmp-0.3.29-1.el9.x86_64.r 11 MB/s | 5.3 MB 00:00 2026-02-23T11:14:53.504 INFO:teuthology.orchestra.run.trial030.stdout:(74/146): protobuf-3.14.0-17.el9.x86_64.rpm 6.2 MB/s | 1.0 MB 00:00 2026-02-23T11:14:54.064 INFO:teuthology.orchestra.run.trial030.stdout:(75/146): python3-babel-2.9.1-2.el9.noarch.rpm 11 MB/s | 6.0 MB 00:00 2026-02-23T11:14:54.223 INFO:teuthology.orchestra.run.trial030.stdout:(76/146): python3-devel-3.9.25-3.el9.x86_64.rpm 1.5 MB/s | 244 kB 00:00 2026-02-23T11:14:54.382 INFO:teuthology.orchestra.run.trial030.stdout:(77/146): python3-iniconfig-1.1.1-7.el9.noarch. 110 kB/s | 17 kB 00:00 2026-02-23T11:14:54.542 INFO:teuthology.orchestra.run.trial030.stdout:(78/146): python3-jinja2-2.11.3-8.el9.noarch.rp 1.5 MB/s | 249 kB 00:00 2026-02-23T11:14:54.701 INFO:teuthology.orchestra.run.trial030.stdout:(79/146): python3-jmespath-1.0.1-1.el9.noarch.r 300 kB/s | 48 kB 00:00 2026-02-23T11:14:54.860 INFO:teuthology.orchestra.run.trial030.stdout:(80/146): python3-libstoragemgmt-1.10.1-1.el9.x 1.1 MB/s | 177 kB 00:00 2026-02-23T11:14:55.019 INFO:teuthology.orchestra.run.trial030.stdout:(81/146): python3-markupsafe-1.1.1-12.el9.x86_6 219 kB/s | 35 kB 00:00 2026-02-23T11:14:55.422 INFO:teuthology.orchestra.run.trial030.stdout:(82/146): python3-numpy-1.23.5-2.el9.x86_64.rpm 15 MB/s | 6.1 MB 00:00 2026-02-23T11:14:55.582 INFO:teuthology.orchestra.run.trial030.stdout:(83/146): python3-numpy-f2py-1.23.5-2.el9.x86_6 2.7 MB/s | 442 kB 00:00 2026-02-23T11:14:55.741 INFO:teuthology.orchestra.run.trial030.stdout:(84/146): python3-packaging-20.9-5.el9.noarch.r 486 kB/s | 77 kB 00:00 2026-02-23T11:14:55.899 INFO:teuthology.orchestra.run.trial030.stdout:(85/146): python3-pluggy-0.13.1-7.el9.noarch.rp 262 kB/s | 41 kB 00:00 2026-02-23T11:14:56.059 INFO:teuthology.orchestra.run.trial030.stdout:(86/146): python3-protobuf-3.14.0-17.el9.noarch 1.6 MB/s | 267 kB 00:00 2026-02-23T11:14:56.219 INFO:teuthology.orchestra.run.trial030.stdout:(87/146): python3-py-1.10.0-6.el9.noarch.rpm 2.9 MB/s | 477 kB 00:00 2026-02-23T11:14:56.378 INFO:teuthology.orchestra.run.trial030.stdout:(88/146): python3-pyasn1-0.4.8-6.el9.noarch.rpm 997 kB/s | 159 kB 00:00 2026-02-23T11:14:56.538 INFO:teuthology.orchestra.run.trial030.stdout:(89/146): python3-pyasn1-modules-0.4.8-6.el9.no 1.7 MB/s | 279 kB 00:00 2026-02-23T11:14:56.698 INFO:teuthology.orchestra.run.trial030.stdout:(90/146): python3-pytest-6.2.2-7.el9.noarch.rpm 3.2 MB/s | 519 kB 00:00 2026-02-23T11:14:56.857 INFO:teuthology.orchestra.run.trial030.stdout:(91/146): python3-requests-oauthlib-1.3.0-12.el 339 kB/s | 54 kB 00:00 2026-02-23T11:14:57.431 INFO:teuthology.orchestra.run.trial030.stdout:(92/146): python3-scipy-1.9.3-2.el9.x86_64.rpm 34 MB/s | 19 MB 00:00 2026-02-23T11:14:57.589 INFO:teuthology.orchestra.run.trial030.stdout:(93/146): python3-toml-0.10.2-6.el9.noarch.rpm 263 kB/s | 42 kB 00:00 2026-02-23T11:14:57.748 INFO:teuthology.orchestra.run.trial030.stdout:(94/146): qatlib-25.08.0-2.el9.x86_64.rpm 1.5 MB/s | 240 kB 00:00 2026-02-23T11:14:57.907 INFO:teuthology.orchestra.run.trial030.stdout:(95/146): qatlib-service-25.08.0-2.el9.x86_64.r 233 kB/s | 37 kB 00:00 2026-02-23T11:14:58.066 INFO:teuthology.orchestra.run.trial030.stdout:(96/146): qatzip-libs-1.3.1-1.el9.x86_64.rpm 419 kB/s | 66 kB 00:00 2026-02-23T11:14:58.228 INFO:teuthology.orchestra.run.trial030.stdout:(97/146): socat-1.7.4.1-8.el9.x86_64.rpm 1.8 MB/s | 303 kB 00:00 2026-02-23T11:14:58.387 INFO:teuthology.orchestra.run.trial030.stdout:(98/146): xmlstarlet-1.6.1-20.el9.x86_64.rpm 399 kB/s | 64 kB 00:00 2026-02-23T11:14:58.512 INFO:teuthology.orchestra.run.trial030.stdout:(99/146): lua-devel-5.4.4-4.el9.x86_64.rpm 179 kB/s | 22 kB 00:00 2026-02-23T11:14:58.614 INFO:teuthology.orchestra.run.trial030.stdout:(100/146): protobuf-compiler-3.14.0-17.el9.x86_ 8.2 MB/s | 862 kB 00:00 2026-02-23T11:14:59.110 INFO:teuthology.orchestra.run.trial030.stdout:(101/146): abseil-cpp-20211102.0-4.el9.x86_64.r 1.1 MB/s | 551 kB 00:00 2026-02-23T11:14:59.543 INFO:teuthology.orchestra.run.trial030.stdout:(102/146): gperftools-libs-2.9.1-3.el9.x86_64.r 710 kB/s | 308 kB 00:00 2026-02-23T11:14:59.760 INFO:teuthology.orchestra.run.trial030.stdout:(103/146): grpc-data-1.46.7-10.el9.noarch.rpm 90 kB/s | 19 kB 00:00 2026-02-23T11:15:00.832 INFO:teuthology.orchestra.run.trial030.stdout:(104/146): libarrow-9.0.0-15.el9.x86_64.rpm 4.1 MB/s | 4.4 MB 00:01 2026-02-23T11:15:01.060 INFO:teuthology.orchestra.run.trial030.stdout:(105/146): libarrow-doc-9.0.0-15.el9.noarch.rpm 109 kB/s | 25 kB 00:00 2026-02-23T11:15:01.349 INFO:teuthology.orchestra.run.trial030.stdout:(106/146): liboath-2.6.12-1.el9.x86_64.rpm 170 kB/s | 49 kB 00:00 2026-02-23T11:15:01.637 INFO:teuthology.orchestra.run.trial030.stdout:(107/146): libunwind-1.6.2-1.el9.x86_64.rpm 233 kB/s | 67 kB 00:00 2026-02-23T11:15:01.999 INFO:teuthology.orchestra.run.trial030.stdout:(108/146): luarocks-3.9.2-5.el9.noarch.rpm 418 kB/s | 151 kB 00:00 2026-02-23T11:15:02.506 INFO:teuthology.orchestra.run.trial030.stdout:(109/146): parquet-libs-9.0.0-15.el9.x86_64.rpm 1.6 MB/s | 838 kB 00:00 2026-02-23T11:15:03.012 INFO:teuthology.orchestra.run.trial030.stdout:(110/146): python3-asyncssh-2.13.2-5.el9.noarch 1.1 MB/s | 548 kB 00:00 2026-02-23T11:15:03.223 INFO:teuthology.orchestra.run.trial030.stdout:(111/146): python3-autocommand-2.2.2-8.el9.noar 140 kB/s | 29 kB 00:00 2026-02-23T11:15:03.501 INFO:teuthology.orchestra.run.trial030.stdout:(112/146): python3-backports-tarfile-1.2.0-1.el 216 kB/s | 60 kB 00:00 2026-02-23T11:15:03.712 INFO:teuthology.orchestra.run.trial030.stdout:(113/146): python3-bcrypt-3.2.2-1.el9.x86_64.rp 206 kB/s | 43 kB 00:00 2026-02-23T11:15:03.930 INFO:teuthology.orchestra.run.trial030.stdout:(114/146): python3-cachetools-4.2.4-1.el9.noarc 148 kB/s | 32 kB 00:00 2026-02-23T11:15:04.069 INFO:teuthology.orchestra.run.trial030.stdout:(115/146): python3-certifi-2023.05.07-4.el9.noa 101 kB/s | 14 kB 00:00 2026-02-23T11:15:04.429 INFO:teuthology.orchestra.run.trial030.stdout:(116/146): python3-cheroot-10.0.1-4.el9.noarch. 483 kB/s | 173 kB 00:00 2026-02-23T11:15:04.859 INFO:teuthology.orchestra.run.trial030.stdout:(117/146): python3-cherrypy-18.6.1-2.el9.noarch 833 kB/s | 358 kB 00:00 2026-02-23T11:15:05.289 INFO:teuthology.orchestra.run.trial030.stdout:(118/146): python3-google-auth-2.45.0-1.el9.noa 592 kB/s | 254 kB 00:00 2026-02-23T11:15:05.991 INFO:teuthology.orchestra.run.trial030.stdout:(119/146): python3-grpcio-1.46.7-10.el9.x86_64. 2.9 MB/s | 2.0 MB 00:00 2026-02-23T11:15:06.338 INFO:teuthology.orchestra.run.trial030.stdout:(120/146): python3-grpcio-tools-1.46.7-10.el9.x 417 kB/s | 144 kB 00:00 2026-02-23T11:15:06.482 INFO:teuthology.orchestra.run.trial030.stdout:(121/146): python3-jaraco-8.2.1-3.el9.noarch.rp 74 kB/s | 11 kB 00:00 2026-02-23T11:15:06.700 INFO:teuthology.orchestra.run.trial030.stdout:(122/146): python3-jaraco-classes-3.2.1-5.el9.n 81 kB/s | 18 kB 00:00 2026-02-23T11:15:06.909 INFO:teuthology.orchestra.run.trial030.stdout:(123/146): python3-jaraco-collections-3.0.0-8.e 111 kB/s | 23 kB 00:00 2026-02-23T11:15:07.120 INFO:teuthology.orchestra.run.trial030.stdout:(124/146): python3-jaraco-context-6.0.1-3.el9.n 93 kB/s | 20 kB 00:00 2026-02-23T11:15:07.337 INFO:teuthology.orchestra.run.trial030.stdout:(125/146): python3-jaraco-functools-3.5.0-2.el9 89 kB/s | 19 kB 00:00 2026-02-23T11:15:07.554 INFO:teuthology.orchestra.run.trial030.stdout:(126/146): python3-jaraco-text-4.0.0-2.el9.noar 121 kB/s | 26 kB 00:00 2026-02-23T11:15:08.133 INFO:teuthology.orchestra.run.trial030.stdout:(127/146): python3-kubernetes-26.1.0-3.el9.noar 1.8 MB/s | 1.0 MB 00:00 2026-02-23T11:15:08.411 INFO:teuthology.orchestra.run.trial030.stdout:(128/146): python3-more-itertools-8.12.0-2.el9. 283 kB/s | 79 kB 00:00 2026-02-23T11:15:08.697 INFO:teuthology.orchestra.run.trial030.stdout:(129/146): python3-natsort-7.1.1-5.el9.noarch.r 201 kB/s | 58 kB 00:00 2026-02-23T11:15:08.907 INFO:teuthology.orchestra.run.trial030.stdout:(130/146): python3-portend-3.1.0-2.el9.noarch.r 79 kB/s | 16 kB 00:00 2026-02-23T11:15:09.185 INFO:teuthology.orchestra.run.trial030.stdout:(131/146): python3-pyOpenSSL-21.0.0-1.el9.noarc 323 kB/s | 90 kB 00:00 2026-02-23T11:15:09.403 INFO:teuthology.orchestra.run.trial030.stdout:(132/146): python3-repoze-lru-0.7-16.el9.noarch 141 kB/s | 31 kB 00:00 2026-02-23T11:15:09.752 INFO:teuthology.orchestra.run.trial030.stdout:(133/146): python3-routes-2.5.1-5.el9.noarch.rp 540 kB/s | 188 kB 00:00 2026-02-23T11:15:10.041 INFO:teuthology.orchestra.run.trial030.stdout:(134/146): python3-rsa-4.9-2.el9.noarch.rpm 204 kB/s | 59 kB 00:00 2026-02-23T11:15:10.256 INFO:teuthology.orchestra.run.trial030.stdout:(135/146): python3-tempora-5.0.0-2.el9.noarch.r 166 kB/s | 36 kB 00:00 2026-02-23T11:15:10.537 INFO:teuthology.orchestra.run.trial030.stdout:(136/146): python3-typing-extensions-4.15.0-1.e 308 kB/s | 86 kB 00:00 2026-02-23T11:15:10.828 INFO:teuthology.orchestra.run.trial030.stdout:(137/146): python3-websocket-client-1.2.3-2.el9 309 kB/s | 90 kB 00:00 2026-02-23T11:15:11.045 INFO:teuthology.orchestra.run.trial030.stdout:(138/146): python3-xmltodict-0.12.0-15.el9.noar 102 kB/s | 22 kB 00:00 2026-02-23T11:15:11.257 INFO:teuthology.orchestra.run.trial030.stdout:(139/146): python3-zc-lockfile-2.0-10.el9.noarc 94 kB/s | 20 kB 00:00 2026-02-23T11:15:11.614 INFO:teuthology.orchestra.run.trial030.stdout:(140/146): re2-20211101-20.el9.x86_64.rpm 536 kB/s | 191 kB 00:00 2026-02-23T11:15:12.239 INFO:teuthology.orchestra.run.trial030.stdout:(141/146): thrift-0.15.0-4.el9.x86_64.rpm 2.5 MB/s | 1.6 MB 00:00 2026-02-23T11:15:12.586 INFO:teuthology.orchestra.run.trial030.stdout:(142/146): yaml-cpp-0.6.3-5.el9.x86_64.rpm 351 kB/s | 122 kB 00:00 2026-02-23T11:15:15.498 INFO:teuthology.orchestra.run.trial030.stdout:(143/146): librados2-20.3.0-5439.ge371433f.el9. 1.1 MB/s | 3.3 MB 00:02 2026-02-23T11:15:17.656 INFO:teuthology.orchestra.run.trial030.stdout:(144/146): librbd1-20.3.0-5439.ge371433f.el9.x8 1.2 MB/s | 2.7 MB 00:02 2026-02-23T11:15:22.542 INFO:teuthology.orchestra.run.trial030.stdout:(145/146): ceph-mgr-dashboard-20.3.0-5439.ge371 864 kB/s | 51 MB 01:00 2026-02-23T11:17:57.673 INFO:teuthology.orchestra.run.trial030.stdout:(146/146): ceph-osd-crimson-debuginfo-20.3.0-54 1.4 MB/s | 429 MB 04:56 2026-02-23T11:17:57.675 INFO:teuthology.orchestra.run.trial030.stdout:-------------------------------------------------------------------------------- 2026-02-23T11:17:57.675 INFO:teuthology.orchestra.run.trial030.stdout:Total 2.8 MB/s | 848 MB 05:06 2026-02-23T11:17:58.186 INFO:teuthology.orchestra.run.trial030.stdout:Running transaction check 2026-02-23T11:17:58.234 INFO:teuthology.orchestra.run.trial030.stdout:Transaction check succeeded. 2026-02-23T11:17:58.235 INFO:teuthology.orchestra.run.trial030.stdout:Running transaction test 2026-02-23T11:18:00.896 INFO:teuthology.orchestra.run.trial030.stdout:Transaction test succeeded. 2026-02-23T11:18:00.896 INFO:teuthology.orchestra.run.trial030.stdout:Running transaction 2026-02-23T11:18:03.631 INFO:teuthology.orchestra.run.trial030.stdout: Preparing : 1/1 2026-02-23T11:18:03.634 INFO:teuthology.orchestra.run.trial030.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/148 2026-02-23T11:18:03.642 INFO:teuthology.orchestra.run.trial030.stdout: Installing : liboath-2.6.12-1.el9.x86_64 2/148 2026-02-23T11:18:03.656 INFO:teuthology.orchestra.run.trial030.stdout: Installing : protobuf-3.14.0-17.el9.x86_64 3/148 2026-02-23T11:18:03.800 INFO:teuthology.orchestra.run.trial030.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 4/148 2026-02-23T11:18:03.802 INFO:teuthology.orchestra.run.trial030.stdout: Upgrading : librados2-2:20.3.0-5439.ge371433f.el9.x86_64 5/148 2026-02-23T11:18:03.830 INFO:teuthology.orchestra.run.trial030.stdout: Running scriptlet: librados2-2:20.3.0-5439.ge371433f.el9.x86_64 5/148 2026-02-23T11:18:03.837 INFO:teuthology.orchestra.run.trial030.stdout: Installing : python3-rados-2:20.3.0-5439.ge371433f.el9.x86_64 6/148 2026-02-23T11:18:03.839 INFO:teuthology.orchestra.run.trial030.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 7/148 2026-02-23T11:18:03.842 INFO:teuthology.orchestra.run.trial030.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 8/148 2026-02-23T11:18:03.844 INFO:teuthology.orchestra.run.trial030.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 9/148 2026-02-23T11:18:03.849 INFO:teuthology.orchestra.run.trial030.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 10/148 2026-02-23T11:18:03.963 INFO:teuthology.orchestra.run.trial030.stdout: Installing : libnbd-1.20.3-4.el9.x86_64 11/148 2026-02-23T11:18:03.964 INFO:teuthology.orchestra.run.trial030.stdout: Upgrading : librbd1-2:20.3.0-5439.ge371433f.el9.x86_64 12/148 2026-02-23T11:18:03.980 INFO:teuthology.orchestra.run.trial030.stdout: Running scriptlet: librbd1-2:20.3.0-5439.ge371433f.el9.x86_64 12/148 2026-02-23T11:18:03.981 INFO:teuthology.orchestra.run.trial030.stdout: Installing : libcephsqlite-2:20.3.0-5439.ge371433f.el9.x86_64 13/148 2026-02-23T11:18:04.002 INFO:teuthology.orchestra.run.trial030.stdout: Running scriptlet: libcephsqlite-2:20.3.0-5439.ge371433f.el9.x86_64 13/148 2026-02-23T11:18:04.003 INFO:teuthology.orchestra.run.trial030.stdout: Installing : libradosstriper1-2:20.3.0-5439.ge371433f.el9.x86 14/148 2026-02-23T11:18:04.015 INFO:teuthology.orchestra.run.trial030.stdout: Running scriptlet: libradosstriper1-2:20.3.0-5439.ge371433f.el9.x86 14/148 2026-02-23T11:18:04.019 INFO:teuthology.orchestra.run.trial030.stdout: Installing : yaml-cpp-0.6.3-5.el9.x86_64 15/148 2026-02-23T11:18:04.052 INFO:teuthology.orchestra.run.trial030.stdout: Installing : re2-1:20211101-20.el9.x86_64 16/148 2026-02-23T11:18:04.057 INFO:teuthology.orchestra.run.trial030.stdout: Installing : libarrow-9.0.0-15.el9.x86_64 17/148 2026-02-23T11:18:04.061 INFO:teuthology.orchestra.run.trial030.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 18/148 2026-02-23T11:18:04.082 INFO:teuthology.orchestra.run.trial030.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 19/148 2026-02-23T11:18:04.088 INFO:teuthology.orchestra.run.trial030.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 20/148 2026-02-23T11:18:04.091 INFO:teuthology.orchestra.run.trial030.stdout: Installing : lua-5.4.4-4.el9.x86_64 21/148 2026-02-23T11:18:04.095 INFO:teuthology.orchestra.run.trial030.stdout: Installing : libubsan-11.5.0-14.el9.x86_64 22/148 2026-02-23T11:18:04.096 INFO:teuthology.orchestra.run.trial030.stdout: Installing : libasan8-15.2.1-7.el9.x86_64 23/148 2026-02-23T11:18:04.107 INFO:teuthology.orchestra.run.trial030.stdout: Running scriptlet: libasan8-15.2.1-7.el9.x86_64 23/148 2026-02-23T11:18:04.112 INFO:teuthology.orchestra.run.trial030.stdout: Installing : flexiblas-3.0.4-9.el9.x86_64 24/148 2026-02-23T11:18:04.135 INFO:teuthology.orchestra.run.trial030.stdout: Installing : unzip-6.0-59.el9.x86_64 25/148 2026-02-23T11:18:04.148 INFO:teuthology.orchestra.run.trial030.stdout: Installing : python3-urllib3-1.26.5-7.el9.noarch 26/148 2026-02-23T11:18:04.154 INFO:teuthology.orchestra.run.trial030.stdout: Installing : python3-requests-2.25.1-10.el9.noarch 27/148 2026-02-23T11:18:04.157 INFO:teuthology.orchestra.run.trial030.stdout: Installing : lksctp-tools-1.0.19-2.el9.x86_64 28/148 2026-02-23T11:18:04.163 INFO:teuthology.orchestra.run.trial030.stdout: Installing : libquadmath-11.5.0-14.el9.x86_64 29/148 2026-02-23T11:18:04.165 INFO:teuthology.orchestra.run.trial030.stdout: Installing : libgfortran-11.5.0-14.el9.x86_64 30/148 2026-02-23T11:18:04.166 INFO:teuthology.orchestra.run.trial030.stdout: Installing : libatomic-11.5.0-14.el9.x86_64 31/148 2026-02-23T11:18:04.176 INFO:teuthology.orchestra.run.trial030.stdout: Installing : ledmon-libs-1.1.0-3.el9.x86_64 32/148 2026-02-23T11:18:04.210 INFO:teuthology.orchestra.run.trial030.stdout: Installing : hwloc-libs-2.4.1-6.el9.x86_64 33/148 2026-02-23T11:18:04.215 INFO:teuthology.orchestra.run.trial030.stdout: Installing : python3-ceph-common-2:20.3.0-5439.ge371433f.el9. 34/148 2026-02-23T11:18:04.217 INFO:teuthology.orchestra.run.trial030.stdout: Installing : python3-ceph-argparse-2:20.3.0-5439.ge371433f.el 35/148 2026-02-23T11:18:04.217 INFO:teuthology.orchestra.run.trial030.stdout: Installing : libcephfs-proxy2-2:20.3.0-5439.ge371433f.el9.x86 36/148 2026-02-23T11:18:04.262 INFO:teuthology.orchestra.run.trial030.stdout: Running scriptlet: libcephfs-proxy2-2:20.3.0-5439.ge371433f.el9.x86 36/148 2026-02-23T11:18:04.263 INFO:teuthology.orchestra.run.trial030.stdout: Installing : libcephfs2-2:20.3.0-5439.ge371433f.el9.x86_64 37/148 2026-02-23T11:18:04.284 INFO:teuthology.orchestra.run.trial030.stdout: Running scriptlet: libcephfs2-2:20.3.0-5439.ge371433f.el9.x86_64 37/148 2026-02-23T11:18:04.297 INFO:teuthology.orchestra.run.trial030.stdout: Installing : python3-cephfs-2:20.3.0-5439.ge371433f.el9.x86_6 38/148 2026-02-23T11:18:04.304 INFO:teuthology.orchestra.run.trial030.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 39/148 2026-02-23T11:18:04.329 INFO:teuthology.orchestra.run.trial030.stdout: Installing : zip-3.0-35.el9.x86_64 40/148 2026-02-23T11:18:04.333 INFO:teuthology.orchestra.run.trial030.stdout: Installing : luarocks-3.9.2-5.el9.noarch 41/148 2026-02-23T11:18:04.383 INFO:teuthology.orchestra.run.trial030.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 42/148 2026-02-23T11:18:04.396 INFO:teuthology.orchestra.run.trial030.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 43/148 2026-02-23T11:18:04.403 INFO:teuthology.orchestra.run.trial030.stdout: Installing : python3-rsa-4.9-2.el9.noarch 44/148 2026-02-23T11:18:04.407 INFO:teuthology.orchestra.run.trial030.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 45/148 2026-02-23T11:18:04.423 INFO:teuthology.orchestra.run.trial030.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 46/148 2026-02-23T11:18:04.428 INFO:teuthology.orchestra.run.trial030.stdout: Installing : python3-rbd-2:20.3.0-5439.ge371433f.el9.x86_64 47/148 2026-02-23T11:18:04.435 INFO:teuthology.orchestra.run.trial030.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 48/148 2026-02-23T11:18:04.442 INFO:teuthology.orchestra.run.trial030.stdout: Installing : librados-devel-2:20.3.0-5439.ge371433f.el9.x86_6 49/148 2026-02-23T11:18:04.454 INFO:teuthology.orchestra.run.trial030.stdout: Installing : protobuf-compiler-3.14.0-17.el9.x86_64 50/148 2026-02-23T11:18:04.459 INFO:teuthology.orchestra.run.trial030.stdout: Installing : thrift-0.15.0-4.el9.x86_64 51/148 2026-02-23T11:18:04.462 INFO:teuthology.orchestra.run.trial030.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 52/148 2026-02-23T11:18:04.476 INFO:teuthology.orchestra.run.trial030.stdout: Installing : python3-xmltodict-0.12.0-15.el9.noarch 53/148 2026-02-23T11:18:04.481 INFO:teuthology.orchestra.run.trial030.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 54/148 2026-02-23T11:18:04.486 INFO:teuthology.orchestra.run.trial030.stdout: Installing : python3-typing-extensions-4.15.0-1.el9.noarch 55/148 2026-02-23T11:18:04.497 INFO:teuthology.orchestra.run.trial030.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 56/148 2026-02-23T11:18:04.506 INFO:teuthology.orchestra.run.trial030.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 57/148 2026-02-23T11:18:04.510 INFO:teuthology.orchestra.run.trial030.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 58/148 2026-02-23T11:18:04.518 INFO:teuthology.orchestra.run.trial030.stdout: Installing : python3-certifi-2023.05.07-4.el9.noarch 59/148 2026-02-23T11:18:04.558 INFO:teuthology.orchestra.run.trial030.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 60/148 2026-02-23T11:18:04.881 INFO:teuthology.orchestra.run.trial030.stdout: Installing : python3-google-auth-1:2.45.0-1.el9.noarch 61/148 2026-02-23T11:18:04.890 INFO:teuthology.orchestra.run.trial030.stdout: Installing : python3-kubernetes-1:26.1.0-3.el9.noarch 62/148 2026-02-23T11:18:04.894 INFO:teuthology.orchestra.run.trial030.stdout: Installing : python3-backports-tarfile-1.2.0-1.el9.noarch 63/148 2026-02-23T11:18:04.900 INFO:teuthology.orchestra.run.trial030.stdout: Installing : python3-jaraco-context-6.0.1-3.el9.noarch 64/148 2026-02-23T11:18:04.906 INFO:teuthology.orchestra.run.trial030.stdout: Installing : python3-autocommand-2.2.2-8.el9.noarch 65/148 2026-02-23T11:18:04.909 INFO:teuthology.orchestra.run.trial030.stdout: Installing : python3-jaraco-text-4.0.0-2.el9.noarch 66/148 2026-02-23T11:18:04.913 INFO:teuthology.orchestra.run.trial030.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 67/148 2026-02-23T11:18:04.919 INFO:teuthology.orchestra.run.trial030.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 68/148 2026-02-23T11:18:04.921 INFO:teuthology.orchestra.run.trial030.stdout: Installing : gperftools-libs-2.9.1-3.el9.x86_64 69/148 2026-02-23T11:18:04.922 INFO:teuthology.orchestra.run.trial030.stdout: Installing : libarrow-doc-9.0.0-15.el9.noarch 70/148 2026-02-23T11:18:04.952 INFO:teuthology.orchestra.run.trial030.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 71/148 2026-02-23T11:18:04.996 INFO:teuthology.orchestra.run.trial030.stdout: Installing : abseil-cpp-20211102.0-4.el9.x86_64 72/148 2026-02-23T11:18:05.005 INFO:teuthology.orchestra.run.trial030.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 73/148 2026-02-23T11:18:05.052 INFO:teuthology.orchestra.run.trial030.stdout: Installing : socat-1.7.4.1-8.el9.x86_64 74/148 2026-02-23T11:18:05.081 INFO:teuthology.orchestra.run.trial030.stdout: Installing : python3-py-1.10.0-6.el9.noarch 75/148 2026-02-23T11:18:05.090 INFO:teuthology.orchestra.run.trial030.stdout: Installing : python3-protobuf-3.14.0-17.el9.noarch 76/148 2026-02-23T11:18:05.097 INFO:teuthology.orchestra.run.trial030.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 77/148 2026-02-23T11:18:05.103 INFO:teuthology.orchestra.run.trial030.stdout: Installing : python3-pluggy-0.13.1-7.el9.noarch 78/148 2026-02-23T11:18:05.110 INFO:teuthology.orchestra.run.trial030.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 79/148 2026-02-23T11:18:05.114 INFO:teuthology.orchestra.run.trial030.stdout: Installing : python3-jmespath-1.0.1-1.el9.noarch 80/148 2026-02-23T11:18:05.148 INFO:teuthology.orchestra.run.trial030.stdout: Installing : python3-iniconfig-1.1.1-7.el9.noarch 81/148 2026-02-23T11:18:05.371 INFO:teuthology.orchestra.run.trial030.stdout: Installing : python3-devel-3.9.25-3.el9.x86_64 82/148 2026-02-23T11:18:05.395 INFO:teuthology.orchestra.run.trial030.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 83/148 2026-02-23T11:18:05.399 INFO:teuthology.orchestra.run.trial030.stdout: Installing : python3-jinja2-2.11.3-8.el9.noarch 84/148 2026-02-23T11:18:05.457 INFO:teuthology.orchestra.run.trial030.stdout: Installing : openblas-0.3.29-1.el9.x86_64 85/148 2026-02-23T11:18:05.459 INFO:teuthology.orchestra.run.trial030.stdout: Installing : openblas-openmp-0.3.29-1.el9.x86_64 86/148 2026-02-23T11:18:05.480 INFO:teuthology.orchestra.run.trial030.stdout: Installing : flexiblas-openblas-openmp-3.0.4-9.el9.x86_64 87/148 2026-02-23T11:18:05.806 INFO:teuthology.orchestra.run.trial030.stdout: Installing : flexiblas-netlib-3.0.4-9.el9.x86_64 88/148 2026-02-23T11:18:05.878 INFO:teuthology.orchestra.run.trial030.stdout: Installing : python3-numpy-1:1.23.5-2.el9.x86_64 89/148 2026-02-23T11:18:06.550 INFO:teuthology.orchestra.run.trial030.stdout: Installing : python3-numpy-f2py-1:1.23.5-2.el9.x86_64 90/148 2026-02-23T11:18:06.567 INFO:teuthology.orchestra.run.trial030.stdout: Installing : python3-scipy-1.9.3-2.el9.x86_64 91/148 2026-02-23T11:18:06.571 INFO:teuthology.orchestra.run.trial030.stdout: Installing : libxslt-1.1.34-12.el9.x86_64 92/148 2026-02-23T11:18:06.573 INFO:teuthology.orchestra.run.trial030.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 93/148 2026-02-23T11:18:06.579 INFO:teuthology.orchestra.run.trial030.stdout: Installing : boost-program-options-1.75.0-13.el9.x86_64 94/148 2026-02-23T11:18:06.825 INFO:teuthology.orchestra.run.trial030.stdout: Installing : parquet-libs-9.0.0-15.el9.x86_64 95/148 2026-02-23T11:18:06.826 INFO:teuthology.orchestra.run.trial030.stdout: Installing : librgw2-2:20.3.0-5439.ge371433f.el9.x86_64 96/148 2026-02-23T11:18:06.843 INFO:teuthology.orchestra.run.trial030.stdout: Running scriptlet: librgw2-2:20.3.0-5439.ge371433f.el9.x86_64 96/148 2026-02-23T11:18:06.845 INFO:teuthology.orchestra.run.trial030.stdout: Installing : python3-rgw-2:20.3.0-5439.ge371433f.el9.x86_64 97/148 2026-02-23T11:18:07.781 INFO:teuthology.orchestra.run.trial030.stdout: Running scriptlet: ceph-common-2:20.3.0-5439.ge371433f.el9.x86_64 98/148 2026-02-23T11:18:07.783 INFO:teuthology.orchestra.run.trial030.stdout: Installing : ceph-common-2:20.3.0-5439.ge371433f.el9.x86_64 98/148 2026-02-23T11:18:07.800 INFO:teuthology.orchestra.run.trial030.stdout: Running scriptlet: ceph-common-2:20.3.0-5439.ge371433f.el9.x86_64 98/148 2026-02-23T11:18:07.810 INFO:teuthology.orchestra.run.trial030.stdout: Installing : python3-pyparsing-2.4.7-9.el9.noarch 99/148 2026-02-23T11:18:07.817 INFO:teuthology.orchestra.run.trial030.stdout: Installing : python3-packaging-20.9-5.el9.noarch 100/148 2026-02-23T11:18:07.830 INFO:teuthology.orchestra.run.trial030.stdout: Installing : python3-ply-3.11-14.el9.noarch 101/148 2026-02-23T11:18:07.846 INFO:teuthology.orchestra.run.trial030.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 102/148 2026-02-23T11:18:07.921 INFO:teuthology.orchestra.run.trial030.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 103/148 2026-02-23T11:18:07.930 INFO:teuthology.orchestra.run.trial030.stdout: Installing : python3-cryptography-36.0.1-5.el9.x86_64 104/148 2026-02-23T11:18:07.953 INFO:teuthology.orchestra.run.trial030.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 105/148 2026-02-23T11:18:07.982 INFO:teuthology.orchestra.run.trial030.stdout: Installing : python3-cheroot-10.0.1-4.el9.noarch 106/148 2026-02-23T11:18:08.033 INFO:teuthology.orchestra.run.trial030.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 107/148 2026-02-23T11:18:08.039 INFO:teuthology.orchestra.run.trial030.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 108/148 2026-02-23T11:18:08.042 INFO:teuthology.orchestra.run.trial030.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 109/148 2026-02-23T11:18:08.046 INFO:teuthology.orchestra.run.trial030.stdout: Installing : pciutils-3.7.0-7.el9.x86_64 110/148 2026-02-23T11:18:08.049 INFO:teuthology.orchestra.run.trial030.stdout: Installing : qatlib-25.08.0-2.el9.x86_64 111/148 2026-02-23T11:18:08.049 INFO:teuthology.orchestra.run.trial030.stdout: Installing : qatlib-service-25.08.0-2.el9.x86_64 112/148 2026-02-23T11:18:08.066 INFO:teuthology.orchestra.run.trial030.stdout: Running scriptlet: qatlib-service-25.08.0-2.el9.x86_64 112/148 2026-02-23T11:18:08.311 INFO:teuthology.orchestra.run.trial030.stdout: Installing : qatzip-libs-1.3.1-1.el9.x86_64 113/148 2026-02-23T11:18:08.314 INFO:teuthology.orchestra.run.trial030.stdout: Installing : ceph-base-2:20.3.0-5439.ge371433f.el9.x86_64 114/148 2026-02-23T11:18:08.351 INFO:teuthology.orchestra.run.trial030.stdout: Running scriptlet: ceph-base-2:20.3.0-5439.ge371433f.el9.x86_64 114/148 2026-02-23T11:18:08.351 INFO:teuthology.orchestra.run.trial030.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2026-02-23T11:18:08.351 INFO:teuthology.orchestra.run.trial030.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2026-02-23T11:18:08.352 INFO:teuthology.orchestra.run.trial030.stdout: 2026-02-23T11:18:08.354 INFO:teuthology.orchestra.run.trial030.stdout: Installing : ceph-selinux-2:20.3.0-5439.ge371433f.el9.x86_64 115/148 2026-02-23T11:18:12.986 INFO:teuthology.orchestra.run.trial030.stdout: Running scriptlet: ceph-selinux-2:20.3.0-5439.ge371433f.el9.x86_64 115/148 2026-02-23T11:18:12.986 INFO:teuthology.orchestra.run.trial030.stdout:skipping the directory /sys 2026-02-23T11:18:12.986 INFO:teuthology.orchestra.run.trial030.stdout:skipping the directory /proc 2026-02-23T11:18:12.986 INFO:teuthology.orchestra.run.trial030.stdout:skipping the directory /mnt 2026-02-23T11:18:12.986 INFO:teuthology.orchestra.run.trial030.stdout:skipping the directory /var/tmp 2026-02-23T11:18:12.986 INFO:teuthology.orchestra.run.trial030.stdout:skipping the directory /home 2026-02-23T11:18:12.986 INFO:teuthology.orchestra.run.trial030.stdout:skipping the directory /root 2026-02-23T11:18:12.986 INFO:teuthology.orchestra.run.trial030.stdout:skipping the directory /tmp 2026-02-23T11:18:12.987 INFO:teuthology.orchestra.run.trial030.stdout: 2026-02-23T11:18:13.074 INFO:teuthology.orchestra.run.trial030.stdout: Installing : ceph-mds-2:20.3.0-5439.ge371433f.el9.x86_64 116/148 2026-02-23T11:18:13.096 INFO:teuthology.orchestra.run.trial030.stdout: Running scriptlet: ceph-mds-2:20.3.0-5439.ge371433f.el9.x86_64 116/148 2026-02-23T11:18:13.096 INFO:teuthology.orchestra.run.trial030.stdout:Glob pattern passed to enable, but globs are not supported for this. 2026-02-23T11:18:13.097 INFO:teuthology.orchestra.run.trial030.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2026-02-23T11:18:13.097 INFO:teuthology.orchestra.run.trial030.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2026-02-23T11:18:13.097 INFO:teuthology.orchestra.run.trial030.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2026-02-23T11:18:13.097 INFO:teuthology.orchestra.run.trial030.stdout: 2026-02-23T11:18:13.300 INFO:teuthology.orchestra.run.trial030.stdout: Installing : ceph-mon-2:20.3.0-5439.ge371433f.el9.x86_64 117/148 2026-02-23T11:18:13.321 INFO:teuthology.orchestra.run.trial030.stdout: Running scriptlet: ceph-mon-2:20.3.0-5439.ge371433f.el9.x86_64 117/148 2026-02-23T11:18:13.321 INFO:teuthology.orchestra.run.trial030.stdout:Glob pattern passed to enable, but globs are not supported for this. 2026-02-23T11:18:13.321 INFO:teuthology.orchestra.run.trial030.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2026-02-23T11:18:13.321 INFO:teuthology.orchestra.run.trial030.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2026-02-23T11:18:13.321 INFO:teuthology.orchestra.run.trial030.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2026-02-23T11:18:13.322 INFO:teuthology.orchestra.run.trial030.stdout: 2026-02-23T11:18:13.328 INFO:teuthology.orchestra.run.trial030.stdout: Installing : mailcap-2.1.49-5.el9.noarch 118/148 2026-02-23T11:18:13.330 INFO:teuthology.orchestra.run.trial030.stdout: Installing : libconfig-1.7.2-9.el9.x86_64 119/148 2026-02-23T11:18:13.348 INFO:teuthology.orchestra.run.trial030.stdout: Running scriptlet: libstoragemgmt-1.10.1-1.el9.x86_64 120/148 2026-02-23T11:18:13.348 INFO:teuthology.orchestra.run.trial030.stdout:Creating group 'qat' with GID 992. 2026-02-23T11:18:13.348 INFO:teuthology.orchestra.run.trial030.stdout:Creating group 'libstoragemgmt' with GID 991. 2026-02-23T11:18:13.348 INFO:teuthology.orchestra.run.trial030.stdout:Creating user 'libstoragemgmt' (daemon account for libstoragemgmt) with UID 991 and GID 991. 2026-02-23T11:18:13.348 INFO:teuthology.orchestra.run.trial030.stdout: 2026-02-23T11:18:13.356 INFO:teuthology.orchestra.run.trial030.stdout: Installing : libstoragemgmt-1.10.1-1.el9.x86_64 120/148 2026-02-23T11:18:13.379 INFO:teuthology.orchestra.run.trial030.stdout: Running scriptlet: libstoragemgmt-1.10.1-1.el9.x86_64 120/148 2026-02-23T11:18:13.379 INFO:teuthology.orchestra.run.trial030.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/libstoragemgmt.service → /usr/lib/systemd/system/libstoragemgmt.service. 2026-02-23T11:18:13.379 INFO:teuthology.orchestra.run.trial030.stdout: 2026-02-23T11:18:13.414 INFO:teuthology.orchestra.run.trial030.stdout: Installing : python3-libstoragemgmt-1.10.1-1.el9.x86_64 121/148 2026-02-23T11:18:14.000 INFO:teuthology.orchestra.run.trial030.stdout: Installing : cryptsetup-2.8.1-3.el9.x86_64 122/148 2026-02-23T11:18:19.325 INFO:teuthology.orchestra.run.trial030.stdout: Installing : ceph-osd-classic-2:20.3.0-5439.ge371433f.el9.x86 123/148 2026-02-23T11:18:19.388 INFO:teuthology.orchestra.run.trial030.stdout: Installing : ceph-osd-crimson-2:20.3.0-5439.ge371433f.el9.x86 124/148 2026-02-23T11:18:19.390 INFO:teuthology.orchestra.run.trial030.stdout: Installing : ceph-volume-2:20.3.0-5439.ge371433f.el9.noarch 125/148 2026-02-23T11:18:19.403 INFO:teuthology.orchestra.run.trial030.stdout: Running scriptlet: ceph-volume-2:20.3.0-5439.ge371433f.el9.noarch 125/148 2026-02-23T11:18:19.403 INFO:teuthology.orchestra.run.trial030.stdout:Glob pattern passed to enable, but globs are not supported for this. 2026-02-23T11:18:19.403 INFO:teuthology.orchestra.run.trial030.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2026-02-23T11:18:19.403 INFO:teuthology.orchestra.run.trial030.stdout: 2026-02-23T11:18:19.414 INFO:teuthology.orchestra.run.trial030.stdout: Installing : ceph-osd-2:20.3.0-5439.ge371433f.el9.x86_64 126/148 2026-02-23T11:18:19.437 INFO:teuthology.orchestra.run.trial030.stdout: Running scriptlet: ceph-osd-2:20.3.0-5439.ge371433f.el9.x86_64 126/148 2026-02-23T11:18:19.437 INFO:teuthology.orchestra.run.trial030.stdout:Glob pattern passed to enable, but globs are not supported for this. 2026-02-23T11:18:19.437 INFO:teuthology.orchestra.run.trial030.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2026-02-23T11:18:19.437 INFO:teuthology.orchestra.run.trial030.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2026-02-23T11:18:19.437 INFO:teuthology.orchestra.run.trial030.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2026-02-23T11:18:19.438 INFO:teuthology.orchestra.run.trial030.stdout: 2026-02-23T11:18:19.491 INFO:teuthology.orchestra.run.trial030.stdout: Running scriptlet: cephadm-2:20.3.0-5439.ge371433f.el9.noarch 127/148 2026-02-23T11:18:19.493 INFO:teuthology.orchestra.run.trial030.stdout: Installing : cephadm-2:20.3.0-5439.ge371433f.el9.noarch 127/148 2026-02-23T11:18:19.502 INFO:teuthology.orchestra.run.trial030.stdout: Installing : ceph-prometheus-alerts-2:20.3.0-5439.ge371433f.e 128/148 2026-02-23T11:18:19.526 INFO:teuthology.orchestra.run.trial030.stdout: Installing : ceph-grafana-dashboards-2:20.3.0-5439.ge371433f. 129/148 2026-02-23T11:18:19.527 INFO:teuthology.orchestra.run.trial030.stdout: Installing : ceph-mgr-cephadm-2:20.3.0-5439.ge371433f.el9.noa 130/148 2026-02-23T11:18:22.412 INFO:teuthology.orchestra.run.trial030.stdout: Running scriptlet: ceph-mgr-cephadm-2:20.3.0-5439.ge371433f.el9.noa 130/148 2026-02-23T11:18:22.419 INFO:teuthology.orchestra.run.trial030.stdout: Installing : ceph-mgr-dashboard-2:20.3.0-5439.ge371433f.el9.n 131/148 2026-02-23T11:18:22.873 INFO:teuthology.orchestra.run.trial030.stdout: Running scriptlet: ceph-mgr-dashboard-2:20.3.0-5439.ge371433f.el9.n 131/148 2026-02-23T11:18:22.874 INFO:teuthology.orchestra.run.trial030.stdout: Installing : ceph-mgr-diskprediction-local-2:20.3.0-5439.ge37 132/148 2026-02-23T11:18:22.934 INFO:teuthology.orchestra.run.trial030.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:20.3.0-5439.ge37 132/148 2026-02-23T11:18:22.973 INFO:teuthology.orchestra.run.trial030.stdout: Installing : ceph-mgr-modules-core-2:20.3.0-5439.ge371433f.el 133/148 2026-02-23T11:18:22.975 INFO:teuthology.orchestra.run.trial030.stdout: Installing : ceph-mgr-2:20.3.0-5439.ge371433f.el9.x86_64 134/148 2026-02-23T11:18:22.995 INFO:teuthology.orchestra.run.trial030.stdout: Running scriptlet: ceph-mgr-2:20.3.0-5439.ge371433f.el9.x86_64 134/148 2026-02-23T11:18:22.995 INFO:teuthology.orchestra.run.trial030.stdout:Glob pattern passed to enable, but globs are not supported for this. 2026-02-23T11:18:22.995 INFO:teuthology.orchestra.run.trial030.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2026-02-23T11:18:22.996 INFO:teuthology.orchestra.run.trial030.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2026-02-23T11:18:22.996 INFO:teuthology.orchestra.run.trial030.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2026-02-23T11:18:22.996 INFO:teuthology.orchestra.run.trial030.stdout: 2026-02-23T11:18:23.008 INFO:teuthology.orchestra.run.trial030.stdout: Installing : ceph-mgr-rook-2:20.3.0-5439.ge371433f.el9.noarch 135/148 2026-02-23T11:18:23.018 INFO:teuthology.orchestra.run.trial030.stdout: Running scriptlet: ceph-mgr-rook-2:20.3.0-5439.ge371433f.el9.noarch 135/148 2026-02-23T11:18:23.505 INFO:teuthology.orchestra.run.trial030.stdout: Installing : ceph-2:20.3.0-5439.ge371433f.el9.x86_64 136/148 2026-02-23T11:18:23.507 INFO:teuthology.orchestra.run.trial030.stdout: Installing : ceph-radosgw-2:20.3.0-5439.ge371433f.el9.x86_64 137/148 2026-02-23T11:18:23.528 INFO:teuthology.orchestra.run.trial030.stdout: Running scriptlet: ceph-radosgw-2:20.3.0-5439.ge371433f.el9.x86_64 137/148 2026-02-23T11:18:23.528 INFO:teuthology.orchestra.run.trial030.stdout:Glob pattern passed to enable, but globs are not supported for this. 2026-02-23T11:18:23.528 INFO:teuthology.orchestra.run.trial030.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2026-02-23T11:18:23.529 INFO:teuthology.orchestra.run.trial030.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2026-02-23T11:18:23.529 INFO:teuthology.orchestra.run.trial030.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2026-02-23T11:18:23.529 INFO:teuthology.orchestra.run.trial030.stdout: 2026-02-23T11:18:23.538 INFO:teuthology.orchestra.run.trial030.stdout: Installing : ceph-immutable-object-cache-2:20.3.0-5439.ge3714 138/148 2026-02-23T11:18:23.557 INFO:teuthology.orchestra.run.trial030.stdout: Running scriptlet: ceph-immutable-object-cache-2:20.3.0-5439.ge3714 138/148 2026-02-23T11:18:23.557 INFO:teuthology.orchestra.run.trial030.stdout:Glob pattern passed to enable, but globs are not supported for this. 2026-02-23T11:18:23.557 INFO:teuthology.orchestra.run.trial030.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2026-02-23T11:18:23.557 INFO:teuthology.orchestra.run.trial030.stdout: 2026-02-23T11:18:23.666 INFO:teuthology.orchestra.run.trial030.stdout: Installing : rbd-mirror-2:20.3.0-5439.ge371433f.el9.x86_64 139/148 2026-02-23T11:18:23.685 INFO:teuthology.orchestra.run.trial030.stdout: Running scriptlet: rbd-mirror-2:20.3.0-5439.ge371433f.el9.x86_64 139/148 2026-02-23T11:18:23.686 INFO:teuthology.orchestra.run.trial030.stdout:Glob pattern passed to enable, but globs are not supported for this. 2026-02-23T11:18:23.686 INFO:teuthology.orchestra.run.trial030.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2026-02-23T11:18:23.686 INFO:teuthology.orchestra.run.trial030.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2026-02-23T11:18:23.686 INFO:teuthology.orchestra.run.trial030.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2026-02-23T11:18:23.686 INFO:teuthology.orchestra.run.trial030.stdout: 2026-02-23T11:18:30.589 INFO:teuthology.orchestra.run.trial030.stdout: Installing : python3-pytest-6.2.2-7.el9.noarch 140/148 2026-02-23T11:18:30.596 INFO:teuthology.orchestra.run.trial030.stdout: Installing : ceph-test-2:20.3.0-5439.ge371433f.el9.x86_64 141/148 2026-02-23T11:18:30.600 INFO:teuthology.orchestra.run.trial030.stdout: Installing : libcephfs-devel-2:20.3.0-5439.ge371433f.el9.x86_ 142/148 2026-02-23T11:18:30.609 INFO:teuthology.orchestra.run.trial030.stdout: Installing : rbd-fuse-2:20.3.0-5439.ge371433f.el9.x86_64 143/148 2026-02-23T11:18:30.650 INFO:teuthology.orchestra.run.trial030.stdout: Installing : rbd-nbd-2:20.3.0-5439.ge371433f.el9.x86_64 144/148 2026-02-23T11:18:52.675 INFO:teuthology.orchestra.run.trial030.stdout: Installing : ceph-fuse-2:20.3.0-5439.ge371433f.el9.x86_64 145/148 2026-02-23T11:18:52.678 INFO:teuthology.orchestra.run.trial030.stdout: Installing : ceph-osd-crimson-debuginfo-2:20.3.0-5439.ge37143 146/148 2026-02-23T11:18:52.678 INFO:teuthology.orchestra.run.trial030.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 147/148 2026-02-23T11:18:52.691 INFO:teuthology.orchestra.run.trial030.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 147/148 2026-02-23T11:18:52.691 INFO:teuthology.orchestra.run.trial030.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 148/148 2026-02-23T11:18:52.702 INFO:teuthology.orchestra.run.trial030.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 148/148 2026-02-23T11:18:52.709 INFO:teuthology.orchestra.run.trial030.stdout: Running scriptlet: ceph-osd-classic-2:20.3.0-5439.ge371433f.el9.x86 148/148 2026-02-23T11:18:52.732 INFO:teuthology.orchestra.run.trial030.stdout: Running scriptlet: ceph-osd-crimson-2:20.3.0-5439.ge371433f.el9.x86 148/148 2026-02-23T11:18:56.331 INFO:teuthology.orchestra.run.trial030.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 148/148 2026-02-23T11:18:56.331 INFO:teuthology.orchestra.run.trial030.stdout: Verifying : ceph-2:20.3.0-5439.ge371433f.el9.x86_64 1/148 2026-02-23T11:18:56.331 INFO:teuthology.orchestra.run.trial030.stdout: Verifying : ceph-base-2:20.3.0-5439.ge371433f.el9.x86_64 2/148 2026-02-23T11:18:56.331 INFO:teuthology.orchestra.run.trial030.stdout: Verifying : ceph-common-2:20.3.0-5439.ge371433f.el9.x86_64 3/148 2026-02-23T11:18:56.331 INFO:teuthology.orchestra.run.trial030.stdout: Verifying : ceph-fuse-2:20.3.0-5439.ge371433f.el9.x86_64 4/148 2026-02-23T11:18:56.331 INFO:teuthology.orchestra.run.trial030.stdout: Verifying : ceph-immutable-object-cache-2:20.3.0-5439.ge3714 5/148 2026-02-23T11:18:56.331 INFO:teuthology.orchestra.run.trial030.stdout: Verifying : ceph-mds-2:20.3.0-5439.ge371433f.el9.x86_64 6/148 2026-02-23T11:18:56.331 INFO:teuthology.orchestra.run.trial030.stdout: Verifying : ceph-mgr-2:20.3.0-5439.ge371433f.el9.x86_64 7/148 2026-02-23T11:18:56.331 INFO:teuthology.orchestra.run.trial030.stdout: Verifying : ceph-mon-2:20.3.0-5439.ge371433f.el9.x86_64 8/148 2026-02-23T11:18:56.332 INFO:teuthology.orchestra.run.trial030.stdout: Verifying : ceph-osd-2:20.3.0-5439.ge371433f.el9.x86_64 9/148 2026-02-23T11:18:56.332 INFO:teuthology.orchestra.run.trial030.stdout: Verifying : ceph-osd-classic-2:20.3.0-5439.ge371433f.el9.x86 10/148 2026-02-23T11:18:56.332 INFO:teuthology.orchestra.run.trial030.stdout: Verifying : ceph-osd-crimson-2:20.3.0-5439.ge371433f.el9.x86 11/148 2026-02-23T11:18:56.332 INFO:teuthology.orchestra.run.trial030.stdout: Verifying : ceph-osd-crimson-debuginfo-2:20.3.0-5439.ge37143 12/148 2026-02-23T11:18:56.332 INFO:teuthology.orchestra.run.trial030.stdout: Verifying : ceph-radosgw-2:20.3.0-5439.ge371433f.el9.x86_64 13/148 2026-02-23T11:18:56.332 INFO:teuthology.orchestra.run.trial030.stdout: Verifying : ceph-selinux-2:20.3.0-5439.ge371433f.el9.x86_64 14/148 2026-02-23T11:18:56.332 INFO:teuthology.orchestra.run.trial030.stdout: Verifying : ceph-test-2:20.3.0-5439.ge371433f.el9.x86_64 15/148 2026-02-23T11:18:56.332 INFO:teuthology.orchestra.run.trial030.stdout: Verifying : libcephfs-devel-2:20.3.0-5439.ge371433f.el9.x86_ 16/148 2026-02-23T11:18:56.332 INFO:teuthology.orchestra.run.trial030.stdout: Verifying : libcephfs-proxy2-2:20.3.0-5439.ge371433f.el9.x86 17/148 2026-02-23T11:18:56.332 INFO:teuthology.orchestra.run.trial030.stdout: Verifying : libcephfs2-2:20.3.0-5439.ge371433f.el9.x86_64 18/148 2026-02-23T11:18:56.332 INFO:teuthology.orchestra.run.trial030.stdout: Verifying : libcephsqlite-2:20.3.0-5439.ge371433f.el9.x86_64 19/148 2026-02-23T11:18:56.332 INFO:teuthology.orchestra.run.trial030.stdout: Verifying : librados-devel-2:20.3.0-5439.ge371433f.el9.x86_6 20/148 2026-02-23T11:18:56.333 INFO:teuthology.orchestra.run.trial030.stdout: Verifying : libradosstriper1-2:20.3.0-5439.ge371433f.el9.x86 21/148 2026-02-23T11:18:56.333 INFO:teuthology.orchestra.run.trial030.stdout: Verifying : librgw2-2:20.3.0-5439.ge371433f.el9.x86_64 22/148 2026-02-23T11:18:56.333 INFO:teuthology.orchestra.run.trial030.stdout: Verifying : python3-ceph-argparse-2:20.3.0-5439.ge371433f.el 23/148 2026-02-23T11:18:56.333 INFO:teuthology.orchestra.run.trial030.stdout: Verifying : python3-ceph-common-2:20.3.0-5439.ge371433f.el9. 24/148 2026-02-23T11:18:56.333 INFO:teuthology.orchestra.run.trial030.stdout: Verifying : python3-cephfs-2:20.3.0-5439.ge371433f.el9.x86_6 25/148 2026-02-23T11:18:56.333 INFO:teuthology.orchestra.run.trial030.stdout: Verifying : python3-rados-2:20.3.0-5439.ge371433f.el9.x86_64 26/148 2026-02-23T11:18:56.333 INFO:teuthology.orchestra.run.trial030.stdout: Verifying : python3-rbd-2:20.3.0-5439.ge371433f.el9.x86_64 27/148 2026-02-23T11:18:56.333 INFO:teuthology.orchestra.run.trial030.stdout: Verifying : python3-rgw-2:20.3.0-5439.ge371433f.el9.x86_64 28/148 2026-02-23T11:18:56.333 INFO:teuthology.orchestra.run.trial030.stdout: Verifying : rbd-fuse-2:20.3.0-5439.ge371433f.el9.x86_64 29/148 2026-02-23T11:18:56.333 INFO:teuthology.orchestra.run.trial030.stdout: Verifying : rbd-mirror-2:20.3.0-5439.ge371433f.el9.x86_64 30/148 2026-02-23T11:18:56.333 INFO:teuthology.orchestra.run.trial030.stdout: Verifying : rbd-nbd-2:20.3.0-5439.ge371433f.el9.x86_64 31/148 2026-02-23T11:18:56.333 INFO:teuthology.orchestra.run.trial030.stdout: Verifying : ceph-grafana-dashboards-2:20.3.0-5439.ge371433f. 32/148 2026-02-23T11:18:56.334 INFO:teuthology.orchestra.run.trial030.stdout: Verifying : ceph-mgr-cephadm-2:20.3.0-5439.ge371433f.el9.noa 33/148 2026-02-23T11:18:56.334 INFO:teuthology.orchestra.run.trial030.stdout: Verifying : ceph-mgr-dashboard-2:20.3.0-5439.ge371433f.el9.n 34/148 2026-02-23T11:18:56.334 INFO:teuthology.orchestra.run.trial030.stdout: Verifying : ceph-mgr-diskprediction-local-2:20.3.0-5439.ge37 35/148 2026-02-23T11:18:56.334 INFO:teuthology.orchestra.run.trial030.stdout: Verifying : ceph-mgr-modules-core-2:20.3.0-5439.ge371433f.el 36/148 2026-02-23T11:18:56.334 INFO:teuthology.orchestra.run.trial030.stdout: Verifying : ceph-mgr-rook-2:20.3.0-5439.ge371433f.el9.noarch 37/148 2026-02-23T11:18:56.334 INFO:teuthology.orchestra.run.trial030.stdout: Verifying : ceph-prometheus-alerts-2:20.3.0-5439.ge371433f.e 38/148 2026-02-23T11:18:56.334 INFO:teuthology.orchestra.run.trial030.stdout: Verifying : ceph-volume-2:20.3.0-5439.ge371433f.el9.noarch 39/148 2026-02-23T11:18:56.334 INFO:teuthology.orchestra.run.trial030.stdout: Verifying : cephadm-2:20.3.0-5439.ge371433f.el9.noarch 40/148 2026-02-23T11:18:56.334 INFO:teuthology.orchestra.run.trial030.stdout: Verifying : cryptsetup-2.8.1-3.el9.x86_64 41/148 2026-02-23T11:18:56.334 INFO:teuthology.orchestra.run.trial030.stdout: Verifying : hwloc-libs-2.4.1-6.el9.x86_64 42/148 2026-02-23T11:18:56.334 INFO:teuthology.orchestra.run.trial030.stdout: Verifying : ledmon-libs-1.1.0-3.el9.x86_64 43/148 2026-02-23T11:18:56.334 INFO:teuthology.orchestra.run.trial030.stdout: Verifying : libatomic-11.5.0-14.el9.x86_64 44/148 2026-02-23T11:18:56.334 INFO:teuthology.orchestra.run.trial030.stdout: Verifying : libconfig-1.7.2-9.el9.x86_64 45/148 2026-02-23T11:18:56.334 INFO:teuthology.orchestra.run.trial030.stdout: Verifying : libgfortran-11.5.0-14.el9.x86_64 46/148 2026-02-23T11:18:56.335 INFO:teuthology.orchestra.run.trial030.stdout: Verifying : libquadmath-11.5.0-14.el9.x86_64 47/148 2026-02-23T11:18:56.335 INFO:teuthology.orchestra.run.trial030.stdout: Verifying : lksctp-tools-1.0.19-2.el9.x86_64 48/148 2026-02-23T11:18:56.335 INFO:teuthology.orchestra.run.trial030.stdout: Verifying : mailcap-2.1.49-5.el9.noarch 49/148 2026-02-23T11:18:56.335 INFO:teuthology.orchestra.run.trial030.stdout: Verifying : pciutils-3.7.0-7.el9.x86_64 50/148 2026-02-23T11:18:56.335 INFO:teuthology.orchestra.run.trial030.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 51/148 2026-02-23T11:18:56.335 INFO:teuthology.orchestra.run.trial030.stdout: Verifying : python3-cryptography-36.0.1-5.el9.x86_64 52/148 2026-02-23T11:18:56.335 INFO:teuthology.orchestra.run.trial030.stdout: Verifying : python3-ply-3.11-14.el9.noarch 53/148 2026-02-23T11:18:56.335 INFO:teuthology.orchestra.run.trial030.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 54/148 2026-02-23T11:18:56.335 INFO:teuthology.orchestra.run.trial030.stdout: Verifying : python3-pyparsing-2.4.7-9.el9.noarch 55/148 2026-02-23T11:18:56.335 INFO:teuthology.orchestra.run.trial030.stdout: Verifying : python3-requests-2.25.1-10.el9.noarch 56/148 2026-02-23T11:18:56.335 INFO:teuthology.orchestra.run.trial030.stdout: Verifying : python3-urllib3-1.26.5-7.el9.noarch 57/148 2026-02-23T11:18:56.335 INFO:teuthology.orchestra.run.trial030.stdout: Verifying : unzip-6.0-59.el9.x86_64 58/148 2026-02-23T11:18:56.336 INFO:teuthology.orchestra.run.trial030.stdout: Verifying : zip-3.0-35.el9.x86_64 59/148 2026-02-23T11:18:56.336 INFO:teuthology.orchestra.run.trial030.stdout: Verifying : boost-program-options-1.75.0-13.el9.x86_64 60/148 2026-02-23T11:18:56.336 INFO:teuthology.orchestra.run.trial030.stdout: Verifying : flexiblas-3.0.4-9.el9.x86_64 61/148 2026-02-23T11:18:56.336 INFO:teuthology.orchestra.run.trial030.stdout: Verifying : flexiblas-netlib-3.0.4-9.el9.x86_64 62/148 2026-02-23T11:18:56.336 INFO:teuthology.orchestra.run.trial030.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-9.el9.x86_64 63/148 2026-02-23T11:18:56.336 INFO:teuthology.orchestra.run.trial030.stdout: Verifying : libasan8-15.2.1-7.el9.x86_64 64/148 2026-02-23T11:18:56.336 INFO:teuthology.orchestra.run.trial030.stdout: Verifying : libnbd-1.20.3-4.el9.x86_64 65/148 2026-02-23T11:18:56.336 INFO:teuthology.orchestra.run.trial030.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 66/148 2026-02-23T11:18:56.336 INFO:teuthology.orchestra.run.trial030.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 67/148 2026-02-23T11:18:56.336 INFO:teuthology.orchestra.run.trial030.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 68/148 2026-02-23T11:18:56.336 INFO:teuthology.orchestra.run.trial030.stdout: Verifying : libstoragemgmt-1.10.1-1.el9.x86_64 69/148 2026-02-23T11:18:56.336 INFO:teuthology.orchestra.run.trial030.stdout: Verifying : libubsan-11.5.0-14.el9.x86_64 70/148 2026-02-23T11:18:56.336 INFO:teuthology.orchestra.run.trial030.stdout: Verifying : libxslt-1.1.34-12.el9.x86_64 71/148 2026-02-23T11:18:56.337 INFO:teuthology.orchestra.run.trial030.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 72/148 2026-02-23T11:18:56.337 INFO:teuthology.orchestra.run.trial030.stdout: Verifying : lua-5.4.4-4.el9.x86_64 73/148 2026-02-23T11:18:56.337 INFO:teuthology.orchestra.run.trial030.stdout: Verifying : openblas-0.3.29-1.el9.x86_64 74/148 2026-02-23T11:18:56.337 INFO:teuthology.orchestra.run.trial030.stdout: Verifying : openblas-openmp-0.3.29-1.el9.x86_64 75/148 2026-02-23T11:18:56.337 INFO:teuthology.orchestra.run.trial030.stdout: Verifying : protobuf-3.14.0-17.el9.x86_64 76/148 2026-02-23T11:18:56.337 INFO:teuthology.orchestra.run.trial030.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 77/148 2026-02-23T11:18:56.337 INFO:teuthology.orchestra.run.trial030.stdout: Verifying : python3-devel-3.9.25-3.el9.x86_64 78/148 2026-02-23T11:18:56.337 INFO:teuthology.orchestra.run.trial030.stdout: Verifying : python3-iniconfig-1.1.1-7.el9.noarch 79/148 2026-02-23T11:18:56.337 INFO:teuthology.orchestra.run.trial030.stdout: Verifying : python3-jinja2-2.11.3-8.el9.noarch 80/148 2026-02-23T11:18:56.337 INFO:teuthology.orchestra.run.trial030.stdout: Verifying : python3-jmespath-1.0.1-1.el9.noarch 81/148 2026-02-23T11:18:56.337 INFO:teuthology.orchestra.run.trial030.stdout: Verifying : python3-libstoragemgmt-1.10.1-1.el9.x86_64 82/148 2026-02-23T11:18:56.337 INFO:teuthology.orchestra.run.trial030.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 83/148 2026-02-23T11:18:56.338 INFO:teuthology.orchestra.run.trial030.stdout: Verifying : python3-numpy-1:1.23.5-2.el9.x86_64 84/148 2026-02-23T11:18:56.338 INFO:teuthology.orchestra.run.trial030.stdout: Verifying : python3-numpy-f2py-1:1.23.5-2.el9.x86_64 85/148 2026-02-23T11:18:56.338 INFO:teuthology.orchestra.run.trial030.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 86/148 2026-02-23T11:18:56.338 INFO:teuthology.orchestra.run.trial030.stdout: Verifying : python3-pluggy-0.13.1-7.el9.noarch 87/148 2026-02-23T11:18:56.338 INFO:teuthology.orchestra.run.trial030.stdout: Verifying : python3-protobuf-3.14.0-17.el9.noarch 88/148 2026-02-23T11:18:56.338 INFO:teuthology.orchestra.run.trial030.stdout: Verifying : python3-py-1.10.0-6.el9.noarch 89/148 2026-02-23T11:18:56.338 INFO:teuthology.orchestra.run.trial030.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 90/148 2026-02-23T11:18:56.338 INFO:teuthology.orchestra.run.trial030.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 91/148 2026-02-23T11:18:56.338 INFO:teuthology.orchestra.run.trial030.stdout: Verifying : python3-pytest-6.2.2-7.el9.noarch 92/148 2026-02-23T11:18:56.338 INFO:teuthology.orchestra.run.trial030.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 93/148 2026-02-23T11:18:56.338 INFO:teuthology.orchestra.run.trial030.stdout: Verifying : python3-scipy-1.9.3-2.el9.x86_64 94/148 2026-02-23T11:18:56.338 INFO:teuthology.orchestra.run.trial030.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 95/148 2026-02-23T11:18:56.338 INFO:teuthology.orchestra.run.trial030.stdout: Verifying : qatlib-25.08.0-2.el9.x86_64 96/148 2026-02-23T11:18:56.338 INFO:teuthology.orchestra.run.trial030.stdout: Verifying : qatlib-service-25.08.0-2.el9.x86_64 97/148 2026-02-23T11:18:56.339 INFO:teuthology.orchestra.run.trial030.stdout: Verifying : qatzip-libs-1.3.1-1.el9.x86_64 98/148 2026-02-23T11:18:56.339 INFO:teuthology.orchestra.run.trial030.stdout: Verifying : socat-1.7.4.1-8.el9.x86_64 99/148 2026-02-23T11:18:56.339 INFO:teuthology.orchestra.run.trial030.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 100/148 2026-02-23T11:18:56.339 INFO:teuthology.orchestra.run.trial030.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 101/148 2026-02-23T11:18:56.339 INFO:teuthology.orchestra.run.trial030.stdout: Verifying : protobuf-compiler-3.14.0-17.el9.x86_64 102/148 2026-02-23T11:18:56.339 INFO:teuthology.orchestra.run.trial030.stdout: Verifying : abseil-cpp-20211102.0-4.el9.x86_64 103/148 2026-02-23T11:18:56.339 INFO:teuthology.orchestra.run.trial030.stdout: Verifying : gperftools-libs-2.9.1-3.el9.x86_64 104/148 2026-02-23T11:18:56.339 INFO:teuthology.orchestra.run.trial030.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 105/148 2026-02-23T11:18:56.339 INFO:teuthology.orchestra.run.trial030.stdout: Verifying : libarrow-9.0.0-15.el9.x86_64 106/148 2026-02-23T11:18:56.339 INFO:teuthology.orchestra.run.trial030.stdout: Verifying : libarrow-doc-9.0.0-15.el9.noarch 107/148 2026-02-23T11:18:56.339 INFO:teuthology.orchestra.run.trial030.stdout: Verifying : liboath-2.6.12-1.el9.x86_64 108/148 2026-02-23T11:18:56.340 INFO:teuthology.orchestra.run.trial030.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 109/148 2026-02-23T11:18:56.340 INFO:teuthology.orchestra.run.trial030.stdout: Verifying : luarocks-3.9.2-5.el9.noarch 110/148 2026-02-23T11:18:56.340 INFO:teuthology.orchestra.run.trial030.stdout: Verifying : parquet-libs-9.0.0-15.el9.x86_64 111/148 2026-02-23T11:18:56.340 INFO:teuthology.orchestra.run.trial030.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 112/148 2026-02-23T11:18:56.340 INFO:teuthology.orchestra.run.trial030.stdout: Verifying : python3-autocommand-2.2.2-8.el9.noarch 113/148 2026-02-23T11:18:56.340 INFO:teuthology.orchestra.run.trial030.stdout: Verifying : python3-backports-tarfile-1.2.0-1.el9.noarch 114/148 2026-02-23T11:18:56.340 INFO:teuthology.orchestra.run.trial030.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 115/148 2026-02-23T11:18:56.340 INFO:teuthology.orchestra.run.trial030.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 116/148 2026-02-23T11:18:56.340 INFO:teuthology.orchestra.run.trial030.stdout: Verifying : python3-certifi-2023.05.07-4.el9.noarch 117/148 2026-02-23T11:18:56.340 INFO:teuthology.orchestra.run.trial030.stdout: Verifying : python3-cheroot-10.0.1-4.el9.noarch 118/148 2026-02-23T11:18:56.340 INFO:teuthology.orchestra.run.trial030.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 119/148 2026-02-23T11:18:56.340 INFO:teuthology.orchestra.run.trial030.stdout: Verifying : python3-google-auth-1:2.45.0-1.el9.noarch 120/148 2026-02-23T11:18:56.340 INFO:teuthology.orchestra.run.trial030.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 121/148 2026-02-23T11:18:56.341 INFO:teuthology.orchestra.run.trial030.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 122/148 2026-02-23T11:18:56.341 INFO:teuthology.orchestra.run.trial030.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 123/148 2026-02-23T11:18:56.341 INFO:teuthology.orchestra.run.trial030.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 124/148 2026-02-23T11:18:56.341 INFO:teuthology.orchestra.run.trial030.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 125/148 2026-02-23T11:18:56.341 INFO:teuthology.orchestra.run.trial030.stdout: Verifying : python3-jaraco-context-6.0.1-3.el9.noarch 126/148 2026-02-23T11:18:56.341 INFO:teuthology.orchestra.run.trial030.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 127/148 2026-02-23T11:18:56.341 INFO:teuthology.orchestra.run.trial030.stdout: Verifying : python3-jaraco-text-4.0.0-2.el9.noarch 128/148 2026-02-23T11:18:56.341 INFO:teuthology.orchestra.run.trial030.stdout: Verifying : python3-kubernetes-1:26.1.0-3.el9.noarch 129/148 2026-02-23T11:18:56.341 INFO:teuthology.orchestra.run.trial030.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 130/148 2026-02-23T11:18:56.341 INFO:teuthology.orchestra.run.trial030.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 131/148 2026-02-23T11:18:56.341 INFO:teuthology.orchestra.run.trial030.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 132/148 2026-02-23T11:18:56.341 INFO:teuthology.orchestra.run.trial030.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 133/148 2026-02-23T11:18:56.341 INFO:teuthology.orchestra.run.trial030.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 134/148 2026-02-23T11:18:56.342 INFO:teuthology.orchestra.run.trial030.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 135/148 2026-02-23T11:18:56.342 INFO:teuthology.orchestra.run.trial030.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 136/148 2026-02-23T11:18:56.342 INFO:teuthology.orchestra.run.trial030.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 137/148 2026-02-23T11:18:56.342 INFO:teuthology.orchestra.run.trial030.stdout: Verifying : python3-typing-extensions-4.15.0-1.el9.noarch 138/148 2026-02-23T11:18:56.342 INFO:teuthology.orchestra.run.trial030.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 139/148 2026-02-23T11:18:56.342 INFO:teuthology.orchestra.run.trial030.stdout: Verifying : python3-xmltodict-0.12.0-15.el9.noarch 140/148 2026-02-23T11:18:56.342 INFO:teuthology.orchestra.run.trial030.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 141/148 2026-02-23T11:18:56.342 INFO:teuthology.orchestra.run.trial030.stdout: Verifying : re2-1:20211101-20.el9.x86_64 142/148 2026-02-23T11:18:56.342 INFO:teuthology.orchestra.run.trial030.stdout: Verifying : thrift-0.15.0-4.el9.x86_64 143/148 2026-02-23T11:18:56.342 INFO:teuthology.orchestra.run.trial030.stdout: Verifying : yaml-cpp-0.6.3-5.el9.x86_64 144/148 2026-02-23T11:18:56.342 INFO:teuthology.orchestra.run.trial030.stdout: Verifying : librados2-2:20.3.0-5439.ge371433f.el9.x86_64 145/148 2026-02-23T11:18:56.342 INFO:teuthology.orchestra.run.trial030.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 146/148 2026-02-23T11:18:56.342 INFO:teuthology.orchestra.run.trial030.stdout: Verifying : librbd1-2:20.3.0-5439.ge371433f.el9.x86_64 147/148 2026-02-23T11:18:56.398 INFO:teuthology.orchestra.run.trial030.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 148/148 2026-02-23T11:18:56.398 INFO:teuthology.orchestra.run.trial030.stdout: 2026-02-23T11:18:56.398 INFO:teuthology.orchestra.run.trial030.stdout:Upgraded: 2026-02-23T11:18:56.398 INFO:teuthology.orchestra.run.trial030.stdout: librados2-2:20.3.0-5439.ge371433f.el9.x86_64 2026-02-23T11:18:56.398 INFO:teuthology.orchestra.run.trial030.stdout: librbd1-2:20.3.0-5439.ge371433f.el9.x86_64 2026-02-23T11:18:56.398 INFO:teuthology.orchestra.run.trial030.stdout:Installed: 2026-02-23T11:18:56.398 INFO:teuthology.orchestra.run.trial030.stdout: abseil-cpp-20211102.0-4.el9.x86_64 2026-02-23T11:18:56.399 INFO:teuthology.orchestra.run.trial030.stdout: boost-program-options-1.75.0-13.el9.x86_64 2026-02-23T11:18:56.399 INFO:teuthology.orchestra.run.trial030.stdout: ceph-2:20.3.0-5439.ge371433f.el9.x86_64 2026-02-23T11:18:56.399 INFO:teuthology.orchestra.run.trial030.stdout: ceph-base-2:20.3.0-5439.ge371433f.el9.x86_64 2026-02-23T11:18:56.399 INFO:teuthology.orchestra.run.trial030.stdout: ceph-common-2:20.3.0-5439.ge371433f.el9.x86_64 2026-02-23T11:18:56.399 INFO:teuthology.orchestra.run.trial030.stdout: ceph-fuse-2:20.3.0-5439.ge371433f.el9.x86_64 2026-02-23T11:18:56.399 INFO:teuthology.orchestra.run.trial030.stdout: ceph-grafana-dashboards-2:20.3.0-5439.ge371433f.el9.noarch 2026-02-23T11:18:56.399 INFO:teuthology.orchestra.run.trial030.stdout: ceph-immutable-object-cache-2:20.3.0-5439.ge371433f.el9.x86_64 2026-02-23T11:18:56.399 INFO:teuthology.orchestra.run.trial030.stdout: ceph-mds-2:20.3.0-5439.ge371433f.el9.x86_64 2026-02-23T11:18:56.399 INFO:teuthology.orchestra.run.trial030.stdout: ceph-mgr-2:20.3.0-5439.ge371433f.el9.x86_64 2026-02-23T11:18:56.399 INFO:teuthology.orchestra.run.trial030.stdout: ceph-mgr-cephadm-2:20.3.0-5439.ge371433f.el9.noarch 2026-02-23T11:18:56.399 INFO:teuthology.orchestra.run.trial030.stdout: ceph-mgr-dashboard-2:20.3.0-5439.ge371433f.el9.noarch 2026-02-23T11:18:56.399 INFO:teuthology.orchestra.run.trial030.stdout: ceph-mgr-diskprediction-local-2:20.3.0-5439.ge371433f.el9.noarch 2026-02-23T11:18:56.400 INFO:teuthology.orchestra.run.trial030.stdout: ceph-mgr-modules-core-2:20.3.0-5439.ge371433f.el9.noarch 2026-02-23T11:18:56.400 INFO:teuthology.orchestra.run.trial030.stdout: ceph-mgr-rook-2:20.3.0-5439.ge371433f.el9.noarch 2026-02-23T11:18:56.400 INFO:teuthology.orchestra.run.trial030.stdout: ceph-mon-2:20.3.0-5439.ge371433f.el9.x86_64 2026-02-23T11:18:56.400 INFO:teuthology.orchestra.run.trial030.stdout: ceph-osd-2:20.3.0-5439.ge371433f.el9.x86_64 2026-02-23T11:18:56.400 INFO:teuthology.orchestra.run.trial030.stdout: ceph-osd-classic-2:20.3.0-5439.ge371433f.el9.x86_64 2026-02-23T11:18:56.400 INFO:teuthology.orchestra.run.trial030.stdout: ceph-osd-crimson-2:20.3.0-5439.ge371433f.el9.x86_64 2026-02-23T11:18:56.400 INFO:teuthology.orchestra.run.trial030.stdout: ceph-osd-crimson-debuginfo-2:20.3.0-5439.ge371433f.el9.x86_64 2026-02-23T11:18:56.400 INFO:teuthology.orchestra.run.trial030.stdout: ceph-prometheus-alerts-2:20.3.0-5439.ge371433f.el9.noarch 2026-02-23T11:18:56.400 INFO:teuthology.orchestra.run.trial030.stdout: ceph-radosgw-2:20.3.0-5439.ge371433f.el9.x86_64 2026-02-23T11:18:56.400 INFO:teuthology.orchestra.run.trial030.stdout: ceph-selinux-2:20.3.0-5439.ge371433f.el9.x86_64 2026-02-23T11:18:56.400 INFO:teuthology.orchestra.run.trial030.stdout: ceph-test-2:20.3.0-5439.ge371433f.el9.x86_64 2026-02-23T11:18:56.400 INFO:teuthology.orchestra.run.trial030.stdout: ceph-volume-2:20.3.0-5439.ge371433f.el9.noarch 2026-02-23T11:18:56.401 INFO:teuthology.orchestra.run.trial030.stdout: cephadm-2:20.3.0-5439.ge371433f.el9.noarch 2026-02-23T11:18:56.401 INFO:teuthology.orchestra.run.trial030.stdout: cryptsetup-2.8.1-3.el9.x86_64 2026-02-23T11:18:56.401 INFO:teuthology.orchestra.run.trial030.stdout: flexiblas-3.0.4-9.el9.x86_64 2026-02-23T11:18:56.401 INFO:teuthology.orchestra.run.trial030.stdout: flexiblas-netlib-3.0.4-9.el9.x86_64 2026-02-23T11:18:56.401 INFO:teuthology.orchestra.run.trial030.stdout: flexiblas-openblas-openmp-3.0.4-9.el9.x86_64 2026-02-23T11:18:56.401 INFO:teuthology.orchestra.run.trial030.stdout: gperftools-libs-2.9.1-3.el9.x86_64 2026-02-23T11:18:56.401 INFO:teuthology.orchestra.run.trial030.stdout: grpc-data-1.46.7-10.el9.noarch 2026-02-23T11:18:56.401 INFO:teuthology.orchestra.run.trial030.stdout: hwloc-libs-2.4.1-6.el9.x86_64 2026-02-23T11:18:56.401 INFO:teuthology.orchestra.run.trial030.stdout: ledmon-libs-1.1.0-3.el9.x86_64 2026-02-23T11:18:56.401 INFO:teuthology.orchestra.run.trial030.stdout: libarrow-9.0.0-15.el9.x86_64 2026-02-23T11:18:56.401 INFO:teuthology.orchestra.run.trial030.stdout: libarrow-doc-9.0.0-15.el9.noarch 2026-02-23T11:18:56.401 INFO:teuthology.orchestra.run.trial030.stdout: libasan8-15.2.1-7.el9.x86_64 2026-02-23T11:18:56.402 INFO:teuthology.orchestra.run.trial030.stdout: libatomic-11.5.0-14.el9.x86_64 2026-02-23T11:18:56.402 INFO:teuthology.orchestra.run.trial030.stdout: libcephfs-devel-2:20.3.0-5439.ge371433f.el9.x86_64 2026-02-23T11:18:56.402 INFO:teuthology.orchestra.run.trial030.stdout: libcephfs-proxy2-2:20.3.0-5439.ge371433f.el9.x86_64 2026-02-23T11:18:56.402 INFO:teuthology.orchestra.run.trial030.stdout: libcephfs2-2:20.3.0-5439.ge371433f.el9.x86_64 2026-02-23T11:18:56.402 INFO:teuthology.orchestra.run.trial030.stdout: libcephsqlite-2:20.3.0-5439.ge371433f.el9.x86_64 2026-02-23T11:18:56.402 INFO:teuthology.orchestra.run.trial030.stdout: libconfig-1.7.2-9.el9.x86_64 2026-02-23T11:18:56.402 INFO:teuthology.orchestra.run.trial030.stdout: libgfortran-11.5.0-14.el9.x86_64 2026-02-23T11:18:56.402 INFO:teuthology.orchestra.run.trial030.stdout: libnbd-1.20.3-4.el9.x86_64 2026-02-23T11:18:56.402 INFO:teuthology.orchestra.run.trial030.stdout: liboath-2.6.12-1.el9.x86_64 2026-02-23T11:18:56.402 INFO:teuthology.orchestra.run.trial030.stdout: libpmemobj-1.12.1-1.el9.x86_64 2026-02-23T11:18:56.402 INFO:teuthology.orchestra.run.trial030.stdout: libquadmath-11.5.0-14.el9.x86_64 2026-02-23T11:18:56.402 INFO:teuthology.orchestra.run.trial030.stdout: librabbitmq-0.11.0-7.el9.x86_64 2026-02-23T11:18:56.403 INFO:teuthology.orchestra.run.trial030.stdout: librados-devel-2:20.3.0-5439.ge371433f.el9.x86_64 2026-02-23T11:18:56.403 INFO:teuthology.orchestra.run.trial030.stdout: libradosstriper1-2:20.3.0-5439.ge371433f.el9.x86_64 2026-02-23T11:18:56.403 INFO:teuthology.orchestra.run.trial030.stdout: librdkafka-1.6.1-102.el9.x86_64 2026-02-23T11:18:56.403 INFO:teuthology.orchestra.run.trial030.stdout: librgw2-2:20.3.0-5439.ge371433f.el9.x86_64 2026-02-23T11:18:56.403 INFO:teuthology.orchestra.run.trial030.stdout: libstoragemgmt-1.10.1-1.el9.x86_64 2026-02-23T11:18:56.403 INFO:teuthology.orchestra.run.trial030.stdout: libubsan-11.5.0-14.el9.x86_64 2026-02-23T11:18:56.403 INFO:teuthology.orchestra.run.trial030.stdout: libunwind-1.6.2-1.el9.x86_64 2026-02-23T11:18:56.403 INFO:teuthology.orchestra.run.trial030.stdout: libxslt-1.1.34-12.el9.x86_64 2026-02-23T11:18:56.403 INFO:teuthology.orchestra.run.trial030.stdout: lksctp-tools-1.0.19-2.el9.x86_64 2026-02-23T11:18:56.403 INFO:teuthology.orchestra.run.trial030.stdout: lttng-ust-2.12.0-6.el9.x86_64 2026-02-23T11:18:56.403 INFO:teuthology.orchestra.run.trial030.stdout: lua-5.4.4-4.el9.x86_64 2026-02-23T11:18:56.403 INFO:teuthology.orchestra.run.trial030.stdout: lua-devel-5.4.4-4.el9.x86_64 2026-02-23T11:18:56.403 INFO:teuthology.orchestra.run.trial030.stdout: luarocks-3.9.2-5.el9.noarch 2026-02-23T11:18:56.404 INFO:teuthology.orchestra.run.trial030.stdout: mailcap-2.1.49-5.el9.noarch 2026-02-23T11:18:56.404 INFO:teuthology.orchestra.run.trial030.stdout: openblas-0.3.29-1.el9.x86_64 2026-02-23T11:18:56.404 INFO:teuthology.orchestra.run.trial030.stdout: openblas-openmp-0.3.29-1.el9.x86_64 2026-02-23T11:18:56.404 INFO:teuthology.orchestra.run.trial030.stdout: parquet-libs-9.0.0-15.el9.x86_64 2026-02-23T11:18:56.404 INFO:teuthology.orchestra.run.trial030.stdout: pciutils-3.7.0-7.el9.x86_64 2026-02-23T11:18:56.404 INFO:teuthology.orchestra.run.trial030.stdout: protobuf-3.14.0-17.el9.x86_64 2026-02-23T11:18:56.404 INFO:teuthology.orchestra.run.trial030.stdout: protobuf-compiler-3.14.0-17.el9.x86_64 2026-02-23T11:18:56.404 INFO:teuthology.orchestra.run.trial030.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2026-02-23T11:18:56.404 INFO:teuthology.orchestra.run.trial030.stdout: python3-autocommand-2.2.2-8.el9.noarch 2026-02-23T11:18:56.404 INFO:teuthology.orchestra.run.trial030.stdout: python3-babel-2.9.1-2.el9.noarch 2026-02-23T11:18:56.404 INFO:teuthology.orchestra.run.trial030.stdout: python3-backports-tarfile-1.2.0-1.el9.noarch 2026-02-23T11:18:56.404 INFO:teuthology.orchestra.run.trial030.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2026-02-23T11:18:56.404 INFO:teuthology.orchestra.run.trial030.stdout: python3-cachetools-4.2.4-1.el9.noarch 2026-02-23T11:18:56.405 INFO:teuthology.orchestra.run.trial030.stdout: python3-ceph-argparse-2:20.3.0-5439.ge371433f.el9.x86_64 2026-02-23T11:18:56.405 INFO:teuthology.orchestra.run.trial030.stdout: python3-ceph-common-2:20.3.0-5439.ge371433f.el9.x86_64 2026-02-23T11:18:56.405 INFO:teuthology.orchestra.run.trial030.stdout: python3-cephfs-2:20.3.0-5439.ge371433f.el9.x86_64 2026-02-23T11:18:56.405 INFO:teuthology.orchestra.run.trial030.stdout: python3-certifi-2023.05.07-4.el9.noarch 2026-02-23T11:18:56.405 INFO:teuthology.orchestra.run.trial030.stdout: python3-cffi-1.14.5-5.el9.x86_64 2026-02-23T11:18:56.405 INFO:teuthology.orchestra.run.trial030.stdout: python3-cheroot-10.0.1-4.el9.noarch 2026-02-23T11:18:56.405 INFO:teuthology.orchestra.run.trial030.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2026-02-23T11:18:56.405 INFO:teuthology.orchestra.run.trial030.stdout: python3-cryptography-36.0.1-5.el9.x86_64 2026-02-23T11:18:56.405 INFO:teuthology.orchestra.run.trial030.stdout: python3-devel-3.9.25-3.el9.x86_64 2026-02-23T11:18:56.405 INFO:teuthology.orchestra.run.trial030.stdout: python3-google-auth-1:2.45.0-1.el9.noarch 2026-02-23T11:18:56.405 INFO:teuthology.orchestra.run.trial030.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2026-02-23T11:18:56.405 INFO:teuthology.orchestra.run.trial030.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2026-02-23T11:18:56.405 INFO:teuthology.orchestra.run.trial030.stdout: python3-iniconfig-1.1.1-7.el9.noarch 2026-02-23T11:18:56.406 INFO:teuthology.orchestra.run.trial030.stdout: python3-jaraco-8.2.1-3.el9.noarch 2026-02-23T11:18:56.406 INFO:teuthology.orchestra.run.trial030.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2026-02-23T11:18:56.406 INFO:teuthology.orchestra.run.trial030.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2026-02-23T11:18:56.406 INFO:teuthology.orchestra.run.trial030.stdout: python3-jaraco-context-6.0.1-3.el9.noarch 2026-02-23T11:18:56.406 INFO:teuthology.orchestra.run.trial030.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2026-02-23T11:18:56.406 INFO:teuthology.orchestra.run.trial030.stdout: python3-jaraco-text-4.0.0-2.el9.noarch 2026-02-23T11:18:56.406 INFO:teuthology.orchestra.run.trial030.stdout: python3-jinja2-2.11.3-8.el9.noarch 2026-02-23T11:18:56.406 INFO:teuthology.orchestra.run.trial030.stdout: python3-jmespath-1.0.1-1.el9.noarch 2026-02-23T11:18:56.406 INFO:teuthology.orchestra.run.trial030.stdout: python3-kubernetes-1:26.1.0-3.el9.noarch 2026-02-23T11:18:56.406 INFO:teuthology.orchestra.run.trial030.stdout: python3-libstoragemgmt-1.10.1-1.el9.x86_64 2026-02-23T11:18:56.406 INFO:teuthology.orchestra.run.trial030.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2026-02-23T11:18:56.406 INFO:teuthology.orchestra.run.trial030.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2026-02-23T11:18:56.406 INFO:teuthology.orchestra.run.trial030.stdout: python3-natsort-7.1.1-5.el9.noarch 2026-02-23T11:18:56.407 INFO:teuthology.orchestra.run.trial030.stdout: python3-numpy-1:1.23.5-2.el9.x86_64 2026-02-23T11:18:56.407 INFO:teuthology.orchestra.run.trial030.stdout: python3-numpy-f2py-1:1.23.5-2.el9.x86_64 2026-02-23T11:18:56.407 INFO:teuthology.orchestra.run.trial030.stdout: python3-packaging-20.9-5.el9.noarch 2026-02-23T11:18:56.407 INFO:teuthology.orchestra.run.trial030.stdout: python3-pluggy-0.13.1-7.el9.noarch 2026-02-23T11:18:56.407 INFO:teuthology.orchestra.run.trial030.stdout: python3-ply-3.11-14.el9.noarch 2026-02-23T11:18:56.407 INFO:teuthology.orchestra.run.trial030.stdout: python3-portend-3.1.0-2.el9.noarch 2026-02-23T11:18:56.407 INFO:teuthology.orchestra.run.trial030.stdout: python3-protobuf-3.14.0-17.el9.noarch 2026-02-23T11:18:56.407 INFO:teuthology.orchestra.run.trial030.stdout: python3-py-1.10.0-6.el9.noarch 2026-02-23T11:18:56.407 INFO:teuthology.orchestra.run.trial030.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2026-02-23T11:18:56.407 INFO:teuthology.orchestra.run.trial030.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2026-02-23T11:18:56.407 INFO:teuthology.orchestra.run.trial030.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2026-02-23T11:18:56.407 INFO:teuthology.orchestra.run.trial030.stdout: python3-pycparser-2.20-6.el9.noarch 2026-02-23T11:18:56.407 INFO:teuthology.orchestra.run.trial030.stdout: python3-pyparsing-2.4.7-9.el9.noarch 2026-02-23T11:18:56.408 INFO:teuthology.orchestra.run.trial030.stdout: python3-pytest-6.2.2-7.el9.noarch 2026-02-23T11:18:56.408 INFO:teuthology.orchestra.run.trial030.stdout: python3-rados-2:20.3.0-5439.ge371433f.el9.x86_64 2026-02-23T11:18:56.408 INFO:teuthology.orchestra.run.trial030.stdout: python3-rbd-2:20.3.0-5439.ge371433f.el9.x86_64 2026-02-23T11:18:56.408 INFO:teuthology.orchestra.run.trial030.stdout: python3-repoze-lru-0.7-16.el9.noarch 2026-02-23T11:18:56.408 INFO:teuthology.orchestra.run.trial030.stdout: python3-requests-2.25.1-10.el9.noarch 2026-02-23T11:18:56.408 INFO:teuthology.orchestra.run.trial030.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2026-02-23T11:18:56.408 INFO:teuthology.orchestra.run.trial030.stdout: python3-rgw-2:20.3.0-5439.ge371433f.el9.x86_64 2026-02-23T11:18:56.408 INFO:teuthology.orchestra.run.trial030.stdout: python3-routes-2.5.1-5.el9.noarch 2026-02-23T11:18:56.408 INFO:teuthology.orchestra.run.trial030.stdout: python3-rsa-4.9-2.el9.noarch 2026-02-23T11:18:56.408 INFO:teuthology.orchestra.run.trial030.stdout: python3-scipy-1.9.3-2.el9.x86_64 2026-02-23T11:18:56.408 INFO:teuthology.orchestra.run.trial030.stdout: python3-tempora-5.0.0-2.el9.noarch 2026-02-23T11:18:56.408 INFO:teuthology.orchestra.run.trial030.stdout: python3-toml-0.10.2-6.el9.noarch 2026-02-23T11:18:56.409 INFO:teuthology.orchestra.run.trial030.stdout: python3-typing-extensions-4.15.0-1.el9.noarch 2026-02-23T11:18:56.409 INFO:teuthology.orchestra.run.trial030.stdout: python3-urllib3-1.26.5-7.el9.noarch 2026-02-23T11:18:56.409 INFO:teuthology.orchestra.run.trial030.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2026-02-23T11:18:56.409 INFO:teuthology.orchestra.run.trial030.stdout: python3-xmltodict-0.12.0-15.el9.noarch 2026-02-23T11:18:56.409 INFO:teuthology.orchestra.run.trial030.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2026-02-23T11:18:56.409 INFO:teuthology.orchestra.run.trial030.stdout: qatlib-25.08.0-2.el9.x86_64 2026-02-23T11:18:56.409 INFO:teuthology.orchestra.run.trial030.stdout: qatlib-service-25.08.0-2.el9.x86_64 2026-02-23T11:18:56.409 INFO:teuthology.orchestra.run.trial030.stdout: qatzip-libs-1.3.1-1.el9.x86_64 2026-02-23T11:18:56.409 INFO:teuthology.orchestra.run.trial030.stdout: rbd-fuse-2:20.3.0-5439.ge371433f.el9.x86_64 2026-02-23T11:18:56.409 INFO:teuthology.orchestra.run.trial030.stdout: rbd-mirror-2:20.3.0-5439.ge371433f.el9.x86_64 2026-02-23T11:18:56.409 INFO:teuthology.orchestra.run.trial030.stdout: rbd-nbd-2:20.3.0-5439.ge371433f.el9.x86_64 2026-02-23T11:18:56.409 INFO:teuthology.orchestra.run.trial030.stdout: re2-1:20211101-20.el9.x86_64 2026-02-23T11:18:56.409 INFO:teuthology.orchestra.run.trial030.stdout: socat-1.7.4.1-8.el9.x86_64 2026-02-23T11:18:56.410 INFO:teuthology.orchestra.run.trial030.stdout: thrift-0.15.0-4.el9.x86_64 2026-02-23T11:18:56.410 INFO:teuthology.orchestra.run.trial030.stdout: unzip-6.0-59.el9.x86_64 2026-02-23T11:18:56.410 INFO:teuthology.orchestra.run.trial030.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2026-02-23T11:18:56.410 INFO:teuthology.orchestra.run.trial030.stdout: yaml-cpp-0.6.3-5.el9.x86_64 2026-02-23T11:18:56.410 INFO:teuthology.orchestra.run.trial030.stdout: zip-3.0-35.el9.x86_64 2026-02-23T11:18:56.410 INFO:teuthology.orchestra.run.trial030.stdout: 2026-02-23T11:18:56.410 INFO:teuthology.orchestra.run.trial030.stdout:Complete! 2026-02-23T11:18:56.542 DEBUG:teuthology.parallel:result is None 2026-02-23T11:18:56.543 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=debug&distros=centos%2F9%2Fx86_64&sha1=e371433fcd1390a0b82cfb18effc422c1f5d8c1d 2026-02-23T11:18:56.725 DEBUG:teuthology.orchestra.run.trial030:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2026-02-23T11:18:56.747 INFO:teuthology.orchestra.run.trial030.stdout:20.3.0-5439.ge371433f.el9 2026-02-23T11:18:56.748 INFO:teuthology.packaging:The installed version of ceph is 20.3.0-5439.ge371433f.el9 2026-02-23T11:18:56.748 INFO:teuthology.task.install:The correct ceph version 20.3.0-5439.ge371433f is installed. 2026-02-23T11:18:56.751 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2026-02-23T11:18:56.751 DEBUG:teuthology.orchestra.run.trial030:> set -ex 2026-02-23T11:18:56.751 DEBUG:teuthology.orchestra.run.trial030:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2026-02-23T11:18:56.815 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2026-02-23T11:18:56.815 DEBUG:teuthology.orchestra.run.trial030:> set -ex 2026-02-23T11:18:56.815 DEBUG:teuthology.orchestra.run.trial030:> sudo dd of=/usr/bin/daemon-helper 2026-02-23T11:18:56.880 DEBUG:teuthology.orchestra.run.trial030:> sudo chmod a=rx -- /usr/bin/daemon-helper 2026-02-23T11:18:56.943 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2026-02-23T11:18:56.944 DEBUG:teuthology.orchestra.run.trial030:> set -ex 2026-02-23T11:18:56.944 DEBUG:teuthology.orchestra.run.trial030:> sudo dd of=/usr/bin/adjust-ulimits 2026-02-23T11:18:57.007 DEBUG:teuthology.orchestra.run.trial030:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2026-02-23T11:18:57.071 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2026-02-23T11:18:57.071 DEBUG:teuthology.orchestra.run.trial030:> set -ex 2026-02-23T11:18:57.072 DEBUG:teuthology.orchestra.run.trial030:> sudo dd of=/usr/bin/stdin-killer 2026-02-23T11:18:57.135 DEBUG:teuthology.orchestra.run.trial030:> sudo chmod a=rx -- /usr/bin/stdin-killer 2026-02-23T11:18:57.199 INFO:teuthology.run_tasks:Running task pexec... 2026-02-23T11:18:57.220 INFO:teuthology.task.pexec:Executing custom commands... 2026-02-23T11:18:57.221 DEBUG:teuthology.orchestra.run.trial030:> TESTDIR=/home/ubuntu/cephtest bash -s 2026-02-23T11:18:57.242 INFO:teuthology.task.pexec:Running commands on host ubuntu@trial030.front.sepia.ceph.com 2026-02-23T11:18:57.242 INFO:teuthology.task.pexec:sudo update-alternatives --set ceph-osd /usr/bin/ceph-osd-crimson 2026-02-23T11:18:57.293 DEBUG:teuthology.parallel:result is None 2026-02-23T11:18:57.293 INFO:teuthology.run_tasks:Running task install... 2026-02-23T11:18:57.308 DEBUG:teuthology.task.install:project ceph 2026-02-23T11:18:57.308 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'extra_system_packages': ['python3-pytest'], 'flavor': 'debug', 'sha1': 'e371433fcd1390a0b82cfb18effc422c1f5d8c1d'}} 2026-02-23T11:18:57.308 DEBUG:teuthology.task.install:config {'extra_system_packages': ['python3-pytest'], 'flavor': 'debug', 'sha1': 'e371433fcd1390a0b82cfb18effc422c1f5d8c1d'} 2026-02-23T11:18:57.308 INFO:teuthology.task.install:Using flavor: debug 2026-02-23T11:18:57.316 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-osd', 'ceph-osd-classic', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-osd', 'ceph-osd-classic', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2026-02-23T11:18:57.317 INFO:teuthology.task.install:extra packages: [] 2026-02-23T11:18:57.317 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': ['python3-pytest'], 'extras': None, 'enable_coprs': [], 'flavor': 'debug', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': 'e371433fcd1390a0b82cfb18effc422c1f5d8c1d', 'tag': None, 'wait_for_package': False} 2026-02-23T11:18:57.317 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=debug&distros=centos%2F9%2Fx86_64&sha1=e371433fcd1390a0b82cfb18effc422c1f5d8c1d 2026-02-23T11:18:57.478 INFO:teuthology.task.install.rpm:Pulling from https://3.chacra.ceph.com/r/ceph/wip-shraddhaag-enable-cephadm-tests/e371433fcd1390a0b82cfb18effc422c1f5d8c1d/centos/9/flavors/debug/ 2026-02-23T11:18:57.478 INFO:teuthology.task.install.rpm:Package version is 20.3.0-5439.ge371433f 2026-02-23T11:18:57.854 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://3.chacra.ceph.com/r/ceph/wip-shraddhaag-enable-cephadm-tests/e371433fcd1390a0b82cfb18effc422c1f5d8c1d/centos/9/flavors/debug/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://3.chacra.ceph.com/r/ceph/wip-shraddhaag-enable-cephadm-tests/e371433fcd1390a0b82cfb18effc422c1f5d8c1d/centos/9/flavors/debug/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://3.chacra.ceph.com/r/ceph/wip-shraddhaag-enable-cephadm-tests/e371433fcd1390a0b82cfb18effc422c1f5d8c1d/centos/9/flavors/debug/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2026-02-23T11:18:57.855 DEBUG:teuthology.orchestra.run.trial030:> set -ex 2026-02-23T11:18:57.855 DEBUG:teuthology.orchestra.run.trial030:> sudo dd of=/etc/yum.repos.d/ceph.repo 2026-02-23T11:18:57.880 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-osd, ceph-osd-classic, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-pytest on remote rpm x86_64 2026-02-23T11:18:57.880 DEBUG:teuthology.orchestra.run.trial030:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/e371433fcd1390a0b82cfb18effc422c1f5d8c1d/;g' /etc/yum.repos.d/ceph.repo ; fi 2026-02-23T11:18:57.944 DEBUG:teuthology.orchestra.run.trial030:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2026-02-23T11:18:58.008 DEBUG:teuthology.orchestra.run.trial030:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2026-02-23T11:18:58.063 INFO:teuthology.orchestra.run.trial030.stdout:check_obsoletes = 1 2026-02-23T11:18:58.073 DEBUG:teuthology.orchestra.run.trial030:> sudo yum clean all 2026-02-23T11:18:58.232 INFO:teuthology.orchestra.run.trial030.stdout:56 files removed 2026-02-23T11:18:58.246 DEBUG:teuthology.orchestra.run.trial030:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-osd ceph-osd-classic ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-pytest 2026-02-23T11:18:59.115 INFO:teuthology.orchestra.run.trial030.stdout:ceph packages for x86_64 127 kB/s | 92 kB 00:00 2026-02-23T11:18:59.845 INFO:teuthology.orchestra.run.trial030.stdout:ceph noarch packages 34 kB/s | 24 kB 00:00 2026-02-23T11:19:00.408 INFO:teuthology.orchestra.run.trial030.stdout:ceph source packages 3.8 kB/s | 2.1 kB 00:00 2026-02-23T11:19:01.139 INFO:teuthology.orchestra.run.trial030.stdout:CentOS Stream 9 - BaseOS 13 MB/s | 8.9 MB 00:00 2026-02-23T11:19:12.766 INFO:teuthology.orchestra.run.trial030.stdout:CentOS Stream 9 - AppStream 2.4 MB/s | 27 MB 00:11 2026-02-23T11:19:23.619 INFO:teuthology.orchestra.run.trial030.stdout:CentOS Stream 9 - CRB 936 kB/s | 7.8 MB 00:08 2026-02-23T11:19:25.346 INFO:teuthology.orchestra.run.trial030.stdout:CentOS Stream 9 - Extras packages 20 kB/s | 20 kB 00:01 2026-02-23T11:19:27.853 INFO:teuthology.orchestra.run.trial030.stdout:Extra Packages for Enterprise Linux 8.3 MB/s | 20 MB 00:02 2026-02-23T11:19:31.021 INFO:teuthology.orchestra.run.trial030.stdout:lab-extras 1.5 MB/s | 50 kB 00:00 2026-02-23T11:19:32.204 INFO:teuthology.orchestra.run.trial030.stdout:Package ceph-radosgw-2:20.3.0-5439.ge371433f.el9.x86_64 is already installed. 2026-02-23T11:19:32.205 INFO:teuthology.orchestra.run.trial030.stdout:Package ceph-test-2:20.3.0-5439.ge371433f.el9.x86_64 is already installed. 2026-02-23T11:19:32.205 INFO:teuthology.orchestra.run.trial030.stdout:Package ceph-2:20.3.0-5439.ge371433f.el9.x86_64 is already installed. 2026-02-23T11:19:32.205 INFO:teuthology.orchestra.run.trial030.stdout:Package ceph-base-2:20.3.0-5439.ge371433f.el9.x86_64 is already installed. 2026-02-23T11:19:32.206 INFO:teuthology.orchestra.run.trial030.stdout:Package cephadm-2:20.3.0-5439.ge371433f.el9.noarch is already installed. 2026-02-23T11:19:32.206 INFO:teuthology.orchestra.run.trial030.stdout:Package ceph-immutable-object-cache-2:20.3.0-5439.ge371433f.el9.x86_64 is already installed. 2026-02-23T11:19:32.206 INFO:teuthology.orchestra.run.trial030.stdout:Package ceph-mgr-2:20.3.0-5439.ge371433f.el9.x86_64 is already installed. 2026-02-23T11:19:32.207 INFO:teuthology.orchestra.run.trial030.stdout:Package ceph-mgr-dashboard-2:20.3.0-5439.ge371433f.el9.noarch is already installed. 2026-02-23T11:19:32.207 INFO:teuthology.orchestra.run.trial030.stdout:Package ceph-mgr-diskprediction-local-2:20.3.0-5439.ge371433f.el9.noarch is already installed. 2026-02-23T11:19:32.208 INFO:teuthology.orchestra.run.trial030.stdout:Package ceph-mgr-rook-2:20.3.0-5439.ge371433f.el9.noarch is already installed. 2026-02-23T11:19:32.208 INFO:teuthology.orchestra.run.trial030.stdout:Package ceph-mgr-cephadm-2:20.3.0-5439.ge371433f.el9.noarch is already installed. 2026-02-23T11:19:32.209 INFO:teuthology.orchestra.run.trial030.stdout:Package ceph-osd-2:20.3.0-5439.ge371433f.el9.x86_64 is already installed. 2026-02-23T11:19:32.209 INFO:teuthology.orchestra.run.trial030.stdout:Package ceph-osd-classic-2:20.3.0-5439.ge371433f.el9.x86_64 is already installed. 2026-02-23T11:19:32.209 INFO:teuthology.orchestra.run.trial030.stdout:Package ceph-fuse-2:20.3.0-5439.ge371433f.el9.x86_64 is already installed. 2026-02-23T11:19:32.210 INFO:teuthology.orchestra.run.trial030.stdout:Package ceph-volume-2:20.3.0-5439.ge371433f.el9.noarch is already installed. 2026-02-23T11:19:32.211 INFO:teuthology.orchestra.run.trial030.stdout:Package librados-devel-2:20.3.0-5439.ge371433f.el9.x86_64 is already installed. 2026-02-23T11:19:32.211 INFO:teuthology.orchestra.run.trial030.stdout:Package libcephfs2-2:20.3.0-5439.ge371433f.el9.x86_64 is already installed. 2026-02-23T11:19:32.212 INFO:teuthology.orchestra.run.trial030.stdout:Package libcephfs-devel-2:20.3.0-5439.ge371433f.el9.x86_64 is already installed. 2026-02-23T11:19:32.212 INFO:teuthology.orchestra.run.trial030.stdout:Package librados2-2:20.3.0-5439.ge371433f.el9.x86_64 is already installed. 2026-02-23T11:19:32.212 INFO:teuthology.orchestra.run.trial030.stdout:Package librbd1-2:20.3.0-5439.ge371433f.el9.x86_64 is already installed. 2026-02-23T11:19:32.213 INFO:teuthology.orchestra.run.trial030.stdout:Package python3-rados-2:20.3.0-5439.ge371433f.el9.x86_64 is already installed. 2026-02-23T11:19:32.213 INFO:teuthology.orchestra.run.trial030.stdout:Package python3-rgw-2:20.3.0-5439.ge371433f.el9.x86_64 is already installed. 2026-02-23T11:19:32.214 INFO:teuthology.orchestra.run.trial030.stdout:Package python3-cephfs-2:20.3.0-5439.ge371433f.el9.x86_64 is already installed. 2026-02-23T11:19:32.214 INFO:teuthology.orchestra.run.trial030.stdout:Package python3-rbd-2:20.3.0-5439.ge371433f.el9.x86_64 is already installed. 2026-02-23T11:19:32.214 INFO:teuthology.orchestra.run.trial030.stdout:Package rbd-fuse-2:20.3.0-5439.ge371433f.el9.x86_64 is already installed. 2026-02-23T11:19:32.215 INFO:teuthology.orchestra.run.trial030.stdout:Package rbd-mirror-2:20.3.0-5439.ge371433f.el9.x86_64 is already installed. 2026-02-23T11:19:32.215 INFO:teuthology.orchestra.run.trial030.stdout:Package rbd-nbd-2:20.3.0-5439.ge371433f.el9.x86_64 is already installed. 2026-02-23T11:19:32.215 INFO:teuthology.orchestra.run.trial030.stdout:Package python3-pytest-6.2.2-7.el9.noarch is already installed. 2026-02-23T11:19:32.234 INFO:teuthology.orchestra.run.trial030.stdout:Dependencies resolved. 2026-02-23T11:19:32.235 INFO:teuthology.orchestra.run.trial030.stdout:Nothing to do. 2026-02-23T11:19:32.235 INFO:teuthology.orchestra.run.trial030.stdout:Complete! 2026-02-23T11:19:32.260 DEBUG:teuthology.parallel:result is None 2026-02-23T11:19:32.261 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=debug&distros=centos%2F9%2Fx86_64&sha1=e371433fcd1390a0b82cfb18effc422c1f5d8c1d 2026-02-23T11:19:32.432 DEBUG:teuthology.orchestra.run.trial030:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2026-02-23T11:19:32.453 INFO:teuthology.orchestra.run.trial030.stdout:20.3.0-5439.ge371433f.el9 2026-02-23T11:19:32.453 INFO:teuthology.packaging:The installed version of ceph is 20.3.0-5439.ge371433f.el9 2026-02-23T11:19:32.453 INFO:teuthology.task.install:The correct ceph version 20.3.0-5439.ge371433f is installed. 2026-02-23T11:19:32.455 INFO:teuthology.task.install.util:Utilities already shipped, skip it... 2026-02-23T11:19:32.455 INFO:teuthology.run_tasks:Running task ceph... 2026-02-23T11:19:32.551 INFO:tasks.ceph:Making ceph log dir writeable by non-root... 2026-02-23T11:19:32.551 DEBUG:teuthology.orchestra.run.trial030:> sudo chmod 777 /var/log/ceph 2026-02-23T11:19:32.576 INFO:tasks.ceph:Disabling ceph logrotate... 2026-02-23T11:19:32.576 DEBUG:teuthology.orchestra.run.trial030:> sudo rm -f -- /etc/logrotate.d/ceph 2026-02-23T11:19:32.639 INFO:tasks.ceph:Creating extra log directories... 2026-02-23T11:19:32.639 DEBUG:teuthology.orchestra.run.trial030:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2026-02-23T11:19:32.714 INFO:tasks.ceph:Creating ceph cluster ceph... 2026-02-23T11:19:32.714 INFO:tasks.ceph:config {'conf': {'mon': {'mon min osdmap epochs': 50, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'paxos service trim min': 10, 'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'osd pool default crimson': True}, 'osd': {'debug monc': 20, 'crimson cpu num': 4, 'crimson osd obc lru size': 10, 'debug ms': 20, 'debug osd': 20, 'debug seastore': 20, 'debug seastore backref': 20, 'debug seastore cache': 20, 'debug seastore cleaner': 20, 'debug seastore device': 20, 'debug seastore epm': 20, 'debug seastore fixedkv tree': 20, 'debug seastore journal': 20, 'debug seastore lba': 20, 'debug seastore odata': 20, 'debug seastore omap': 20, 'debug seastore onode': 20, 'debug seastore t': 20, 'debug seastore tm': 20, 'osd objectstore': 'seastore', 'osd shutdown pgref assert': True, 'seastore cachepin size pershard': '64M', 'seastore cachepin type': '2Q', 'seastore max concurrent transactions': 8}, 'global': {'enable experimental unrecoverable data corrupting features': 'crimson', 'ms client mode': 'crc', 'ms cluster mode': 'crc', 'ms mon client mode': 'crc', 'ms mon cluster mode': 'crc', 'ms mon service mode': 'crc', 'ms service mode': 'crc'}, 'mgr': {'debug mgr': 20, 'debug ms': 1}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(POOL_APP_NOT_ENABLED\\)', '\\(SLOW_OPS\\)', 'slow request'], 'cpu_profile': set(), 'cluster': 'ceph', 'mon_bind_msgr2': True, 'mon_bind_addrvec': True} 2026-02-23T11:19:32.714 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/mnt/teuthology/shraddhaag-2026-02-23_09:58:48-crimson-rados-main-distro-debug-trial/66207', 'branch': 'main', 'description': 'crimson-rados/rbd/{0-crimson_install clusters/crimson-fixed-1 crimson-supported-all-distro/centos_latest crimson_qa_overrides deploy/ceph objectstore/seastore/segmented$/{crimson_seastore_segmented_2q} tasks/rbd_python_api_tests_old_format}', 'email': None, 'first_in_suite': False, 'flavor': 'debug', 'job_id': '66207', 'kernel': {'branch': 'distro', 'kdb': True, 'sha1': 'distro'}, 'last_in_suite': False, 'machine_type': 'trial', 'name': 'shraddhaag-2026-02-23_09:58:48-crimson-rados-main-distro-debug-trial', 'no_nested_subset': False, 'os_type': 'centos', 'os_version': '9.stream', 'overrides': {'admin_socket': {'branch': 'main'}, 'ceph': {'conf': {'global': {'enable experimental unrecoverable data corrupting features': 'crimson', 'ms client mode': 'crc', 'ms cluster mode': 'crc', 'ms mon client mode': 'crc', 'ms mon cluster mode': 'crc', 'ms mon service mode': 'crc', 'ms service mode': 'crc'}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'osd pool default crimson': True}, 'osd': {'crimson cpu num': 4, 'crimson osd obc lru size': 10, 'debug ms': 20, 'debug osd': 20, 'debug seastore': 20, 'debug seastore backref': 20, 'debug seastore cache': 20, 'debug seastore cleaner': 20, 'debug seastore device': 20, 'debug seastore epm': 20, 'debug seastore fixedkv tree': 20, 'debug seastore journal': 20, 'debug seastore lba': 20, 'debug seastore odata': 20, 'debug seastore omap': 20, 'debug seastore onode': 20, 'debug seastore t': 20, 'debug seastore tm': 20, 'osd objectstore': 'seastore', 'osd shutdown pgref assert': True, 'seastore cachepin size pershard': '64M', 'seastore cachepin type': '2Q', 'seastore max concurrent transactions': 8}}, 'crimson_compat': True, 'flavor': 'debug', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(POOL_APP_NOT_ENABLED\\)', '\\(SLOW_OPS\\)', 'slow request'], 'sha1': 'e371433fcd1390a0b82cfb18effc422c1f5d8c1d'}, 'ceph-deploy': {'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'global': {'osd crush chooseleaf type': 0, 'osd pool default pg num': 128, 'osd pool default pgp num': 128, 'osd pool default size': 2}, 'mon': {}}}, 'install': {'ceph': {'extra_system_packages': ['python3-pytest'], 'flavor': 'debug', 'sha1': 'e371433fcd1390a0b82cfb18effc422c1f5d8c1d'}}, 'workunit': {'branch': 'wip-shraddhaag-enable-cephadm-tests', 'env': {'CRIMSON_COMPAT': '1'}, 'sha1': 'e371433fcd1390a0b82cfb18effc422c1f5d8c1d'}}, 'owner': 'scheduled_shraddhaag@soko04.front.sepia.ceph.com', 'priority': 49, 'repo': 'https://git.ceph.com/ceph-ci.git', 'roles': [['mon.a', 'mgr.x', 'osd.0', 'osd.1', 'osd.2', 'client.0']], 'seed': 66, 'sha1': 'e371433fcd1390a0b82cfb18effc422c1f5d8c1d', 'sleep_before_teardown': 0, 'suite': 'crimson-rados', 'suite_branch': 'wip-shraddhaag-enable-cephadm-tests', 'suite_path': '/home/teuthworker/src/github.com_shraddhaag_ceph_e371433fcd1390a0b82cfb18effc422c1f5d8c1d/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://github.com/shraddhaag/ceph.git', 'suite_sha1': 'e371433fcd1390a0b82cfb18effc422c1f5d8c1d', 'targets': {'trial030.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBJTQTYanJEsMxlFPAGNwdTzaNEO9mmBJiTGJf/qdsjP35Ic4gX1XnBG/0B2NzQslEsCQApuXOnrtX5njotcds4Q='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'branch': 'distro', 'kdb': True, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': {'extra_packages': {'deb': ['ceph-osd-crimson', 'ceph-osd-crimson-dbg'], 'rpm': ['ceph-osd-crimson', 'ceph-osd-crimson-debuginfo']}, 'extra_system_packages': ['python3-pytest'], 'flavor': 'debug', 'sha1': 'e371433fcd1390a0b82cfb18effc422c1f5d8c1d'}}, {'pexec': {'all': ['sudo update-alternatives --set ceph-osd /usr/bin/ceph-osd-crimson']}}, {'install': None}, {'ceph': {'conf': {'mon': {'mon min osdmap epochs': 50, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'paxos service trim min': 10, 'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'osd pool default crimson': True}, 'osd': {'debug monc': 20, 'crimson cpu num': 4, 'crimson osd obc lru size': 10, 'debug ms': 20, 'debug osd': 20, 'debug seastore': 20, 'debug seastore backref': 20, 'debug seastore cache': 20, 'debug seastore cleaner': 20, 'debug seastore device': 20, 'debug seastore epm': 20, 'debug seastore fixedkv tree': 20, 'debug seastore journal': 20, 'debug seastore lba': 20, 'debug seastore odata': 20, 'debug seastore omap': 20, 'debug seastore onode': 20, 'debug seastore t': 20, 'debug seastore tm': 20, 'osd objectstore': 'seastore', 'osd shutdown pgref assert': True, 'seastore cachepin size pershard': '64M', 'seastore cachepin type': '2Q', 'seastore max concurrent transactions': 8}, 'global': {'enable experimental unrecoverable data corrupting features': 'crimson', 'ms client mode': 'crc', 'ms cluster mode': 'crc', 'ms mon client mode': 'crc', 'ms mon cluster mode': 'crc', 'ms mon service mode': 'crc', 'ms service mode': 'crc'}, 'mgr': {'debug mgr': 20, 'debug ms': 1}}, 'crimson_compat': True, 'flavor': 'debug', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(POOL_APP_NOT_ENABLED\\)', '\\(SLOW_OPS\\)', 'slow request'], 'sha1': 'e371433fcd1390a0b82cfb18effc422c1f5d8c1d', 'cluster': 'ceph'}}, {'workunit': {'clients': {'client.0': ['rbd/test_librbd_python.sh']}}}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': []}, 'teuthology_branch': 'main', 'teuthology_sha1': 'fa17720d0088c3ac28e473468bfc79eeaff5cd38', 'timestamp': '2026-02-23_09:58:48', 'tube': 'trial', 'user': 'shraddhaag', 'verbose': False, 'worker_log': '/home/teuthworker/mnt/teuthology/worker_logs/dispatcher.trial.3318653'} 2026-02-23T11:19:32.715 DEBUG:teuthology.orchestra.run.trial030:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2026-02-23T11:19:32.759 DEBUG:teuthology.orchestra.run.trial030:> sudo install -d -m0777 -- /var/run/ceph 2026-02-23T11:19:32.823 DEBUG:teuthology.orchestra.run.trial030:> set -ex 2026-02-23T11:19:32.823 DEBUG:teuthology.orchestra.run.trial030:> dd if=/scratch_devs of=/dev/stdout 2026-02-23T11:19:32.881 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2026-02-23T11:19:32.881 DEBUG:teuthology.orchestra.run.trial030:> stat /dev/vg_nvme/lv_1 2026-02-23T11:19:32.937 INFO:teuthology.orchestra.run.trial030.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2026-02-23T11:19:32.937 INFO:teuthology.orchestra.run.trial030.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2026-02-23T11:19:32.937 INFO:teuthology.orchestra.run.trial030.stdout:Device: 6h/6d Inode: 908 Links: 1 2026-02-23T11:19:32.938 INFO:teuthology.orchestra.run.trial030.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2026-02-23T11:19:32.938 INFO:teuthology.orchestra.run.trial030.stdout:Context: system_u:object_r:device_t:s0 2026-02-23T11:19:32.938 INFO:teuthology.orchestra.run.trial030.stdout:Access: 2026-02-23 11:18:52.933618003 +0000 2026-02-23T11:19:32.938 INFO:teuthology.orchestra.run.trial030.stdout:Modify: 2026-02-23 11:12:22.883206965 +0000 2026-02-23T11:19:32.938 INFO:teuthology.orchestra.run.trial030.stdout:Change: 2026-02-23 11:12:22.883206965 +0000 2026-02-23T11:19:32.938 INFO:teuthology.orchestra.run.trial030.stdout: Birth: 2026-02-23 11:12:22.883206965 +0000 2026-02-23T11:19:32.938 DEBUG:teuthology.orchestra.run.trial030:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2026-02-23T11:19:33.001 INFO:teuthology.orchestra.run.trial030.stderr:1+0 records in 2026-02-23T11:19:33.002 INFO:teuthology.orchestra.run.trial030.stderr:1+0 records out 2026-02-23T11:19:33.002 INFO:teuthology.orchestra.run.trial030.stderr:512 bytes copied, 0.00014539 s, 3.5 MB/s 2026-02-23T11:19:33.003 DEBUG:teuthology.orchestra.run.trial030:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2026-02-23T11:19:33.060 DEBUG:teuthology.orchestra.run.trial030:> stat /dev/vg_nvme/lv_2 2026-02-23T11:19:33.116 INFO:teuthology.orchestra.run.trial030.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2026-02-23T11:19:33.116 INFO:teuthology.orchestra.run.trial030.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2026-02-23T11:19:33.117 INFO:teuthology.orchestra.run.trial030.stdout:Device: 6h/6d Inode: 939 Links: 1 2026-02-23T11:19:33.117 INFO:teuthology.orchestra.run.trial030.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2026-02-23T11:19:33.117 INFO:teuthology.orchestra.run.trial030.stdout:Context: system_u:object_r:device_t:s0 2026-02-23T11:19:33.117 INFO:teuthology.orchestra.run.trial030.stdout:Access: 2026-02-23 11:18:52.933618003 +0000 2026-02-23T11:19:33.117 INFO:teuthology.orchestra.run.trial030.stdout:Modify: 2026-02-23 11:12:23.125199877 +0000 2026-02-23T11:19:33.117 INFO:teuthology.orchestra.run.trial030.stdout:Change: 2026-02-23 11:12:23.125199877 +0000 2026-02-23T11:19:33.117 INFO:teuthology.orchestra.run.trial030.stdout: Birth: 2026-02-23 11:12:23.125199877 +0000 2026-02-23T11:19:33.117 DEBUG:teuthology.orchestra.run.trial030:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2026-02-23T11:19:33.178 INFO:teuthology.orchestra.run.trial030.stderr:1+0 records in 2026-02-23T11:19:33.178 INFO:teuthology.orchestra.run.trial030.stderr:1+0 records out 2026-02-23T11:19:33.178 INFO:teuthology.orchestra.run.trial030.stderr:512 bytes copied, 0.000166369 s, 3.1 MB/s 2026-02-23T11:19:33.179 DEBUG:teuthology.orchestra.run.trial030:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2026-02-23T11:19:33.237 DEBUG:teuthology.orchestra.run.trial030:> stat /dev/vg_nvme/lv_3 2026-02-23T11:19:33.293 INFO:teuthology.orchestra.run.trial030.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2026-02-23T11:19:33.293 INFO:teuthology.orchestra.run.trial030.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2026-02-23T11:19:33.293 INFO:teuthology.orchestra.run.trial030.stdout:Device: 6h/6d Inode: 963 Links: 1 2026-02-23T11:19:33.293 INFO:teuthology.orchestra.run.trial030.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2026-02-23T11:19:33.293 INFO:teuthology.orchestra.run.trial030.stdout:Context: system_u:object_r:device_t:s0 2026-02-23T11:19:33.294 INFO:teuthology.orchestra.run.trial030.stdout:Access: 2026-02-23 11:18:52.933618003 +0000 2026-02-23T11:19:33.294 INFO:teuthology.orchestra.run.trial030.stdout:Modify: 2026-02-23 11:12:23.377192497 +0000 2026-02-23T11:19:33.294 INFO:teuthology.orchestra.run.trial030.stdout:Change: 2026-02-23 11:12:23.377192497 +0000 2026-02-23T11:19:33.294 INFO:teuthology.orchestra.run.trial030.stdout: Birth: 2026-02-23 11:12:23.377192497 +0000 2026-02-23T11:19:33.294 DEBUG:teuthology.orchestra.run.trial030:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2026-02-23T11:19:33.356 INFO:teuthology.orchestra.run.trial030.stderr:1+0 records in 2026-02-23T11:19:33.357 INFO:teuthology.orchestra.run.trial030.stderr:1+0 records out 2026-02-23T11:19:33.357 INFO:teuthology.orchestra.run.trial030.stderr:512 bytes copied, 0.000152964 s, 3.3 MB/s 2026-02-23T11:19:33.358 DEBUG:teuthology.orchestra.run.trial030:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2026-02-23T11:19:33.415 DEBUG:teuthology.orchestra.run.trial030:> stat /dev/vg_nvme/lv_4 2026-02-23T11:19:33.471 INFO:teuthology.orchestra.run.trial030.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2026-02-23T11:19:33.471 INFO:teuthology.orchestra.run.trial030.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2026-02-23T11:19:33.471 INFO:teuthology.orchestra.run.trial030.stdout:Device: 6h/6d Inode: 993 Links: 1 2026-02-23T11:19:33.471 INFO:teuthology.orchestra.run.trial030.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2026-02-23T11:19:33.471 INFO:teuthology.orchestra.run.trial030.stdout:Context: system_u:object_r:device_t:s0 2026-02-23T11:19:33.471 INFO:teuthology.orchestra.run.trial030.stdout:Access: 2026-02-23 11:18:52.934617980 +0000 2026-02-23T11:19:33.471 INFO:teuthology.orchestra.run.trial030.stdout:Modify: 2026-02-23 11:12:23.615185527 +0000 2026-02-23T11:19:33.472 INFO:teuthology.orchestra.run.trial030.stdout:Change: 2026-02-23 11:12:23.615185527 +0000 2026-02-23T11:19:33.472 INFO:teuthology.orchestra.run.trial030.stdout: Birth: 2026-02-23 11:12:23.615185527 +0000 2026-02-23T11:19:33.472 DEBUG:teuthology.orchestra.run.trial030:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2026-02-23T11:19:33.535 INFO:teuthology.orchestra.run.trial030.stderr:1+0 records in 2026-02-23T11:19:33.535 INFO:teuthology.orchestra.run.trial030.stderr:1+0 records out 2026-02-23T11:19:33.535 INFO:teuthology.orchestra.run.trial030.stderr:512 bytes copied, 0.00013474 s, 3.8 MB/s 2026-02-23T11:19:33.536 DEBUG:teuthology.orchestra.run.trial030:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2026-02-23T11:19:33.593 INFO:tasks.ceph:osd dev map: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2026-02-23T11:19:33.593 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@trial030.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'}} 2026-02-23T11:19:33.593 INFO:tasks.ceph:Generating config... 2026-02-23T11:19:33.595 INFO:tasks.ceph:[mon] mon min osdmap epochs = 50 2026-02-23T11:19:33.596 INFO:tasks.ceph:[mon] mon osdmap full prune interval = 2 2026-02-23T11:19:33.596 INFO:tasks.ceph:[mon] mon osdmap full prune min = 15 2026-02-23T11:19:33.596 INFO:tasks.ceph:[mon] mon osdmap full prune txsize = 2 2026-02-23T11:19:33.596 INFO:tasks.ceph:[mon] paxos service trim min = 10 2026-02-23T11:19:33.596 INFO:tasks.ceph:[mon] debug mon = 20 2026-02-23T11:19:33.596 INFO:tasks.ceph:[mon] debug ms = 1 2026-02-23T11:19:33.596 INFO:tasks.ceph:[mon] debug paxos = 20 2026-02-23T11:19:33.596 INFO:tasks.ceph:[mon] osd pool default crimson = True 2026-02-23T11:19:33.596 INFO:tasks.ceph:[osd] debug monc = 20 2026-02-23T11:19:33.596 INFO:tasks.ceph:[osd] crimson cpu num = 4 2026-02-23T11:19:33.597 INFO:tasks.ceph:[osd] crimson osd obc lru size = 10 2026-02-23T11:19:33.597 INFO:tasks.ceph:[osd] debug ms = 20 2026-02-23T11:19:33.597 INFO:tasks.ceph:[osd] debug osd = 20 2026-02-23T11:19:33.597 INFO:tasks.ceph:[osd] debug seastore = 20 2026-02-23T11:19:33.597 INFO:tasks.ceph:[osd] debug seastore backref = 20 2026-02-23T11:19:33.597 INFO:tasks.ceph:[osd] debug seastore cache = 20 2026-02-23T11:19:33.597 INFO:tasks.ceph:[osd] debug seastore cleaner = 20 2026-02-23T11:19:33.597 INFO:tasks.ceph:[osd] debug seastore device = 20 2026-02-23T11:19:33.597 INFO:tasks.ceph:[osd] debug seastore epm = 20 2026-02-23T11:19:33.597 INFO:tasks.ceph:[osd] debug seastore fixedkv tree = 20 2026-02-23T11:19:33.597 INFO:tasks.ceph:[osd] debug seastore journal = 20 2026-02-23T11:19:33.597 INFO:tasks.ceph:[osd] debug seastore lba = 20 2026-02-23T11:19:33.597 INFO:tasks.ceph:[osd] debug seastore odata = 20 2026-02-23T11:19:33.598 INFO:tasks.ceph:[osd] debug seastore omap = 20 2026-02-23T11:19:33.598 INFO:tasks.ceph:[osd] debug seastore onode = 20 2026-02-23T11:19:33.598 INFO:tasks.ceph:[osd] debug seastore t = 20 2026-02-23T11:19:33.598 INFO:tasks.ceph:[osd] debug seastore tm = 20 2026-02-23T11:19:33.598 INFO:tasks.ceph:[osd] osd objectstore = seastore 2026-02-23T11:19:33.598 INFO:tasks.ceph:[osd] osd shutdown pgref assert = True 2026-02-23T11:19:33.598 INFO:tasks.ceph:[osd] seastore cachepin size pershard = 64M 2026-02-23T11:19:33.598 INFO:tasks.ceph:[osd] seastore cachepin type = 2Q 2026-02-23T11:19:33.598 INFO:tasks.ceph:[osd] seastore max concurrent transactions = 8 2026-02-23T11:19:33.598 INFO:tasks.ceph:[global] enable experimental unrecoverable data corrupting features = crimson 2026-02-23T11:19:33.598 INFO:tasks.ceph:[global] ms client mode = crc 2026-02-23T11:19:33.598 INFO:tasks.ceph:[global] ms cluster mode = crc 2026-02-23T11:19:33.599 INFO:tasks.ceph:[global] ms mon client mode = crc 2026-02-23T11:19:33.599 INFO:tasks.ceph:[global] ms mon cluster mode = crc 2026-02-23T11:19:33.599 INFO:tasks.ceph:[global] ms mon service mode = crc 2026-02-23T11:19:33.599 INFO:tasks.ceph:[global] ms service mode = crc 2026-02-23T11:19:33.599 INFO:tasks.ceph:[mgr] debug mgr = 20 2026-02-23T11:19:33.599 INFO:tasks.ceph:[mgr] debug ms = 1 2026-02-23T11:19:33.599 INFO:tasks.ceph:Setting up mon.a... 2026-02-23T11:19:33.599 DEBUG:teuthology.orchestra.run.trial030:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/ceph.keyring 2026-02-23T11:19:33.676 INFO:teuthology.orchestra.run.trial030.stdout:creating /etc/ceph/ceph.keyring 2026-02-23T11:19:33.678 DEBUG:teuthology.orchestra.run.trial030:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/ceph.keyring 2026-02-23T11:19:33.762 DEBUG:teuthology.orchestra.run.trial030:> sudo chmod 0644 /etc/ceph/ceph.keyring 2026-02-23T11:19:33.787 DEBUG:tasks.ceph:Ceph mon addresses: [('mon.a', '10.20.193.30')] 2026-02-23T11:19:33.788 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'cephx', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=isa technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '10.20.193.30', 'enable experimental unrecoverable data corrupting features': 'crimson', 'ms client mode': 'crc', 'ms cluster mode': 'crc', 'ms mon client mode': 'crc', 'ms mon cluster mode': 'crc', 'ms mon service mode': 'crc', 'ms service mode': 'crc'}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd mclock profile': 'high_recovery_ops', 'osd mclock skip benchmark': 'true', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd debug trim objects': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'debug monc': 20, 'crimson cpu num': 4, 'crimson osd obc lru size': 10, 'debug ms': 20, 'debug osd': 20, 'debug seastore': 20, 'debug seastore backref': 20, 'debug seastore cache': 20, 'debug seastore cleaner': 20, 'debug seastore device': 20, 'debug seastore epm': 20, 'debug seastore fixedkv tree': 20, 'debug seastore journal': 20, 'debug seastore lba': 20, 'debug seastore odata': 20, 'debug seastore omap': 20, 'debug seastore onode': 20, 'debug seastore t': 20, 'debug seastore tm': 20, 'osd objectstore': 'seastore', 'osd shutdown pgref assert': True, 'seastore cachepin size pershard': '64M', 'seastore cachepin type': '2Q', 'seastore max concurrent transactions': 8}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false'}, 'mon': {'debug ms': 1, 'debug mon': 20, 'debug paxos': 20, 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'mon_down_mkfs_grace': '2m', 'mon_warn_on_filestore_osds': 'false', 'mon min osdmap epochs': 50, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'paxos service trim min': 10, 'osd pool default crimson': True}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok'}, 'mon.a': {}} 2026-02-23T11:19:33.789 DEBUG:teuthology.orchestra.run.trial030:> set -ex 2026-02-23T11:19:33.789 DEBUG:teuthology.orchestra.run.trial030:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2026-02-23T11:19:33.843 DEBUG:teuthology.orchestra.run.trial030:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --add a 10.20.193.30 --print /home/ubuntu/cephtest/ceph.monmap 2026-02-23T11:19:33.923 INFO:teuthology.orchestra.run.trial030.stderr:2026-02-23T11:19:33.921+0000 7f2dc583bf40 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T11:19:33.923 INFO:teuthology.orchestra.run.trial030.stderr:2026-02-23T11:19:33.921+0000 7f2dc583bf40 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T11:19:33.923 INFO:teuthology.orchestra.run.trial030.stderr:2026-02-23T11:19:33.921+0000 7f2dc583bf40 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T11:19:33.924 INFO:teuthology.orchestra.run.trial030.stdout:monmaptool: monmap file /home/ubuntu/cephtest/ceph.monmap 2026-02-23T11:19:33.924 INFO:teuthology.orchestra.run.trial030.stdout:monmaptool: generated fsid 4c6fca00-8e92-4876-9bd8-e5358ac35226 2026-02-23T11:19:33.925 INFO:teuthology.orchestra.run.trial030.stdout:setting min_mon_release = tentacle 2026-02-23T11:19:33.925 INFO:teuthology.orchestra.run.trial030.stdout:epoch 0 2026-02-23T11:19:33.925 INFO:teuthology.orchestra.run.trial030.stdout:fsid 4c6fca00-8e92-4876-9bd8-e5358ac35226 2026-02-23T11:19:33.925 INFO:teuthology.orchestra.run.trial030.stdout:last_changed 2026-02-23T11:19:33.923045+0000 2026-02-23T11:19:33.925 INFO:teuthology.orchestra.run.trial030.stdout:created 2026-02-23T11:19:33.923045+0000 2026-02-23T11:19:33.925 INFO:teuthology.orchestra.run.trial030.stdout:min_mon_release 20 (tentacle) 2026-02-23T11:19:33.925 INFO:teuthology.orchestra.run.trial030.stdout:election_strategy: 1 2026-02-23T11:19:33.925 INFO:teuthology.orchestra.run.trial030.stdout:0: [v2:10.20.193.30:3300/0,v1:10.20.193.30:6789/0] mon.a 2026-02-23T11:19:33.925 INFO:teuthology.orchestra.run.trial030.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/ceph.monmap (1 monitors) 2026-02-23T11:19:33.926 DEBUG:teuthology.orchestra.run.trial030:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2026-02-23T11:19:33.983 INFO:tasks.ceph:Writing /etc/ceph/ceph.conf for FSID 4c6fca00-8e92-4876-9bd8-e5358ac35226... 2026-02-23T11:19:33.985 DEBUG:teuthology.orchestra.run.trial030:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2026-02-23T11:19:34.064 INFO:teuthology.orchestra.run.trial030.stdout:[global] 2026-02-23T11:19:34.064 INFO:teuthology.orchestra.run.trial030.stdout: chdir = "" 2026-02-23T11:19:34.064 INFO:teuthology.orchestra.run.trial030.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2026-02-23T11:19:34.064 INFO:teuthology.orchestra.run.trial030.stdout: auth supported = cephx 2026-02-23T11:19:34.064 INFO:teuthology.orchestra.run.trial030.stdout: 2026-02-23T11:19:34.065 INFO:teuthology.orchestra.run.trial030.stdout: filestore xattr use omap = true 2026-02-23T11:19:34.065 INFO:teuthology.orchestra.run.trial030.stdout: 2026-02-23T11:19:34.065 INFO:teuthology.orchestra.run.trial030.stdout: mon clock drift allowed = 1.000 2026-02-23T11:19:34.065 INFO:teuthology.orchestra.run.trial030.stdout: 2026-02-23T11:19:34.065 INFO:teuthology.orchestra.run.trial030.stdout: osd crush chooseleaf type = 0 2026-02-23T11:19:34.065 INFO:teuthology.orchestra.run.trial030.stdout: auth debug = true 2026-02-23T11:19:34.065 INFO:teuthology.orchestra.run.trial030.stdout: 2026-02-23T11:19:34.065 INFO:teuthology.orchestra.run.trial030.stdout: ms die on old message = true 2026-02-23T11:19:34.065 INFO:teuthology.orchestra.run.trial030.stdout: ms die on bug = true 2026-02-23T11:19:34.065 INFO:teuthology.orchestra.run.trial030.stdout: 2026-02-23T11:19:34.065 INFO:teuthology.orchestra.run.trial030.stdout: mon max pg per osd = 10000 # >= luminous 2026-02-23T11:19:34.066 INFO:teuthology.orchestra.run.trial030.stdout: mon pg warn max object skew = 0 2026-02-23T11:19:34.066 INFO:teuthology.orchestra.run.trial030.stdout: 2026-02-23T11:19:34.066 INFO:teuthology.orchestra.run.trial030.stdout: # disable pg_autoscaler by default for new pools 2026-02-23T11:19:34.066 INFO:teuthology.orchestra.run.trial030.stdout: osd_pool_default_pg_autoscale_mode = off 2026-02-23T11:19:34.066 INFO:teuthology.orchestra.run.trial030.stdout: 2026-02-23T11:19:34.066 INFO:teuthology.orchestra.run.trial030.stdout: osd pool default size = 2 2026-02-23T11:19:34.066 INFO:teuthology.orchestra.run.trial030.stdout: 2026-02-23T11:19:34.066 INFO:teuthology.orchestra.run.trial030.stdout: mon osd allow primary affinity = true 2026-02-23T11:19:34.066 INFO:teuthology.orchestra.run.trial030.stdout: mon osd allow pg remap = true 2026-02-23T11:19:34.066 INFO:teuthology.orchestra.run.trial030.stdout: mon warn on legacy crush tunables = false 2026-02-23T11:19:34.066 INFO:teuthology.orchestra.run.trial030.stdout: mon warn on crush straw calc version zero = false 2026-02-23T11:19:34.066 INFO:teuthology.orchestra.run.trial030.stdout: mon warn on no sortbitwise = false 2026-02-23T11:19:34.066 INFO:teuthology.orchestra.run.trial030.stdout: mon warn on osd down out interval zero = false 2026-02-23T11:19:34.067 INFO:teuthology.orchestra.run.trial030.stdout: mon warn on too few osds = false 2026-02-23T11:19:34.067 INFO:teuthology.orchestra.run.trial030.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2026-02-23T11:19:34.067 INFO:teuthology.orchestra.run.trial030.stdout: mon_warn_on_pool_no_redundancy = false 2026-02-23T11:19:34.067 INFO:teuthology.orchestra.run.trial030.stdout: mon_allow_pool_size_one = true 2026-02-23T11:19:34.067 INFO:teuthology.orchestra.run.trial030.stdout: 2026-02-23T11:19:34.067 INFO:teuthology.orchestra.run.trial030.stdout: osd pool default erasure code profile = plugin=isa technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2026-02-23T11:19:34.067 INFO:teuthology.orchestra.run.trial030.stdout: 2026-02-23T11:19:34.067 INFO:teuthology.orchestra.run.trial030.stdout: osd default data pool replay window = 5 2026-02-23T11:19:34.067 INFO:teuthology.orchestra.run.trial030.stdout: 2026-02-23T11:19:34.067 INFO:teuthology.orchestra.run.trial030.stdout: mon allow pool delete = true 2026-02-23T11:19:34.067 INFO:teuthology.orchestra.run.trial030.stdout: 2026-02-23T11:19:34.067 INFO:teuthology.orchestra.run.trial030.stdout: mon cluster log file level = debug 2026-02-23T11:19:34.068 INFO:teuthology.orchestra.run.trial030.stdout: debug asserts on shutdown = true 2026-02-23T11:19:34.068 INFO:teuthology.orchestra.run.trial030.stdout: mon health detail to clog = false 2026-02-23T11:19:34.068 INFO:teuthology.orchestra.run.trial030.stdout: mon host = 10.20.193.30 2026-02-23T11:19:34.068 INFO:teuthology.orchestra.run.trial030.stdout: enable experimental unrecoverable data corrupting features = crimson 2026-02-23T11:19:34.068 INFO:teuthology.orchestra.run.trial030.stdout: ms client mode = crc 2026-02-23T11:19:34.068 INFO:teuthology.orchestra.run.trial030.stdout: ms cluster mode = crc 2026-02-23T11:19:34.068 INFO:teuthology.orchestra.run.trial030.stdout: ms mon client mode = crc 2026-02-23T11:19:34.068 INFO:teuthology.orchestra.run.trial030.stdout: ms mon cluster mode = crc 2026-02-23T11:19:34.068 INFO:teuthology.orchestra.run.trial030.stdout: ms mon service mode = crc 2026-02-23T11:19:34.068 INFO:teuthology.orchestra.run.trial030.stdout: ms service mode = crc 2026-02-23T11:19:34.068 INFO:teuthology.orchestra.run.trial030.stdout: fsid = 4c6fca00-8e92-4876-9bd8-e5358ac35226 2026-02-23T11:19:34.068 INFO:teuthology.orchestra.run.trial030.stdout: 2026-02-23T11:19:34.068 INFO:teuthology.orchestra.run.trial030.stdout:[osd] 2026-02-23T11:19:34.069 INFO:teuthology.orchestra.run.trial030.stdout: osd journal size = 100 2026-02-23T11:19:34.069 INFO:teuthology.orchestra.run.trial030.stdout: 2026-02-23T11:19:34.069 INFO:teuthology.orchestra.run.trial030.stdout: osd scrub load threshold = 5.0 2026-02-23T11:19:34.069 INFO:teuthology.orchestra.run.trial030.stdout: osd scrub max interval = 600 2026-02-23T11:19:34.069 INFO:teuthology.orchestra.run.trial030.stdout: osd mclock profile = high_recovery_ops 2026-02-23T11:19:34.069 INFO:teuthology.orchestra.run.trial030.stdout: osd mclock skip benchmark = true 2026-02-23T11:19:34.069 INFO:teuthology.orchestra.run.trial030.stdout: 2026-02-23T11:19:34.069 INFO:teuthology.orchestra.run.trial030.stdout: osd recover clone overlap = true 2026-02-23T11:19:34.069 INFO:teuthology.orchestra.run.trial030.stdout: osd recovery max chunk = 1048576 2026-02-23T11:19:34.069 INFO:teuthology.orchestra.run.trial030.stdout: 2026-02-23T11:19:34.069 INFO:teuthology.orchestra.run.trial030.stdout: osd debug shutdown = true 2026-02-23T11:19:34.069 INFO:teuthology.orchestra.run.trial030.stdout: osd debug op order = true 2026-02-23T11:19:34.070 INFO:teuthology.orchestra.run.trial030.stdout: osd debug verify stray on activate = true 2026-02-23T11:19:34.070 INFO:teuthology.orchestra.run.trial030.stdout: osd debug trim objects = true 2026-02-23T11:19:34.070 INFO:teuthology.orchestra.run.trial030.stdout: 2026-02-23T11:19:34.070 INFO:teuthology.orchestra.run.trial030.stdout: osd open classes on start = true 2026-02-23T11:19:34.070 INFO:teuthology.orchestra.run.trial030.stdout: osd debug pg log writeout = true 2026-02-23T11:19:34.070 INFO:teuthology.orchestra.run.trial030.stdout: 2026-02-23T11:19:34.070 INFO:teuthology.orchestra.run.trial030.stdout: osd deep scrub update digest min age = 30 2026-02-23T11:19:34.070 INFO:teuthology.orchestra.run.trial030.stdout: 2026-02-23T11:19:34.070 INFO:teuthology.orchestra.run.trial030.stdout: osd map max advance = 10 2026-02-23T11:19:34.070 INFO:teuthology.orchestra.run.trial030.stdout: 2026-02-23T11:19:34.070 INFO:teuthology.orchestra.run.trial030.stdout: journal zero on create = true 2026-02-23T11:19:34.070 INFO:teuthology.orchestra.run.trial030.stdout: 2026-02-23T11:19:34.070 INFO:teuthology.orchestra.run.trial030.stdout: filestore ondisk finisher threads = 3 2026-02-23T11:19:34.071 INFO:teuthology.orchestra.run.trial030.stdout: filestore apply finisher threads = 3 2026-02-23T11:19:34.071 INFO:teuthology.orchestra.run.trial030.stdout: 2026-02-23T11:19:34.071 INFO:teuthology.orchestra.run.trial030.stdout: bdev debug aio = true 2026-02-23T11:19:34.071 INFO:teuthology.orchestra.run.trial030.stdout: osd debug misdirected ops = true 2026-02-23T11:19:34.071 INFO:teuthology.orchestra.run.trial030.stdout: debug monc = 20 2026-02-23T11:19:34.071 INFO:teuthology.orchestra.run.trial030.stdout: crimson cpu num = 4 2026-02-23T11:19:34.071 INFO:teuthology.orchestra.run.trial030.stdout: crimson osd obc lru size = 10 2026-02-23T11:19:34.071 INFO:teuthology.orchestra.run.trial030.stdout: debug ms = 20 2026-02-23T11:19:34.071 INFO:teuthology.orchestra.run.trial030.stdout: debug osd = 20 2026-02-23T11:19:34.071 INFO:teuthology.orchestra.run.trial030.stdout: debug seastore = 20 2026-02-23T11:19:34.071 INFO:teuthology.orchestra.run.trial030.stdout: debug seastore backref = 20 2026-02-23T11:19:34.071 INFO:teuthology.orchestra.run.trial030.stdout: debug seastore cache = 20 2026-02-23T11:19:34.071 INFO:teuthology.orchestra.run.trial030.stdout: debug seastore cleaner = 20 2026-02-23T11:19:34.072 INFO:teuthology.orchestra.run.trial030.stdout: debug seastore device = 20 2026-02-23T11:19:34.072 INFO:teuthology.orchestra.run.trial030.stdout: debug seastore epm = 20 2026-02-23T11:19:34.072 INFO:teuthology.orchestra.run.trial030.stdout: debug seastore fixedkv tree = 20 2026-02-23T11:19:34.072 INFO:teuthology.orchestra.run.trial030.stdout: debug seastore journal = 20 2026-02-23T11:19:34.072 INFO:teuthology.orchestra.run.trial030.stdout: debug seastore lba = 20 2026-02-23T11:19:34.072 INFO:teuthology.orchestra.run.trial030.stdout: debug seastore odata = 20 2026-02-23T11:19:34.072 INFO:teuthology.orchestra.run.trial030.stdout: debug seastore omap = 20 2026-02-23T11:19:34.072 INFO:teuthology.orchestra.run.trial030.stdout: debug seastore onode = 20 2026-02-23T11:19:34.072 INFO:teuthology.orchestra.run.trial030.stdout: debug seastore t = 20 2026-02-23T11:19:34.072 INFO:teuthology.orchestra.run.trial030.stdout: debug seastore tm = 20 2026-02-23T11:19:34.072 INFO:teuthology.orchestra.run.trial030.stdout: osd objectstore = seastore 2026-02-23T11:19:34.072 INFO:teuthology.orchestra.run.trial030.stdout: osd shutdown pgref assert = True 2026-02-23T11:19:34.072 INFO:teuthology.orchestra.run.trial030.stdout: seastore cachepin size pershard = 64M 2026-02-23T11:19:34.072 INFO:teuthology.orchestra.run.trial030.stdout: seastore cachepin type = 2Q 2026-02-23T11:19:34.073 INFO:teuthology.orchestra.run.trial030.stdout: seastore max concurrent transactions = 8 2026-02-23T11:19:34.073 INFO:teuthology.orchestra.run.trial030.stdout: 2026-02-23T11:19:34.073 INFO:teuthology.orchestra.run.trial030.stdout:[mgr] 2026-02-23T11:19:34.073 INFO:teuthology.orchestra.run.trial030.stdout: debug ms = 1 2026-02-23T11:19:34.073 INFO:teuthology.orchestra.run.trial030.stdout: debug mgr = 20 2026-02-23T11:19:34.073 INFO:teuthology.orchestra.run.trial030.stdout: debug mon = 20 2026-02-23T11:19:34.073 INFO:teuthology.orchestra.run.trial030.stdout: debug auth = 20 2026-02-23T11:19:34.073 INFO:teuthology.orchestra.run.trial030.stdout: mon reweight min pgs per osd = 4 2026-02-23T11:19:34.073 INFO:teuthology.orchestra.run.trial030.stdout: mon reweight min bytes per osd = 10 2026-02-23T11:19:34.073 INFO:teuthology.orchestra.run.trial030.stdout: mgr/telemetry/nag = false 2026-02-23T11:19:34.073 INFO:teuthology.orchestra.run.trial030.stdout: 2026-02-23T11:19:34.073 INFO:teuthology.orchestra.run.trial030.stdout:[mon] 2026-02-23T11:19:34.074 INFO:teuthology.orchestra.run.trial030.stdout: debug ms = 1 2026-02-23T11:19:34.074 INFO:teuthology.orchestra.run.trial030.stdout: debug mon = 20 2026-02-23T11:19:34.074 INFO:teuthology.orchestra.run.trial030.stdout: debug paxos = 20 2026-02-23T11:19:34.074 INFO:teuthology.orchestra.run.trial030.stdout: debug auth = 20 2026-02-23T11:19:34.074 INFO:teuthology.orchestra.run.trial030.stdout: mon data avail warn = 5 2026-02-23T11:19:34.074 INFO:teuthology.orchestra.run.trial030.stdout: mon mgr mkfs grace = 240 2026-02-23T11:19:34.074 INFO:teuthology.orchestra.run.trial030.stdout: mon reweight min pgs per osd = 4 2026-02-23T11:19:34.074 INFO:teuthology.orchestra.run.trial030.stdout: mon osd reporter subtree level = osd 2026-02-23T11:19:34.074 INFO:teuthology.orchestra.run.trial030.stdout: mon osd prime pg temp = true 2026-02-23T11:19:34.074 INFO:teuthology.orchestra.run.trial030.stdout: mon reweight min bytes per osd = 10 2026-02-23T11:19:34.074 INFO:teuthology.orchestra.run.trial030.stdout: 2026-02-23T11:19:34.074 INFO:teuthology.orchestra.run.trial030.stdout: # rotate auth tickets quickly to exercise renewal paths 2026-02-23T11:19:34.074 INFO:teuthology.orchestra.run.trial030.stdout: auth mon ticket ttl = 660 # 11m 2026-02-23T11:19:34.075 INFO:teuthology.orchestra.run.trial030.stdout: auth service ticket ttl = 240 # 4m 2026-02-23T11:19:34.075 INFO:teuthology.orchestra.run.trial030.stdout: 2026-02-23T11:19:34.075 INFO:teuthology.orchestra.run.trial030.stdout: # don't complain about insecure global_id in the test suite 2026-02-23T11:19:34.075 INFO:teuthology.orchestra.run.trial030.stdout: mon_warn_on_insecure_global_id_reclaim = false 2026-02-23T11:19:34.075 INFO:teuthology.orchestra.run.trial030.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2026-02-23T11:19:34.075 INFO:teuthology.orchestra.run.trial030.stdout: 2026-02-23T11:19:34.075 INFO:teuthology.orchestra.run.trial030.stdout: # 1m isn't quite enough 2026-02-23T11:19:34.075 INFO:teuthology.orchestra.run.trial030.stdout: mon_down_mkfs_grace = 2m 2026-02-23T11:19:34.075 INFO:teuthology.orchestra.run.trial030.stdout: 2026-02-23T11:19:34.075 INFO:teuthology.orchestra.run.trial030.stdout: mon_warn_on_filestore_osds = false 2026-02-23T11:19:34.075 INFO:teuthology.orchestra.run.trial030.stdout: mon min osdmap epochs = 50 2026-02-23T11:19:34.075 INFO:teuthology.orchestra.run.trial030.stdout: mon osdmap full prune interval = 2 2026-02-23T11:19:34.075 INFO:teuthology.orchestra.run.trial030.stdout: mon osdmap full prune min = 15 2026-02-23T11:19:34.076 INFO:teuthology.orchestra.run.trial030.stdout: mon osdmap full prune txsize = 2 2026-02-23T11:19:34.076 INFO:teuthology.orchestra.run.trial030.stdout: paxos service trim min = 10 2026-02-23T11:19:34.076 INFO:teuthology.orchestra.run.trial030.stdout: osd pool default crimson = True 2026-02-23T11:19:34.076 INFO:teuthology.orchestra.run.trial030.stdout: 2026-02-23T11:19:34.076 INFO:teuthology.orchestra.run.trial030.stdout:[client] 2026-02-23T11:19:34.076 INFO:teuthology.orchestra.run.trial030.stdout: rgw cache enabled = true 2026-02-23T11:19:34.076 INFO:teuthology.orchestra.run.trial030.stdout: rgw enable ops log = true 2026-02-23T11:19:34.076 INFO:teuthology.orchestra.run.trial030.stdout: rgw enable usage log = true 2026-02-23T11:19:34.076 INFO:teuthology.orchestra.run.trial030.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2026-02-23T11:19:34.076 INFO:teuthology.orchestra.run.trial030.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2026-02-23T11:19:34.076 INFO:teuthology.orchestra.run.trial030.stdout:[mon.a] 2026-02-23T11:19:34.077 INFO:tasks.ceph:Creating admin key on mon.a... 2026-02-23T11:19:34.078 DEBUG:teuthology.orchestra.run.trial030:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/ceph.keyring 2026-02-23T11:19:34.169 INFO:tasks.ceph:Copying monmap to all nodes... 2026-02-23T11:19:34.169 DEBUG:teuthology.orchestra.run.trial030:> set -ex 2026-02-23T11:19:34.170 DEBUG:teuthology.orchestra.run.trial030:> dd if=/etc/ceph/ceph.keyring of=/dev/stdout 2026-02-23T11:19:34.187 DEBUG:teuthology.orchestra.run.trial030:> set -ex 2026-02-23T11:19:34.187 DEBUG:teuthology.orchestra.run.trial030:> dd if=/home/ubuntu/cephtest/ceph.monmap of=/dev/stdout 2026-02-23T11:19:34.243 INFO:tasks.ceph:Sending monmap to node ubuntu@trial030.front.sepia.ceph.com 2026-02-23T11:19:34.243 DEBUG:teuthology.orchestra.run.trial030:> set -ex 2026-02-23T11:19:34.243 DEBUG:teuthology.orchestra.run.trial030:> sudo dd of=/etc/ceph/ceph.keyring 2026-02-23T11:19:34.243 DEBUG:teuthology.orchestra.run.trial030:> sudo chmod 0644 /etc/ceph/ceph.keyring 2026-02-23T11:19:34.316 DEBUG:teuthology.orchestra.run.trial030:> set -ex 2026-02-23T11:19:34.316 DEBUG:teuthology.orchestra.run.trial030:> dd of=/home/ubuntu/cephtest/ceph.monmap 2026-02-23T11:19:34.372 INFO:tasks.ceph:Setting up mon nodes... 2026-02-23T11:19:34.372 INFO:tasks.ceph:Setting up mgr nodes... 2026-02-23T11:19:34.372 DEBUG:teuthology.orchestra.run.trial030:> sudo mkdir -p /var/lib/ceph/mgr/ceph-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/ceph-x/keyring 2026-02-23T11:19:34.462 INFO:teuthology.orchestra.run.trial030.stdout:creating /var/lib/ceph/mgr/ceph-x/keyring 2026-02-23T11:19:34.465 INFO:tasks.ceph:Setting up mds nodes... 2026-02-23T11:19:34.465 INFO:tasks.ceph_client:Setting up client nodes... 2026-02-23T11:19:34.465 DEBUG:teuthology.orchestra.run.trial030:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/ceph.client.0.keyring && sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2026-02-23T11:19:34.507 INFO:teuthology.orchestra.run.trial030.stdout:creating /etc/ceph/ceph.client.0.keyring 2026-02-23T11:19:34.518 INFO:tasks.ceph:Running mkfs on osd nodes... 2026-02-23T11:19:34.519 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@trial030.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'}} 2026-02-23T11:19:34.519 DEBUG:teuthology.orchestra.run.trial030:> sudo mkdir -p /var/lib/ceph/osd/ceph-0 2026-02-23T11:19:34.582 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2026-02-23T11:19:34.582 INFO:tasks.ceph:role: osd.0 2026-02-23T11:19:34.582 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@trial030.front.sepia.ceph.com 2026-02-23T11:19:34.582 DEBUG:teuthology.orchestra.run.trial030:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2026-02-23T11:19:34.647 INFO:teuthology.orchestra.run.trial030.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=45875200 blks 2026-02-23T11:19:34.647 INFO:teuthology.orchestra.run.trial030.stdout: = sectsz=4096 attr=2, projid32bit=1 2026-02-23T11:19:34.647 INFO:teuthology.orchestra.run.trial030.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2026-02-23T11:19:34.647 INFO:teuthology.orchestra.run.trial030.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2026-02-23T11:19:34.648 INFO:teuthology.orchestra.run.trial030.stdout:data = bsize=4096 blocks=183500800, imaxpct=25 2026-02-23T11:19:34.648 INFO:teuthology.orchestra.run.trial030.stdout: = sunit=0 swidth=0 blks 2026-02-23T11:19:34.648 INFO:teuthology.orchestra.run.trial030.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2026-02-23T11:19:34.648 INFO:teuthology.orchestra.run.trial030.stdout:log =internal log bsize=4096 blocks=89600, version=2 2026-02-23T11:19:34.648 INFO:teuthology.orchestra.run.trial030.stdout: = sectsz=4096 sunit=1 blks, lazy-count=1 2026-02-23T11:19:34.648 INFO:teuthology.orchestra.run.trial030.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2026-02-23T11:19:35.224 INFO:teuthology.orchestra.run.trial030.stdout:Discarding blocks...Done. 2026-02-23T11:19:35.227 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@trial030.front.sepia.ceph.com -o noatime 2026-02-23T11:19:35.227 DEBUG:teuthology.orchestra.run.trial030:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-0 2026-02-23T11:19:35.296 DEBUG:teuthology.orchestra.run.trial030:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-0 2026-02-23T11:19:35.363 DEBUG:teuthology.orchestra.run.trial030:> sudo mkdir -p /var/lib/ceph/osd/ceph-1 2026-02-23T11:19:35.426 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2026-02-23T11:19:35.427 INFO:tasks.ceph:role: osd.1 2026-02-23T11:19:35.427 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@trial030.front.sepia.ceph.com 2026-02-23T11:19:35.427 DEBUG:teuthology.orchestra.run.trial030:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2026-02-23T11:19:35.493 INFO:teuthology.orchestra.run.trial030.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=45875200 blks 2026-02-23T11:19:35.493 INFO:teuthology.orchestra.run.trial030.stdout: = sectsz=4096 attr=2, projid32bit=1 2026-02-23T11:19:35.493 INFO:teuthology.orchestra.run.trial030.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2026-02-23T11:19:35.493 INFO:teuthology.orchestra.run.trial030.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2026-02-23T11:19:35.493 INFO:teuthology.orchestra.run.trial030.stdout:data = bsize=4096 blocks=183500800, imaxpct=25 2026-02-23T11:19:35.494 INFO:teuthology.orchestra.run.trial030.stdout: = sunit=0 swidth=0 blks 2026-02-23T11:19:35.494 INFO:teuthology.orchestra.run.trial030.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2026-02-23T11:19:35.494 INFO:teuthology.orchestra.run.trial030.stdout:log =internal log bsize=4096 blocks=89600, version=2 2026-02-23T11:19:35.494 INFO:teuthology.orchestra.run.trial030.stdout: = sectsz=4096 sunit=1 blks, lazy-count=1 2026-02-23T11:19:35.494 INFO:teuthology.orchestra.run.trial030.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2026-02-23T11:19:36.051 INFO:teuthology.orchestra.run.trial030.stdout:Discarding blocks...Done. 2026-02-23T11:19:36.053 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@trial030.front.sepia.ceph.com -o noatime 2026-02-23T11:19:36.054 DEBUG:teuthology.orchestra.run.trial030:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-1 2026-02-23T11:19:36.122 DEBUG:teuthology.orchestra.run.trial030:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-1 2026-02-23T11:19:36.190 DEBUG:teuthology.orchestra.run.trial030:> sudo mkdir -p /var/lib/ceph/osd/ceph-2 2026-02-23T11:19:36.253 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2026-02-23T11:19:36.254 INFO:tasks.ceph:role: osd.2 2026-02-23T11:19:36.254 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@trial030.front.sepia.ceph.com 2026-02-23T11:19:36.254 DEBUG:teuthology.orchestra.run.trial030:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2026-02-23T11:19:36.318 INFO:teuthology.orchestra.run.trial030.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=45875200 blks 2026-02-23T11:19:36.319 INFO:teuthology.orchestra.run.trial030.stdout: = sectsz=4096 attr=2, projid32bit=1 2026-02-23T11:19:36.319 INFO:teuthology.orchestra.run.trial030.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2026-02-23T11:19:36.319 INFO:teuthology.orchestra.run.trial030.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2026-02-23T11:19:36.319 INFO:teuthology.orchestra.run.trial030.stdout:data = bsize=4096 blocks=183500800, imaxpct=25 2026-02-23T11:19:36.319 INFO:teuthology.orchestra.run.trial030.stdout: = sunit=0 swidth=0 blks 2026-02-23T11:19:36.319 INFO:teuthology.orchestra.run.trial030.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2026-02-23T11:19:36.319 INFO:teuthology.orchestra.run.trial030.stdout:log =internal log bsize=4096 blocks=89600, version=2 2026-02-23T11:19:36.319 INFO:teuthology.orchestra.run.trial030.stdout: = sectsz=4096 sunit=1 blks, lazy-count=1 2026-02-23T11:19:36.319 INFO:teuthology.orchestra.run.trial030.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2026-02-23T11:19:36.876 INFO:teuthology.orchestra.run.trial030.stdout:Discarding blocks...Done. 2026-02-23T11:19:36.878 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@trial030.front.sepia.ceph.com -o noatime 2026-02-23T11:19:36.878 DEBUG:teuthology.orchestra.run.trial030:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-2 2026-02-23T11:19:36.945 DEBUG:teuthology.orchestra.run.trial030:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-2 2026-02-23T11:19:37.012 DEBUG:teuthology.orchestra.run.trial030:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/ceph.monmap 2026-02-23T11:19:37.165 INFO:teuthology.orchestra.run.trial030.stderr:INFO  osd - OSD::main: parsing early config 2026-02-23T11:19:37.223 INFO:teuthology.orchestra.run.trial030.stderr:INFO 2026-02-23 11:19:37,223 seastar - Reactor backend: linux-aio 2026-02-23T11:19:37.233 INFO:teuthology.orchestra.run.trial030.stderr:INFO 2026-02-23 11:19:37,233 [shard 0:main] io - IO queue was unable to find a suitable maximum request length, the search was cut-off early at: 16MB 2026-02-23T11:19:37.233 INFO:teuthology.orchestra.run.trial030.stderr:INFO 2026-02-23 11:19:37,233 [shard 0:main] io - IO queue was unable to find a suitable maximum request length, the search was cut-off early at: 16MB 2026-02-23T11:19:37.235 INFO:teuthology.orchestra.run.trial030.stderr:==35545==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2026-02-23T11:19:37.268 INFO:teuthology.orchestra.run.trial030.stderr:Reactor stalled for 32 ms on shard 0, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0x9f10dfc 0x9d045c3 0x9d40a13 0x9d43fc6 0x9d44234 0x9d4437c 0x9d4483b /lib64/libc.so.6+0x3fc2f 0x357c103 0x35879a0 0x3587be6 0x3587d3a 0x3587e72 0x3588149 0x35883af 0x3588677 0x35889cb 0x3588b3f 0x3588f66 0x3589179 0x35893fa 0x35895bc 0x35896e4 0x358b19a 0x358db23 0x358dd8c 0x358e223 0x358e33f 0x358e571 0x358e7f4 0x358eb73 0x358ee83 0x358ef41 0x358f4df 0x358f947 0x358fd84 0x35900aa 0x35901cc 0x359268e 0x3594527 0x35947de 0x359586f 0x359592d 0x35959cf 0x3595af8 0x3595e1d 0x359601f 0x35963a1 0x35cd7b0 0x35ce05a 0x35ce5ba 0x35ce932 0x35cedd0 0x35cf20f 0x3596cf0 0x3597d27 0x3597fde 0x359847d 0x3598599 0x359884d 0x3598bdf 0x3598f12 0x359926f 0x35994a7 0x3599ae2 0x3599e12 0x359a178 0x359a625 0x359aa64 0x359ac90 0x359b09f 0x359b356 0x359b4a7 0x359b778 0x359bb03 0x359be36 0x359c329 0x359c70b 0x359c897 0x359cccf 0x359d03b 0x359d3a0 0x359d6c3 0x359d7a5 0x359da33 0x359dc0a 0x359de71 0x35d1229 0x35d1979 0x35d1e65 0x35d21df 0x35d245e 0x35d2513 0x357b132 0x357ba95 0x357c0cf 0x357b132 0x357b541 0x357b7f8 2026-02-23T11:19:37.291 INFO:teuthology.orchestra.run.trial030.stderr:INFO 2026-02-23 11:19:37,290 [shard 0:main] osd - bypassing the config fetch due to --no-mon-config 2026-02-23T11:19:37.291 INFO:teuthology.orchestra.run.trial030.stderr:INFO 2026-02-23 11:19:37,290 [shard 0:main] osd - Configure option_name crimson_reactor_task_quota_ms with value : 0.500000 2026-02-23T11:19:37.291 INFO:teuthology.orchestra.run.trial030.stderr:INFO 2026-02-23 11:19:37,290 [shard 0:main] osd - Configure option_name crimson_reactor_idle_poll_time_us with value : 200 2026-02-23T11:19:37.291 INFO:teuthology.orchestra.run.trial030.stderr:INFO 2026-02-23 11:19:37,290 [shard 0:main] osd - get_early_config: set --thread-affinity 0 --smp 4 2026-02-23T11:19:37.291 INFO:teuthology.orchestra.run.trial030.stderr:DEBUG 2026-02-23 11:19:37,290 [shard 0:main] seastar - reactor::drain 2026-02-23T11:19:39.340 INFO:teuthology.orchestra.run.trial030.stderr:----------------------------------------------------- 2026-02-23T11:19:39.341 INFO:teuthology.orchestra.run.trial030.stderr:Suppressions used: 2026-02-23T11:19:39.341 INFO:teuthology.orchestra.run.trial030.stderr: count bytes template 2026-02-23T11:19:39.341 INFO:teuthology.orchestra.run.trial030.stderr: 1 8 MallocExtension::Register 2026-02-23T11:19:39.341 INFO:teuthology.orchestra.run.trial030.stderr:----------------------------------------------------- 2026-02-23T11:19:39.341 INFO:teuthology.orchestra.run.trial030.stderr: 2026-02-23T11:19:39.360 INFO:teuthology.orchestra.run.trial030.stderr:INFO  osd - OSD::main: early config parsed successfully 2026-02-23T11:19:39.360 INFO:teuthology.orchestra.run.trial030.stderr:INFO  osd - OSD::main: initializing seastar app_template 2026-02-23T11:19:39.401 INFO:teuthology.orchestra.run.trial030.stderr:INFO  osd - OSD::main: registering CLI options 2026-02-23T11:19:39.401 INFO:teuthology.orchestra.run.trial030.stderr:INFO  osd - OSD::main: entering seastar runtime 2026-02-23T11:19:39.408 INFO:teuthology.orchestra.run.trial030.stderr:INFO 2026-02-23 11:19:39,408 seastar - Reactor backend: linux-aio 2026-02-23T11:19:39.418 INFO:teuthology.orchestra.run.trial030.stderr:INFO 2026-02-23 11:19:39,417 [shard 3:main] io - IO queue was unable to find a suitable maximum request length, the search was cut-off early at: 16MB 2026-02-23T11:19:39.418 INFO:teuthology.orchestra.run.trial030.stderr:INFO 2026-02-23 11:19:39,417 [shard 3:main] io - IO queue was unable to find a suitable maximum request length, the search was cut-off early at: 16MB 2026-02-23T11:19:39.422 INFO:teuthology.orchestra.run.trial030.stderr:==35543==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2026-02-23T11:19:39.422 INFO:teuthology.orchestra.run.trial030.stderr:INFO 2026-02-23 11:19:39,422 [shard 0:main] osd - OSD::main: seastar runtime started 2026-02-23T11:19:39.483 INFO:teuthology.orchestra.run.trial030.stderr:DEBUG 2026-02-23 11:19:39,483 [shard 0:main] osd - OSD::main: initializing logger output 2026-02-23T11:19:39.563 INFO:teuthology.orchestra.run.trial030.stderr:Reactor stalled for 32 ms on shard 1, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0x9f10dfc 0x9d045c3 0x9d40a13 0x9d43fc6 0x9d44234 0x9d4437c 0x9d4483b /lib64/libc.so.6+0x3fc2f /lib64/libasan.so.8+0x12978a /lib64/libasan.so.8+0xc65d1 /lib64/libasan.so.8+0x15197 0x9b03707 0x9b041b6 0x9c864c2 0x9c865bb 0x7be87ab 0x7bf23a6 0x7a51d33 0x7a8d6c5 0x711ff75 0x6f9f19a 0x6fa2f9b 0x6fa8b45 0x7482004 0x72a89fa 0x72a8afc 0x72a8bf4 0x735cb16 0x735cbcf 0x735cd82 0x735ceba 0x735cf35 0x735cfba 0x735d115 0x7425e13 0x748208d 0x728f241 0x72a6bae 0x73b5274 0x7432e6c 0x7432ee7 0x7432f62 0x74eccd2 0x75e6286 0x9e08ac1 0x9d0cb62 0x9dc6e39 0x9e285fb 0x9f988cc 0x9f98f48 0x26ed527 0xa5ce02e /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-23T11:19:39.564 INFO:teuthology.orchestra.run.trial030.stderr:Reactor stalled for 32 ms on shard 2, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0x9f10dfc 0x9d045c3 0x9d40a13 0x9d43fc6 0x9d44234 0x9d4437c 0x9d4483b /lib64/libc.so.6+0x3fc2f 0xb75b52 0xbb9354 0xe05f94 0x7ce5687 0x7bd8faf 0x7be8413 0x7bf23a6 0x7a51d33 0x7a8d6c5 0x711ff75 0x6f9f19a 0x6fa2f9b 0x6fa8b45 0x7482004 0x72a89fa 0x72a8afc 0x72a8bf4 0x735cb16 0x735cbcf 0x735cd82 0x735ceba 0x735cf35 0x735cfba 0x735d115 0x7425e13 0x748208d 0x728f241 0x72a6bae 0x73b5274 0x7432e6c 0x7432ee7 0x7432f62 0x74eccd2 0x75e6286 0x9e08ac1 0x9d0cb62 0x9dc6e39 0x9e285fb 0x9f988cc 0x9f98f48 0x26ed527 0xa5ce02e /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-23T11:19:39.564 INFO:teuthology.orchestra.run.trial030.stderr:Reactor stalled for 32 ms on shard 3, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0x9f10dfc 0x9d045c3 0x9d40a13 0x9d43fc6 0x9d44234 0x9d4437c 0x9d4483b /lib64/libc.so.6+0x3fc2f /lib64/libc.so.6+0x16b36a /lib64/libasan.so.8+0xb0541 /lib64/libasan.so.8+0xb1437 0xb69729 0x10912a0 0xf2e3ce 0x9b7dc0b 0x9b7e0d5 0x9b7e348 0x9b7e5fe 0x9b7f0ad 0x9b7f5ab 0x9b7fbdb 0x9b7fd45 0x9c8fd63 0x9ca2b6a 0x9c052e1 0x9c042e1 0x7be8837 0x7bf23a6 0x7a51d33 0x7a8d6c5 0x711ff75 0x6f9f19a 0x6fa2f9b 0x6fa8b45 0x7482004 0x72a89fa 0x72a8afc 0x72a8bf4 0x735cb16 0x735cbcf 0x735cd82 0x735ceba 0x735cf35 0x735cfba 0x735d115 0x7425e13 0x748208d 0x728f241 0x72a6bae 0x73b5274 0x7432e6c 0x7432ee7 0x7432f62 0x74eccd2 0x75e6286 0x9e08ac1 0x9d0cb62 0x9dc6e39 0x9e285fb 0x9f988cc 0x9f98f48 0x26ed527 0xa5ce02e /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-23T11:19:39.596 INFO:teuthology.orchestra.run.trial030.stderr:Reactor stalled for 65 ms on shard 0, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0x9f10dfc 0x9d045c3 0x9d40a13 0x9d43fc6 0x9d44234 0x9d4437c 0x9d4483b /lib64/libc.so.6+0x3fc2f 0xb6e5e3 0x109121d 0xf2e3ce 0x9b7ddd8 0x9b7e0d5 0x9b7e348 0x9b7e5fe 0x9b7f0ad 0x9b7f5ab 0x9b7fbdb 0x9b7fd45 0x9c8fd63 0x9ca2b6a 0x9c052e1 0x9c042e1 0x7e58655 0x7e5b450 0x8f80d69 0x8f836d7 0x787a27f 0x78aad7c 0x6fa0365 0x75590f6 0x755924a 0x9e08ac1 0x9d0cb62 0x9dc6e39 0x9e285fb 0x9e29e18 0x9c44517 0x9c44d52 0xecf43f /lib64/libc.so.6+0x2a60f /lib64/libc.so.6+0x2a6bf 0xb46f54 2026-02-23T11:19:39.637 INFO:teuthology.orchestra.run.trial030.stderr:Reactor stalled for 65 ms on shard 1, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0x9f10dfc 0x9d045c3 0x9d40a13 0x9d43fc6 0x9d44234 0x9d4437c 0x9d4483b /lib64/libc.so.6+0x3fc2f /lib64/libasan.so.8+0x10ceb3 /lib64/libasan.so.8+0x111056 /lib64/libasan.so.8+0xb9c8 /lib64/libasan.so.8+0xbf98 /lib64/libasan.so.8+0xc08c /lib64/libasan.so.8+0xc360 /lib64/libasan.so.8+0xc553 /lib64/libasan.so.8+0x91c1 /lib64/libasan.so.8+0xee7dc 0x133ef18 0x2a58640 0x7f7f804 0x7f7fdba 0x7f7fee6 0x7f82371 0x7f82fc9 0x7dc31d8 0x7e39297 0x7e5ae04 0x8f80d69 0x8f836d7 0x787a27f 0x78aad7c 0x6fa0365 0x75590f6 0x755924a 0x9e08ac1 0x9d0cb62 0x9dc6e39 0x9e285fb 0x9f988cc 0x9f98f48 0x26ed527 0xa5ce02e /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-23T11:19:39.637 INFO:teuthology.orchestra.run.trial030.stderr:kernel callstack: 0xffffffffffffff80 0xffffffff9edd8877 0xffffffff9edda3c2 0xffffffff9edddc1e 0xffffffff9edde57e 0xffffffff9eda7d9a 0xffffffff9f748f8c 0xffffffff9f80012b 2026-02-23T11:19:39.637 INFO:teuthology.orchestra.run.trial030.stderr:Reactor stalled for 65 ms on shard 3, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0x9f10dfc 0x9d045c3 0x9d40a13 0x9d43fc6 0x9d44234 0x9d4437c 0x9d4483b /lib64/libc.so.6+0x3fc2f /lib64/libc.so.6+0x191ccd /lib64/libasan.so.8+0xc748 /lib64/libasan.so.8+0x91c1 /lib64/libasan.so.8+0xee7dc 0x133ef18 0x2a58640 0x7f7f804 0x7f7fdba 0x7f7fee6 0x7f82371 0x7f82fc9 0x7dc31d8 0x7e39297 0x7e5ae04 0x8f80d69 0x8f836d7 0x787a27f 0x78aad7c 0x6fa0365 0x75590f6 0x755924a 0x9e08ac1 0x9d0cb62 0x9dc6e39 0x9e285fb 0x9f988cc 0x9f98f48 0x26ed527 0xa5ce02e /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-23T11:19:39.637 INFO:teuthology.orchestra.run.trial030.stderr:kernel callstack: 0xffffffffffffff80 0xffffffff9ee60fd0 0xffffffff9ee623e3 0xffffffff9edcb82d 0xffffffff9edd3aeb 0xffffffff9edd3f4b 0xffffffff9ea910d2 0xffffffff9f74d451 0xffffffff9f800ba2 2026-02-23T11:19:39.661 INFO:teuthology.orchestra.run.trial030.stderr:Reactor stalled for 65 ms on shard 0, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0x9f10dfc 0x9d045c3 0x9d40a13 0x9d43fc6 0x9d44234 0x9d4437c 0x9d4483b /lib64/libc.so.6+0x3fc2f /lib64/libc.so.6+0x8cc0c /lib64/libasan.so.8+0x10069c /lib64/libasan.so.8+0xc65b9 /lib64/libasan.so.8+0x15197 0x2a5304b 0x7d7e215 0x7d9da0c 0x7e39ed3 0x7dbe035 0x78750f3 0x7875de8 0x79b38b2 0x7833a0a 0x78a9398 0x78f345f 0x78f36d8 0x79a02cc 0x79a042b 0x79b3a15 0x7864b7c 0x799073e 0x79907ce 0x7990872 0x73eb37a 0x7409299 0x740c6cb 0x75440fc 0x9e08ac1 0x9d0cb62 0x9dc6e39 0x9e285fb 0x9e29e18 0x9c44517 0x9c44d52 0xecf43f /lib64/libc.so.6+0x2a60f /lib64/libc.so.6+0x2a6bf 0xb46f54 2026-02-23T11:19:39.666 INFO:teuthology.orchestra.run.trial030.stderr:Reactor stalled for 32 ms on shard 2, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0x9f10dfc 0x9d045c3 0x9d40a13 0x9d43fc6 0x9d44234 0x9d4437c 0x9d4483b /lib64/libc.so.6+0x3fc2f /lib64/libc.so.6+0x8cc0c /lib64/libasan.so.8+0x10069c /lib64/libasan.so.8+0xc65b9 /lib64/libasan.so.8+0x15197 0x2a5304b 0x7d7e215 0x7d9da0c 0x7e39ed3 0x7dbe035 0x78750f3 0x7875de8 0x79b38b2 0x7833a0a 0x78a9398 0x78f345f 0x78f36d8 0x79a02cc 0x79a042b 0x79b3a15 0x7864b7c 0x799073e 0x79907ce 0x7990872 0x73eb37a 0x7409299 0x740c6cb 0x75440fc 0x9e08ac1 0x9d0cb62 0x9dc6e39 0x9e285fb 0x9f988cc 0x9f98f48 0x26ed527 0xa5ce02e /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-23T11:19:39.703 INFO:teuthology.orchestra.run.trial030.stderr:Reactor stalled for 65 ms on shard 1, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0x9f10dfc 0x9d045c3 0x9d40a13 0x9d43fc6 0x9d44234 0x9d4437c 0x9d4483b /lib64/libc.so.6+0x3fc2f 0x7d7e226 0x7d9da0c 0x7e39ed3 0x7dbe035 0x78750f3 0x7875de8 0x79b38b2 0x7833a0a 0x78a9398 0x78f345f 0x78f36d8 0x79a02cc 0x79a042b 0x79b3a15 0x7864b7c 0x799073e 0x79907ce 0x7990872 0x73eb37a 0x7409299 0x740c6cb 0x75440fc 0x9e08ac1 0x9d0cb62 0x9dc6e39 0x9e285fb 0x9f988cc 0x9f98f48 0x26ed527 0xa5ce02e /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-23T11:19:39.703 INFO:teuthology.orchestra.run.trial030.stderr:Reactor stalled for 65 ms on shard 3, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0x9f10dfc 0x9d045c3 0x9d40a13 0x9d43fc6 0x9d44234 0x9d4437c 0x9d4483b /lib64/libc.so.6+0x3fc2f /lib64/libc.so.6+0x8cc0c /lib64/libasan.so.8+0x10069c /lib64/libasan.so.8+0xc65b9 /lib64/libasan.so.8+0x15197 0x2a5304b 0x7d7e215 0x7d9da0c 0x7e39ed3 0x7dbe035 0x78750f3 0x7875de8 0x79b38b2 0x7833a0a 0x78a9398 0x78f345f 0x78f36d8 0x79a02cc 0x79a042b 0x79b3a15 0x7864b7c 0x799073e 0x79907ce 0x7990872 0x73eb37a 0x7409299 0x740c6cb 0x75440fc 0x9e08ac1 0x9d0cb62 0x9dc6e39 0x9e285fb 0x9f988cc 0x9f98f48 0x26ed527 0xa5ce02e /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-23T11:19:39.828 INFO:teuthology.orchestra.run.trial030.stderr:Reactor stalled for 32 ms on shard 1, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0x9f10dfc 0x9d045c3 0x9d40a13 0x9d43fc6 0x9d44234 0x9d4437c 0x9d4483b /lib64/libc.so.6+0x3fc2f 0xcb064f 0x10913a7 0xf2e3ce 0x9b7dc0b 0x9b7e0d5 0x9b7e348 0x9b8006f 0x9b805bc 0x9b807bb 0x9c9578b 0x9c95ee6 0x9ca0178 0x9ca45f1 0x9c052e1 0x9c042e1 0x7be8837 0x7bf23a6 0x7a51d33 0x7a8d6c5 0x711ff75 0x6f3ec5e 0x6f40fc6 0x747ed38 0x72a6837 0x72a6939 0x72a6a31 0x735bce2 0x735bd9b 0x735bf4e 0x735c086 0x735c101 0x735c186 0x735c2e1 0x7425af5 0x747edc1 0x728e178 0x72a6bae 0x73b5274 0x7432e6c 0x7432ee7 0x7432f62 0x74eccd2 0x75e6286 0x9e08ac1 0x9d0cb62 0x9dc6e39 0x9e285fb 0x9f988cc 0x9f98f48 0x26ed527 0xa5ce02e /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-23T11:19:39.828 INFO:teuthology.orchestra.run.trial030.stderr:Reactor stalled for 32 ms on shard 2, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0x9f10dfc 0x9d045c3 0x9d40a13 0x9d43fc6 0x9d44234 0x9d4437c 0x9d4483b /lib64/libc.so.6+0x3fc2f /lib64/libc.so.6+0x191d45 /lib64/libasan.so.8+0xc748 /lib64/libasan.so.8+0x8eea /lib64/libasan.so.8+0xedc45 0xe1d5af 0x9c0545a 0x9c055fc 0x9c82450 0x9c84edb 0x9c850ec 0x9c85198 0x9c9f447 0x9ca45f1 0x9c052e1 0x9c042e1 0x7be8837 0x7bf23a6 0x7a51d33 0x7a8d6c5 0x711ff75 0x6f3ec5e 0x6f40fc6 0x747ed38 0x72a6837 0x72a6939 0x72a6a31 0x735bce2 0x735bd9b 0x735bf4e 0x735c086 0x735c101 0x735c186 0x735c2e1 0x7425af5 0x747edc1 0x728e178 0x72a6bae 0x73b5274 0x7432e6c 0x7432ee7 0x7432f62 0x74eccd2 0x75e6286 0x9e08ac1 0x9d0cb62 0x9dc6e39 0x9e285fb 0x9f988cc 0x9f98f48 0x26ed527 0xa5ce02e /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-23T11:19:39.828 INFO:teuthology.orchestra.run.trial030.stderr:kernel callstack: 0xffffffffffffff80 0xffffffff9ea91052 0xffffffff9f74d451 0xffffffff9f800ba2 2026-02-23T11:19:39.828 INFO:teuthology.orchestra.run.trial030.stderr:Reactor stalled for 32 ms on shard 3, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0x9f10dfc 0x9d045c3 0x9d40a13 0x9d43fc6 0x9d44234 0x9d4437c 0x9d4483b /lib64/libc.so.6+0x3fc2f /lib64/libasan.so.8+0x151f5 0x9b7dc74 0x9b7e0d5 0x9b7e348 0x9b7f348 0x9b84049 0x9b843c5 0x9c9ff46 0x9ca45f1 0x9c052e1 0x9c042e1 0x7be8837 0x7bf23a6 0x7a51d33 0x7a8d6c5 0x711ff75 0x6f3ec5e 0x6f40fc6 0x747ed38 0x72a6837 0x72a6939 0x72a6a31 0x735bce2 0x735bd9b 0x735bf4e 0x735c086 0x735c101 0x735c186 0x735c2e1 0x7425af5 0x747edc1 0x728e178 0x72a6bae 0x73b5274 0x7432e6c 0x7432ee7 0x7432f62 0x74eccd2 0x75e6286 0x9e08ac1 0x9d0cb62 0x9dc6e39 0x9e285fb 0x9f988cc 0x9f98f48 0x26ed527 0xa5ce02e /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-23T11:19:39.910 INFO:teuthology.orchestra.run.trial030.stderr:Reactor stalled for 65 ms on shard 0, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0x9f10dfc 0x9d045c3 0x9d40a13 0x9d43fc6 0x9d44234 0x9d4437c 0x9d4483b /lib64/libc.so.6+0x3fc2f 0x2a52044 0x7d7e22e 0x7d9da0c 0x7e39ed3 0x7dbe035 0x78750f3 0x7875de8 0x79b38b2 0x7833a0a 0x78a9398 0x78f345f 0x78f36d8 0x79a02cc 0x79a042b 0x79b3a15 0x7864b7c 0x799073e 0x79907ce 0x7990872 0x73eb37a 0x7409299 0x740c6cb 0x75440fc 0x9e08ac1 0x9d0cb62 0x9dc6e39 0x9e285fb 0x9e29e18 0x9c44517 0x9c44d52 0xecf43f /lib64/libc.so.6+0x2a60f /lib64/libc.so.6+0x2a6bf 0xb46f54 2026-02-23T11:19:39.924 INFO:teuthology.orchestra.run.trial030.stderr:Reactor stalled for 65 ms on shard 1, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0x9f10dfc 0x9d045c3 0x9d40a13 0x9d43fc6 0x9d44234 0x9d4437c 0x9d4483b /lib64/libc.so.6+0x3fc2f 0x2a52044 0x7d7e22e 0x7d9da0c 0x7e39ed3 0x7dbe035 0x78750f3 0x7875de8 0x79b38b2 0x7833a0a 0x78a9398 0x78f345f 0x78f36d8 0x79a02cc 0x79a042b 0x79b3a15 0x7864b7c 0x799073e 0x79907ce 0x7990872 0x73eb37a 0x7409299 0x740c6cb 0x75440fc 0x9e08ac1 0x9d0cb62 0x9dc6e39 0x9e285fb 0x9f988cc 0x9f98f48 0x26ed527 0xa5ce02e /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-23T11:19:39.924 INFO:teuthology.orchestra.run.trial030.stderr:Reactor stalled for 65 ms on shard 2, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0x9f10dfc 0x9d045c3 0x9d40a13 0x9d43fc6 0x9d44234 0x9d4437c 0x9d4483b /lib64/libc.so.6+0x3fc2f /lib64/libasan.so.8+0xc6592 /lib64/libasan.so.8+0x15197 0x2a5304b 0x7d7e215 0x7d9da0c 0x7e39ed3 0x7dbe035 0x78750f3 0x7875de8 0x79b38b2 0x7833a0a 0x78a9398 0x78f345f 0x78f36d8 0x79a02cc 0x79a042b 0x79b3a15 0x7864b7c 0x799073e 0x79907ce 0x7990872 0x73eb37a 0x7409299 0x740c6cb 0x75440fc 0x9e08ac1 0x9d0cb62 0x9dc6e39 0x9e285fb 0x9f988cc 0x9f98f48 0x26ed527 0xa5ce02e /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-23T11:19:39.924 INFO:teuthology.orchestra.run.trial030.stderr:Reactor stalled for 65 ms on shard 3, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0x9f10dfc 0x9d045c3 0x9d40a13 0x9d43fc6 0x9d44234 0x9d4437c 0x9d4483b /lib64/libc.so.6+0x3fc2f /lib64/libasan.so.8+0x129764 /lib64/libasan.so.8+0xc65d1 /lib64/libasan.so.8+0x15197 0x2a5304b 0x7d7e215 0x7d9da0c 0x7e39ed3 0x7dbe035 0x78750f3 0x7875de8 0x79b38b2 0x7833a0a 0x78a9398 0x78f345f 0x78f36d8 0x79a02cc 0x79a042b 0x79b3a15 0x7864b7c 0x799073e 0x79907ce 0x7990872 0x73eb37a 0x7409299 0x740c6cb 0x75440fc 0x9e08ac1 0x9d0cb62 0x9dc6e39 0x9e285fb 0x9f988cc 0x9f98f48 0x26ed527 0xa5ce02e /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-23T11:19:42.097 INFO:teuthology.orchestra.run.trial030.stderr:----------------------------------------------------- 2026-02-23T11:19:42.097 INFO:teuthology.orchestra.run.trial030.stderr:Suppressions used: 2026-02-23T11:19:42.097 INFO:teuthology.orchestra.run.trial030.stderr: count bytes template 2026-02-23T11:19:42.097 INFO:teuthology.orchestra.run.trial030.stderr: 1 8 MallocExtension::Register 2026-02-23T11:19:42.097 INFO:teuthology.orchestra.run.trial030.stderr:----------------------------------------------------- 2026-02-23T11:19:42.098 INFO:teuthology.orchestra.run.trial030.stderr: 2026-02-23T11:19:42.132 DEBUG:teuthology.orchestra.run.trial030:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2026-02-23T11:19:42.158 DEBUG:teuthology.orchestra.run.trial030:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/ceph.monmap 2026-02-23T11:19:42.311 INFO:teuthology.orchestra.run.trial030.stderr:INFO  osd - OSD::main: parsing early config 2026-02-23T11:19:42.368 INFO:teuthology.orchestra.run.trial030.stderr:INFO 2026-02-23 11:19:42,367 seastar - Reactor backend: linux-aio 2026-02-23T11:19:42.382 INFO:teuthology.orchestra.run.trial030.stderr:INFO 2026-02-23 11:19:42,381 [shard 0:main] io - IO queue was unable to find a suitable maximum request length, the search was cut-off early at: 16MB 2026-02-23T11:19:42.382 INFO:teuthology.orchestra.run.trial030.stderr:INFO 2026-02-23 11:19:42,381 [shard 0:main] io - IO queue was unable to find a suitable maximum request length, the search was cut-off early at: 16MB 2026-02-23T11:19:42.384 INFO:teuthology.orchestra.run.trial030.stderr:==35610==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2026-02-23T11:19:42.439 INFO:teuthology.orchestra.run.trial030.stderr:INFO 2026-02-23 11:19:42,438 [shard 0:main] osd - bypassing the config fetch due to --no-mon-config 2026-02-23T11:19:42.439 INFO:teuthology.orchestra.run.trial030.stderr:INFO 2026-02-23 11:19:42,438 [shard 0:main] osd - Configure option_name crimson_reactor_task_quota_ms with value : 0.500000 2026-02-23T11:19:42.439 INFO:teuthology.orchestra.run.trial030.stderr:INFO 2026-02-23 11:19:42,438 [shard 0:main] osd - Configure option_name crimson_reactor_idle_poll_time_us with value : 200 2026-02-23T11:19:42.439 INFO:teuthology.orchestra.run.trial030.stderr:INFO 2026-02-23 11:19:42,438 [shard 0:main] osd - get_early_config: set --thread-affinity 0 --smp 4 2026-02-23T11:19:42.439 INFO:teuthology.orchestra.run.trial030.stderr:DEBUG 2026-02-23 11:19:42,438 [shard 0:main] seastar - reactor::drain 2026-02-23T11:19:44.474 INFO:teuthology.orchestra.run.trial030.stderr:----------------------------------------------------- 2026-02-23T11:19:44.474 INFO:teuthology.orchestra.run.trial030.stderr:Suppressions used: 2026-02-23T11:19:44.475 INFO:teuthology.orchestra.run.trial030.stderr: count bytes template 2026-02-23T11:19:44.475 INFO:teuthology.orchestra.run.trial030.stderr: 1 8 MallocExtension::Register 2026-02-23T11:19:44.475 INFO:teuthology.orchestra.run.trial030.stderr:----------------------------------------------------- 2026-02-23T11:19:44.475 INFO:teuthology.orchestra.run.trial030.stderr: 2026-02-23T11:19:44.494 INFO:teuthology.orchestra.run.trial030.stderr:INFO  osd - OSD::main: early config parsed successfully 2026-02-23T11:19:44.494 INFO:teuthology.orchestra.run.trial030.stderr:INFO  osd - OSD::main: initializing seastar app_template 2026-02-23T11:19:44.536 INFO:teuthology.orchestra.run.trial030.stderr:INFO  osd - OSD::main: registering CLI options 2026-02-23T11:19:44.537 INFO:teuthology.orchestra.run.trial030.stderr:INFO  osd - OSD::main: entering seastar runtime 2026-02-23T11:19:44.544 INFO:teuthology.orchestra.run.trial030.stderr:INFO 2026-02-23 11:19:44,543 seastar - Reactor backend: linux-aio 2026-02-23T11:19:44.552 INFO:teuthology.orchestra.run.trial030.stderr:INFO 2026-02-23 11:19:44,551 [shard 3:main] io - IO queue was unable to find a suitable maximum request length, the search was cut-off early at: 16MB 2026-02-23T11:19:44.552 INFO:teuthology.orchestra.run.trial030.stderr:INFO 2026-02-23 11:19:44,551 [shard 3:main] io - IO queue was unable to find a suitable maximum request length, the search was cut-off early at: 16MB 2026-02-23T11:19:44.556 INFO:teuthology.orchestra.run.trial030.stderr:==35608==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2026-02-23T11:19:44.556 INFO:teuthology.orchestra.run.trial030.stderr:INFO 2026-02-23 11:19:44,556 [shard 0:main] osd - OSD::main: seastar runtime started 2026-02-23T11:19:44.617 INFO:teuthology.orchestra.run.trial030.stderr:DEBUG 2026-02-23 11:19:44,617 [shard 0:main] osd - OSD::main: initializing logger output 2026-02-23T11:19:44.696 INFO:teuthology.orchestra.run.trial030.stderr:Reactor stalled for 32 ms on shard 3, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0x9f10dfc 0x9d045c3 0x9d40a13 0x9d43fc6 0x9d44234 0x9d4437c 0x9d4483b /lib64/libc.so.6+0x3fc2f 0x10911b3 0xf2e3ce 0x9b7ddd8 0x9b7e0d5 0x9b7e348 0x9b7e5fe 0x9b7ebc0 0x9b83ce5 0x9b843c5 0x9c9ff46 0x9ca45f1 0x9c052e1 0x9c042e1 0x7be8837 0x7bf23a6 0x7a51d33 0x7a8d6c5 0x711ff75 0x6f9f19a 0x6fa2f9b 0x6fa8b45 0x7482004 0x72a89fa 0x72a8afc 0x72a8bf4 0x735cb16 0x735cbcf 0x735cd82 0x735ceba 0x735cf35 0x735cfba 0x735d115 0x7425e13 0x748208d 0x728f241 0x72a6bae 0x73b5274 0x7432e6c 0x7432ee7 0x7432f62 0x74eccd2 0x75e6286 0x9e08ac1 0x9d0cb62 0x9dc6e39 0x9e285fb 0x9f988cc 0x9f98f48 0x26ed527 0xa5ce02e /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-23T11:19:44.696 INFO:teuthology.orchestra.run.trial030.stderr:Reactor stalled for 32 ms on shard 1, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0x9f10dfc 0x9d045c3 0x9d40a13 0x9d43fc6 0x9d44234 0x9d4437c 0x9d4483b /lib64/libc.so.6+0x3fc2f /lib64/libasan.so.8+0xc59b /lib64/libasan.so.8+0x91c1 /lib64/libasan.so.8+0xee7dc 0x9a48dbb 0x9b02622 0x9b02738 0x9b02814 0x9b02ed0 0x9b02b49 0x9b03037 0x9b0343b 0x9b036f8 0x9b041b6 0x9c864c2 0x9c865bb 0x7be87ab 0x7bf23a6 0x7a51d33 0x7a8d6c5 0x711ff75 0x6f9f19a 0x6fa2f9b 0x6fa8b45 0x7482004 0x72a89fa 0x72a8afc 0x72a8bf4 0x735cb16 0x735cbcf 0x735cd82 0x735ceba 0x735cf35 0x735cfba 0x735d115 0x7425e13 0x748208d 0x728f241 0x72a6bae 0x73b5274 0x7432e6c 0x7432ee7 0x7432f62 0x74eccd2 0x75e6286 0x9e08ac1 0x9d0cb62 0x9dc6e39 0x9e285fb 0x9f988cc 0x9f98f48 0x26ed527 0xa5ce02e /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-23T11:19:44.696 INFO:teuthology.orchestra.run.trial030.stderr:Reactor stalled for 32 ms on shard 2, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0x9f10dfc 0x9d045c3 0x9d40a13 0x9d43fc6 0x9d44234 0x9d4437c 0x9d4483b /lib64/libc.so.6+0x3fc2f /lib64/libasan.so.8+0xc6592 /lib64/libasan.so.8+0x15197 0x1125e3c 0xf2e3c3 0x9b81833 0x9ca67bb 0x77f3d8d 0x7be876e 0x7bf23a6 0x7a51d33 0x7a8d6c5 0x711ff75 0x6f9f19a 0x6fa2f9b 0x6fa8b45 0x7482004 0x72a89fa 0x72a8afc 0x72a8bf4 0x735cb16 0x735cbcf 0x735cd82 0x735ceba 0x735cf35 0x735cfba 0x735d115 0x7425e13 0x748208d 0x728f241 0x72a6bae 0x73b5274 0x7432e6c 0x7432ee7 0x7432f62 0x74eccd2 0x75e6286 0x9e08ac1 0x9d0cb62 0x9dc6e39 0x9e285fb 0x9f988cc 0x9f98f48 0x26ed527 0xa5ce02e /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-23T11:19:44.769 INFO:teuthology.orchestra.run.trial030.stderr:Reactor stalled for 65 ms on shard 2, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0x9f10dfc 0x9d045c3 0x9d40a13 0x9d43fc6 0x9d44234 0x9d4437c 0x9d4483b /lib64/libc.so.6+0x3fc2f /lib64/libasan.so.8+0x10b6c3 0x35ff75d 0x62e7ef5 0x7f7fae3 0x7f7fdba 0x7f7fee6 0x7f82371 0x7f82fc9 0x7dc31d8 0x7e39297 0x7e5ae04 0x8f80d69 0x8f836d7 0x787a27f 0x78aad7c 0x6fa0365 0x75590f6 0x755924a 0x9e08ac1 0x9d0cb62 0x9dc6e39 0x9e285fb 0x9f988cc 0x9f98f48 0x26ed527 0xa5ce02e /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-23T11:19:44.770 INFO:teuthology.orchestra.run.trial030.stderr:Reactor stalled for 65 ms on shard 3, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0x9f10dfc 0x9d045c3 0x9d40a13 0x9d43fc6 0x9d44234 0x9d4437c 0x9d4483b /lib64/libc.so.6+0x3fc2f /lib64/libasan.so.8+0xa323 /lib64/libasan.so.8+0xc488 /lib64/libasan.so.8+0x91c1 /lib64/libasan.so.8+0xee7dc 0xbb8cb5 0xbb8ebe 0xbb93fb 0xe05f94 0x18eb398 0x9c87dd0 0x9c88176 0x9ca186a 0x9ca45f1 0x9c052e1 0x9c042e1 0x7e58655 0x7e5b450 0x8f80d69 0x8f836d7 0x787a27f 0x78aad7c 0x6fa0365 0x75590f6 0x755924a 0x9e08ac1 0x9d0cb62 0x9dc6e39 0x9e285fb 0x9f988cc 0x9f98f48 0x26ed527 0xa5ce02e /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-23T11:19:44.793 INFO:teuthology.orchestra.run.trial030.stderr:Reactor stalled for 65 ms on shard 0, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0x9f10dfc 0x9d045c3 0x9d40a13 0x9d43fc6 0x9d44234 0x9d4437c 0x9d4483b /lib64/libc.so.6+0x3fc2f /lib64/libasan.so.8+0xc6592 /lib64/libasan.so.8+0x15197 0x2a5304b 0x7d7e215 0x7d9da0c 0x7e39ed3 0x7dbe035 0x78750f3 0x7875de8 0x79b38b2 0x7833a0a 0x78a9398 0x78f345f 0x78f36d8 0x79a02cc 0x79a042b 0x79b3a15 0x7864b7c 0x799073e 0x79907ce 0x7990872 0x73eb37a 0x7409299 0x740c6cb 0x75440fc 0x9e08ac1 0x9d0cb62 0x9dc6e39 0x9e285fb 0x9e29e18 0x9c44517 0x9c44d52 0xecf43f /lib64/libc.so.6+0x2a60f /lib64/libc.so.6+0x2a6bf 0xb46f54 2026-02-23T11:19:44.802 INFO:teuthology.orchestra.run.trial030.stderr:Reactor stalled for 32 ms on shard 3, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0x9f10dfc 0x9d045c3 0x9d40a13 0x9d43fc6 0x9d44234 0x9d4437c 0x9d4483b /lib64/libc.so.6+0x3fc2f /lib64/libasan.so.8+0x15277 0x2a5304b 0x7d7e215 0x7d9da0c 0x7e39ed3 0x7dbe035 0x78750f3 0x7875de8 0x79b38b2 0x7833a0a 0x78a9398 0x78f345f 0x78f36d8 0x79a02cc 0x79a042b 0x79b3a15 0x7864b7c 0x799073e 0x79907ce 0x7990872 0x73eb37a 0x7409299 0x740c6cb 0x75440fc 0x9e08ac1 0x9d0cb62 0x9dc6e39 0x9e285fb 0x9f988cc 0x9f98f48 0x26ed527 0xa5ce02e /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-23T11:19:44.832 INFO:teuthology.orchestra.run.trial030.stderr:Reactor stalled for 65 ms on shard 1, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0x9f10dfc 0x9d045c3 0x9d40a13 0x9d43fc6 0x9d44234 0x9d4437c 0x9d4483b /lib64/libc.so.6+0x3fc2f /lib64/libasan.so.8+0xc6592 /lib64/libasan.so.8+0x15197 0x2a5304b 0x7d7e215 0x7d9da0c 0x7e39ed3 0x7dbe035 0x78750f3 0x7875de8 0x79b38b2 0x7833a0a 0x78a9398 0x78f345f 0x78f36d8 0x79a02cc 0x79a042b 0x79b3a15 0x7864b7c 0x799073e 0x79907ce 0x7990872 0x73eb37a 0x7409299 0x740c6cb 0x75440fc 0x9e08ac1 0x9d0cb62 0x9dc6e39 0x9e285fb 0x9f988cc 0x9f98f48 0x26ed527 0xa5ce02e /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-23T11:19:44.835 INFO:teuthology.orchestra.run.trial030.stderr:Reactor stalled for 65 ms on shard 2, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0x9f10dfc 0x9d045c3 0x9d40a13 0x9d43fc6 0x9d44234 0x9d4437c 0x9d4483b /lib64/libc.so.6+0x3fc2f /lib64/libasan.so.8+0xfcaf3 /lib64/libasan.so.8+0xc6592 /lib64/libasan.so.8+0x15197 0x2a5304b 0x7d7e215 0x7d9da0c 0x7e39ed3 0x7dbe035 0x78750f3 0x7875de8 0x79b38b2 0x7833a0a 0x78a9398 0x78f345f 0x78f36d8 0x79a02cc 0x79a042b 0x79b3a15 0x7864b7c 0x799073e 0x79907ce 0x7990872 0x73eb37a 0x7409299 0x740c6cb 0x75440fc 0x9e08ac1 0x9d0cb62 0x9dc6e39 0x9e285fb 0x9f988cc 0x9f98f48 0x26ed527 0xa5ce02e /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-23T11:19:44.959 INFO:teuthology.orchestra.run.trial030.stderr:Reactor stalled for 32 ms on shard 1, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0x9f10dfc 0x9d045c3 0x9d40a13 0x9d43fc6 0x9d44234 0x9d4437c 0x9d4483b /lib64/libc.so.6+0x3fc2f 0x1125d31 0xf2e3c3 0x9b7dc0b 0x9b7e0d5 0x9b7e348 0x9b7e5fe 0x9b7ebc0 0x9b83ce5 0x9b843c5 0x9c9ff46 0x9ca45f1 0x9c052e1 0x9c042e1 0x7be8837 0x7bf23a6 0x7a51d33 0x7a8d6c5 0x711ff75 0x6f3ec5e 0x6f40fc6 0x747ed38 0x72a6837 0x72a6939 0x72a6a31 0x735bce2 0x735bd9b 0x735bf4e 0x735c086 0x735c101 0x735c186 0x735c2e1 0x7425af5 0x747edc1 0x728e178 0x72a6bae 0x73b5274 0x7432e6c 0x7432ee7 0x7432f62 0x74eccd2 0x75e6286 0x9e08ac1 0x9d0cb62 0x9dc6e39 0x9e285fb 0x9f988cc 0x9f98f48 0x26ed527 0xa5ce02e /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-23T11:19:44.959 INFO:teuthology.orchestra.run.trial030.stderr:Reactor stalled for 32 ms on shard 2, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0x9f10dfc 0x9d045c3 0x9d40a13 0x9d43fc6 0x9d44234 0x9d4437c 0x9d4483b /lib64/libc.so.6+0x3fc2f 0x316b578 0x9b7e05e 0x9b7e348 0x9b7e5fe 0x9b7f0ad 0x9b7f5ab 0x9b7fbdb 0x9b7fd45 0x9c8fd63 0x9ca2b6a 0x9c052e1 0x9c042e1 0x7be8837 0x7bf23a6 0x7a51d33 0x7a8d6c5 0x711ff75 0x6f3ec5e 0x6f40fc6 0x747ed38 0x72a6837 0x72a6939 0x72a6a31 0x735bce2 0x735bd9b 0x735bf4e 0x735c086 0x735c101 0x735c186 0x735c2e1 0x7425af5 0x747edc1 0x728e178 0x72a6bae 0x73b5274 0x7432e6c 0x7432ee7 0x7432f62 0x74eccd2 0x75e6286 0x9e08ac1 0x9d0cb62 0x9dc6e39 0x9e285fb 0x9f988cc 0x9f98f48 0x26ed527 0xa5ce02e /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-23T11:19:44.959 INFO:teuthology.orchestra.run.trial030.stderr:Reactor stalled for 32 ms on shard 3, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0x9f10dfc 0x9d045c3 0x9d40a13 0x9d43fc6 0x9d44234 0x9d4437c 0x9d4483b /lib64/libc.so.6+0x3fc2f /lib64/ld-linux-x86-64.so.2+0x16ec6 /lib64/libasan.so.8+0xc65b1 /lib64/libasan.so.8+0x157c7 0x1091317 0xf2e3ce 0x9c906c3 0x9c90cb0 0x9c91033 0x9ca118e 0x9ca45f1 0x9c052e1 0x9c042e1 0x7be8837 0x7bf23a6 0x7a51d33 0x7a8d6c5 0x711ff75 0x6f3ec5e 0x6f40fc6 0x747ed38 0x72a6837 0x72a6939 0x72a6a31 0x735bce2 0x735bd9b 0x735bf4e 0x735c086 0x735c101 0x735c186 0x735c2e1 0x7425af5 0x747edc1 0x728e178 0x72a6bae 0x73b5274 0x7432e6c 0x7432ee7 0x7432f62 0x74eccd2 0x75e6286 0x9e08ac1 0x9d0cb62 0x9dc6e39 0x9e285fb 0x9f988cc 0x9f98f48 0x26ed527 0xa5ce02e /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-23T11:19:45.041 INFO:teuthology.orchestra.run.trial030.stderr:Reactor stalled for 65 ms on shard 0, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0x9f10dfc 0x9d045c3 0x9d40a13 0x9d43fc6 0x9d44234 0x9d4437c 0x9d4483b /lib64/libc.so.6+0x3fc2f /lib64/libc.so.6+0x8cbff /lib64/libasan.so.8+0x10069c /lib64/libasan.so.8+0xc65b9 /lib64/libasan.so.8+0x15197 0x2a5304b 0x7d7e215 0x7d9da0c 0x7e39ed3 0x7dbe035 0x78750f3 0x7875de8 0x79b38b2 0x7833a0a 0x78a9398 0x78f345f 0x78f36d8 0x79a02cc 0x79a042b 0x79b3a15 0x7864b7c 0x799073e 0x79907ce 0x7990872 0x73eb37a 0x7409299 0x740c6cb 0x75440fc 0x9e08ac1 0x9d0cb62 0x9dc6e39 0x9e285fb 0x9e29e18 0x9c44517 0x9c44d52 0xecf43f /lib64/libc.so.6+0x2a60f /lib64/libc.so.6+0x2a6bf 0xb46f54 2026-02-23T11:19:45.055 INFO:teuthology.orchestra.run.trial030.stderr:Reactor stalled for 65 ms on shard 1, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0x9f10dfc 0x9d045c3 0x9d40a13 0x9d43fc6 0x9d44234 0x9d4437c 0x9d4483b /lib64/libc.so.6+0x3fc2f 0x29fe1c4 0x2a52fec 0x7d7e215 0x7d9da0c 0x7e39ed3 0x7dbe035 0x78750f3 0x7875de8 0x79b38b2 0x7833a0a 0x78a9398 0x78f345f 0x78f36d8 0x79a02cc 0x79a042b 0x79b3a15 0x7864b7c 0x799073e 0x79907ce 0x7990872 0x73eb37a 0x7409299 0x740c6cb 0x75440fc 0x9e08ac1 0x9d0cb62 0x9dc6e39 0x9e285fb 0x9f988cc 0x9f98f48 0x26ed527 0xa5ce02e /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-23T11:19:45.056 INFO:teuthology.orchestra.run.trial030.stderr:Reactor stalled for 65 ms on shard 2, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0x9f10dfc 0x9d045c3 0x9d40a13 0x9d43fc6 0x9d44234 0x9d4437c 0x9d4483b /lib64/libc.so.6+0x3fc2f /lib64/libasan.so.8+0x129787 /lib64/libasan.so.8+0xc65d1 /lib64/libasan.so.8+0x15197 0x2a5304b 0x7d7e215 0x7d9da0c 0x7e39ed3 0x7dbe035 0x78750f3 0x7875de8 0x79b38b2 0x7833a0a 0x78a9398 0x78f345f 0x78f36d8 0x79a02cc 0x79a042b 0x79b3a15 0x7864b7c 0x799073e 0x79907ce 0x7990872 0x73eb37a 0x7409299 0x740c6cb 0x75440fc 0x9e08ac1 0x9d0cb62 0x9dc6e39 0x9e285fb 0x9f988cc 0x9f98f48 0x26ed527 0xa5ce02e /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-23T11:19:45.056 INFO:teuthology.orchestra.run.trial030.stderr:Reactor stalled for 65 ms on shard 3, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0x9f10dfc 0x9d045c3 0x9d40a13 0x9d43fc6 0x9d44234 0x9d4437c 0x9d4483b /lib64/libc.so.6+0x3fc2f /lib64/libasan.so.8+0xc6592 /lib64/libasan.so.8+0x15197 0x2a5304b 0x7d7e215 0x7d9da0c 0x7e39ed3 0x7dbe035 0x78750f3 0x7875de8 0x79b38b2 0x7833a0a 0x78a9398 0x78f345f 0x78f36d8 0x79a02cc 0x79a042b 0x79b3a15 0x7864b7c 0x799073e 0x79907ce 0x7990872 0x73eb37a 0x7409299 0x740c6cb 0x75440fc 0x9e08ac1 0x9d0cb62 0x9dc6e39 0x9e285fb 0x9f988cc 0x9f98f48 0x26ed527 0xa5ce02e /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-23T11:19:47.208 INFO:teuthology.orchestra.run.trial030.stderr:----------------------------------------------------- 2026-02-23T11:19:47.208 INFO:teuthology.orchestra.run.trial030.stderr:Suppressions used: 2026-02-23T11:19:47.208 INFO:teuthology.orchestra.run.trial030.stderr: count bytes template 2026-02-23T11:19:47.208 INFO:teuthology.orchestra.run.trial030.stderr: 1 8 MallocExtension::Register 2026-02-23T11:19:47.208 INFO:teuthology.orchestra.run.trial030.stderr:----------------------------------------------------- 2026-02-23T11:19:47.208 INFO:teuthology.orchestra.run.trial030.stderr: 2026-02-23T11:19:47.244 DEBUG:teuthology.orchestra.run.trial030:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2026-02-23T11:19:47.269 DEBUG:teuthology.orchestra.run.trial030:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/ceph.monmap 2026-02-23T11:19:47.422 INFO:teuthology.orchestra.run.trial030.stderr:INFO  osd - OSD::main: parsing early config 2026-02-23T11:19:47.476 INFO:teuthology.orchestra.run.trial030.stderr:INFO 2026-02-23 11:19:47,475 seastar - Reactor backend: linux-aio 2026-02-23T11:19:47.485 INFO:teuthology.orchestra.run.trial030.stderr:INFO 2026-02-23 11:19:47,485 [shard 0:main] io - IO queue was unable to find a suitable maximum request length, the search was cut-off early at: 16MB 2026-02-23T11:19:47.486 INFO:teuthology.orchestra.run.trial030.stderr:INFO 2026-02-23 11:19:47,485 [shard 0:main] io - IO queue was unable to find a suitable maximum request length, the search was cut-off early at: 16MB 2026-02-23T11:19:47.488 INFO:teuthology.orchestra.run.trial030.stderr:==35674==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2026-02-23T11:19:47.542 INFO:teuthology.orchestra.run.trial030.stderr:INFO 2026-02-23 11:19:47,542 [shard 0:main] osd - bypassing the config fetch due to --no-mon-config 2026-02-23T11:19:47.542 INFO:teuthology.orchestra.run.trial030.stderr:INFO 2026-02-23 11:19:47,542 [shard 0:main] osd - Configure option_name crimson_reactor_task_quota_ms with value : 0.500000 2026-02-23T11:19:47.542 INFO:teuthology.orchestra.run.trial030.stderr:INFO 2026-02-23 11:19:47,542 [shard 0:main] osd - Configure option_name crimson_reactor_idle_poll_time_us with value : 200 2026-02-23T11:19:47.542 INFO:teuthology.orchestra.run.trial030.stderr:INFO 2026-02-23 11:19:47,542 [shard 0:main] osd - get_early_config: set --thread-affinity 0 --smp 4 2026-02-23T11:19:47.543 INFO:teuthology.orchestra.run.trial030.stderr:DEBUG 2026-02-23 11:19:47,542 [shard 0:main] seastar - reactor::drain 2026-02-23T11:19:49.581 INFO:teuthology.orchestra.run.trial030.stderr:----------------------------------------------------- 2026-02-23T11:19:49.581 INFO:teuthology.orchestra.run.trial030.stderr:Suppressions used: 2026-02-23T11:19:49.582 INFO:teuthology.orchestra.run.trial030.stderr: count bytes template 2026-02-23T11:19:49.582 INFO:teuthology.orchestra.run.trial030.stderr: 1 8 MallocExtension::Register 2026-02-23T11:19:49.582 INFO:teuthology.orchestra.run.trial030.stderr:----------------------------------------------------- 2026-02-23T11:19:49.582 INFO:teuthology.orchestra.run.trial030.stderr: 2026-02-23T11:19:49.601 INFO:teuthology.orchestra.run.trial030.stderr:INFO  osd - OSD::main: early config parsed successfully 2026-02-23T11:19:49.601 INFO:teuthology.orchestra.run.trial030.stderr:INFO  osd - OSD::main: initializing seastar app_template 2026-02-23T11:19:49.645 INFO:teuthology.orchestra.run.trial030.stderr:INFO  osd - OSD::main: registering CLI options 2026-02-23T11:19:49.646 INFO:teuthology.orchestra.run.trial030.stderr:INFO  osd - OSD::main: entering seastar runtime 2026-02-23T11:19:49.652 INFO:teuthology.orchestra.run.trial030.stderr:INFO 2026-02-23 11:19:49,652 seastar - Reactor backend: linux-aio 2026-02-23T11:19:49.664 INFO:teuthology.orchestra.run.trial030.stderr:INFO 2026-02-23 11:19:49,664 [shard 1:main] io - IO queue was unable to find a suitable maximum request length, the search was cut-off early at: 16MB 2026-02-23T11:19:49.664 INFO:teuthology.orchestra.run.trial030.stderr:INFO 2026-02-23 11:19:49,664 [shard 1:main] io - IO queue was unable to find a suitable maximum request length, the search was cut-off early at: 16MB 2026-02-23T11:19:49.668 INFO:teuthology.orchestra.run.trial030.stderr:==35672==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2026-02-23T11:19:49.668 INFO:teuthology.orchestra.run.trial030.stderr:INFO 2026-02-23 11:19:49,668 [shard 0:main] osd - OSD::main: seastar runtime started 2026-02-23T11:19:49.724 INFO:teuthology.orchestra.run.trial030.stderr:DEBUG 2026-02-23 11:19:49,724 [shard 0:main] osd - OSD::main: initializing logger output 2026-02-23T11:19:49.805 INFO:teuthology.orchestra.run.trial030.stderr:Reactor stalled for 32 ms on shard 2, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0x9f10dfc 0x9d045c3 0x9d40a13 0x9d43fc6 0x9d44234 0x9d4437c 0x9d4483b /lib64/libc.so.6+0x3fc2f 0x9b89cd9 0x9c9ba76 0x9c9c048 0x9ca6515 0x77f3d8d 0x7be8169 0x7bf23a6 0x7a51d33 0x7a8d6c5 0x711ff75 0x6f9f19a 0x6fa2f9b 0x6fa8b45 0x7482004 0x72a89fa 0x72a8afc 0x72a8bf4 0x735cb16 0x735cbcf 0x735cd82 0x735ceba 0x735cf35 0x735cfba 0x735d115 0x7425e13 0x748208d 0x728f241 0x72a6bae 0x73b5274 0x7432e6c 0x7432ee7 0x7432f62 0x74eccd2 0x75e6286 0x9e08ac1 0x9d0cb62 0x9dc6e39 0x9e285fb 0x9f988cc 0x9f98f48 0x26ed527 0xa5ce02e /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-23T11:19:49.805 INFO:teuthology.orchestra.run.trial030.stderr:Reactor stalled for 32 ms on shard 1, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0x9f10dfc 0x9d045c3 0x9d40a13 0x9d43fc6 0x9d44234 0x9d4437c 0x9d4483b /lib64/libc.so.6+0x3fc2f /lib64/libasan.so.8+0xb04b9 /lib64/libasan.so.8+0xb1437 0xb69729 0x10912a0 0xf2e3ce 0x9b811ae 0x9b8145b 0x9ca67bb 0x77f3d8d 0x7be876e 0x7bf23a6 0x7a51d33 0x7a8d6c5 0x711ff75 0x6f9f19a 0x6fa2f9b 0x6fa8b45 0x7482004 0x72a89fa 0x72a8afc 0x72a8bf4 0x735cb16 0x735cbcf 0x735cd82 0x735ceba 0x735cf35 0x735cfba 0x735d115 0x7425e13 0x748208d 0x728f241 0x72a6bae 0x73b5274 0x7432e6c 0x7432ee7 0x7432f62 0x74eccd2 0x75e6286 0x9e08ac1 0x9d0cb62 0x9dc6e39 0x9e285fb 0x9f988cc 0x9f98f48 0x26ed527 0xa5ce02e /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-23T11:19:49.805 INFO:teuthology.orchestra.run.trial030.stderr:Reactor stalled for 32 ms on shard 3, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0x9f10dfc 0x9d045c3 0x9d40a13 0x9d43fc6 0x9d44234 0x9d4437c 0x9d4483b /lib64/libc.so.6+0x3fc2f /lib64/libasan.so.8+0x11910d /lib64/libasan.so.8+0xebbf /lib64/libasan.so.8+0xec776 0xcb0505 0x6f6f0b6 0x7be8a9b 0x7bf23a6 0x7a51d33 0x7a8d6c5 0x711ff75 0x6f9f19a 0x6fa2f9b 0x6fa8b45 0x7482004 0x72a89fa 0x72a8afc 0x72a8bf4 0x735cb16 0x735cbcf 0x735cd82 0x735ceba 0x735cf35 0x735cfba 0x735d115 0x7425e13 0x748208d 0x728f241 0x72a6bae 0x73b5274 0x7432e6c 0x7432ee7 0x7432f62 0x74eccd2 0x75e6286 0x9e08ac1 0x9d0cb62 0x9dc6e39 0x9e285fb 0x9f988cc 0x9f98f48 0x26ed527 0xa5ce02e /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-23T11:19:49.837 INFO:teuthology.orchestra.run.trial030.stderr:Reactor stalled for 65 ms on shard 0, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0x9f10dfc 0x9d045c3 0x9d40a13 0x9d43fc6 0x9d44234 0x9d4437c 0x9d4483b /lib64/libc.so.6+0x3fc2f /lib64/libasan.so.8+0xc597 /lib64/libasan.so.8+0x8eea /lib64/libasan.so.8+0xedc45 0xe1d5af 0x9ca5d35 0x77f3d8d 0x7e57e16 0x7e5b450 0x8f80d69 0x8f836d7 0x787a27f 0x78aad7c 0x6fa0365 0x75590f6 0x755924a 0x9e08ac1 0x9d0cb62 0x9dc6e39 0x9e285fb 0x9e29e18 0x9c44517 0x9c44d52 0xecf43f /lib64/libc.so.6+0x2a60f /lib64/libc.so.6+0x2a6bf 0xb46f54 2026-02-23T11:19:49.838 INFO:teuthology.orchestra.run.trial030.stderr:kernel callstack: 0xffffffffffffff80 0xffffffff9ea75c69 0xffffffff9ec1f8c7 0xffffffff9ec1fb20 0xffffffff9ea9525e 0xffffffff9ede61a9 0xffffffff9edc9715 0xffffffff9edd3aeb 0xffffffff9edd3f4b 0xffffffff9ea910d2 0xffffffff9f74d451 0xffffffff9f800ba2 2026-02-23T11:19:49.903 INFO:teuthology.orchestra.run.trial030.stderr:Reactor stalled for 65 ms on shard 0, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0x9f10dfc 0x9d045c3 0x9d40a13 0x9d43fc6 0x9d44234 0x9d4437c 0x9d4483b /lib64/libc.so.6+0x3fc2f /lib64/libasan.so.8+0xc6583 /lib64/libasan.so.8+0x15197 0x2a5304b 0x7d7e215 0x7d9da0c 0x7e39ed3 0x7dbe035 0x78750f3 0x7875de8 0x79b38b2 0x7833a0a 0x78a9398 0x78f345f 0x78f36d8 0x79a02cc 0x79a042b 0x79b3a15 0x7864b7c 0x799073e 0x79907ce 0x7990872 0x73eb37a 0x7409299 0x740c6cb 0x75440fc 0x9e08ac1 0x9d0cb62 0x9dc6e39 0x9e285fb 0x9e29e18 0x9c44517 0x9c44d52 0xecf43f /lib64/libc.so.6+0x2a60f /lib64/libc.so.6+0x2a6bf 0xb46f54 2026-02-23T11:19:49.942 INFO:teuthology.orchestra.run.trial030.stderr:Reactor stalled for 65 ms on shard 3, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0x9f10dfc 0x9d045c3 0x9d40a13 0x9d43fc6 0x9d44234 0x9d4437c 0x9d4483b /lib64/libc.so.6+0x3fc2f 0x2a52f60 0x7d7e215 0x7d9da0c 0x7e39ed3 0x7dbe035 0x78750f3 0x7875de8 0x79b38b2 0x7833a0a 0x78a9398 0x78f345f 0x78f36d8 0x79a02cc 0x79a042b 0x79b3a15 0x7864b7c 0x799073e 0x79907ce 0x7990872 0x73eb37a 0x7409299 0x740c6cb 0x75440fc 0x9e08ac1 0x9d0cb62 0x9dc6e39 0x9e285fb 0x9f988cc 0x9f98f48 0x26ed527 0xa5ce02e /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-23T11:19:49.943 INFO:teuthology.orchestra.run.trial030.stderr:Reactor stalled for 65 ms on shard 2, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0x9f10dfc 0x9d045c3 0x9d40a13 0x9d43fc6 0x9d44234 0x9d4437c 0x9d4483b /lib64/libc.so.6+0x3fc2f 0x29fe1f6 0x2a52fec 0x7d7e215 0x7d9da0c 0x7e39ed3 0x7dbe035 0x78750f3 0x7875de8 0x79b38b2 0x7833a0a 0x78a9398 0x78f345f 0x78f36d8 0x79a02cc 0x79a042b 0x79b3a15 0x7864b7c 0x799073e 0x79907ce 0x7990872 0x73eb37a 0x7409299 0x740c6cb 0x75440fc 0x9e08ac1 0x9d0cb62 0x9dc6e39 0x9e285fb 0x9f988cc 0x9f98f48 0x26ed527 0xa5ce02e /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-23T11:19:49.943 INFO:teuthology.orchestra.run.trial030.stderr:Reactor stalled for 65 ms on shard 1, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0x9f10dfc 0x9d045c3 0x9d40a13 0x9d43fc6 0x9d44234 0x9d4437c 0x9d4483b /lib64/libc.so.6+0x3fc2f 0x7d7e243 0x7d9da0c 0x7e39ed3 0x7dbe035 0x78750f3 0x7875de8 0x79b38b2 0x7833a0a 0x78a9398 0x78f345f 0x78f36d8 0x79a02cc 0x79a042b 0x79b3a15 0x7864b7c 0x799073e 0x79907ce 0x7990872 0x73eb37a 0x7409299 0x740c6cb 0x75440fc 0x9e08ac1 0x9d0cb62 0x9dc6e39 0x9e285fb 0x9f988cc 0x9f98f48 0x26ed527 0xa5ce02e /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-23T11:19:50.068 INFO:teuthology.orchestra.run.trial030.stderr:Reactor stalled for 32 ms on shard 1, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0x9f10dfc 0x9d045c3 0x9d40a13 0x9d43fc6 0x9d44234 0x9d4437c 0x9d4483b /lib64/libc.so.6+0x3fc2f 0x316f0cc 0x9c9fabe 0x9ca45f1 0x9c052e1 0x9c042e1 0x7be8837 0x7bf23a6 0x7a51d33 0x7a8d6c5 0x711ff75 0x6f3ec5e 0x6f40fc6 0x747ed38 0x72a6837 0x72a6939 0x72a6a31 0x735bce2 0x735bd9b 0x735bf4e 0x735c086 0x735c101 0x735c186 0x735c2e1 0x7425af5 0x747edc1 0x728e178 0x72a6bae 0x73b5274 0x7432e6c 0x7432ee7 0x7432f62 0x74eccd2 0x75e6286 0x9e08ac1 0x9d0cb62 0x9dc6e39 0x9e285fb 0x9f988cc 0x9f98f48 0x26ed527 0xa5ce02e /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-23T11:19:50.068 INFO:teuthology.orchestra.run.trial030.stderr:Reactor stalled for 32 ms on shard 3, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0x9f10dfc 0x9d045c3 0x9d40a13 0x9d43fc6 0x9d44234 0x9d4437c 0x9d4483b /lib64/libc.so.6+0x3fc2f 0x9c96405 0x9c9ba0a 0x9c9c048 0x9ca693e 0x77f3d8d 0x7be8169 0x7bf23a6 0x7a51d33 0x7a8d6c5 0x711ff75 0x6f3ec5e 0x6f40fc6 0x747ed38 0x72a6837 0x72a6939 0x72a6a31 0x735bce2 0x735bd9b 0x735bf4e 0x735c086 0x735c101 0x735c186 0x735c2e1 0x7425af5 0x747edc1 0x728e178 0x72a6bae 0x73b5274 0x7432e6c 0x7432ee7 0x7432f62 0x74eccd2 0x75e6286 0x9e08ac1 0x9d0cb62 0x9dc6e39 0x9e285fb 0x9f988cc 0x9f98f48 0x26ed527 0xa5ce02e /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-23T11:19:50.068 INFO:teuthology.orchestra.run.trial030.stderr:Reactor stalled for 32 ms on shard 2, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0x9f10dfc 0x9d045c3 0x9d40a13 0x9d43fc6 0x9d44234 0x9d4437c 0x9d4483b /lib64/libc.so.6+0x3fc2f /lib64/libasan.so.8+0x1297ad /lib64/libasan.so.8+0xc65d1 /lib64/libasan.so.8+0x15197 0x3165fa6 0x4d4ab79 0x9c7cd0b 0x9c87c81 0x9c88176 0x9c9f909 0x9ca45f1 0x9c052e1 0x9c042e1 0x7be8837 0x7bf23a6 0x7a51d33 0x7a8d6c5 0x711ff75 0x6f3ec5e 0x6f40fc6 0x747ed38 0x72a6837 0x72a6939 0x72a6a31 0x735bce2 0x735bd9b 0x735bf4e 0x735c086 0x735c101 0x735c186 0x735c2e1 0x7425af5 0x747edc1 0x728e178 0x72a6bae 0x73b5274 0x7432e6c 0x7432ee7 0x7432f62 0x74eccd2 0x75e6286 0x9e08ac1 0x9d0cb62 0x9dc6e39 0x9e285fb 0x9f988cc 0x9f98f48 0x26ed527 0xa5ce02e /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-23T11:19:50.150 INFO:teuthology.orchestra.run.trial030.stderr:Reactor stalled for 65 ms on shard 0, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0x9f10dfc 0x9d045c3 0x9d40a13 0x9d43fc6 0x9d44234 0x9d4437c 0x9d4483b /lib64/libc.so.6+0x3fc2f 0x2a5236c 0x7d7e243 0x7d9da0c 0x7e39ed3 0x7dbe035 0x78750f3 0x7875de8 0x79b38b2 0x7833a0a 0x78a9398 0x78f345f 0x78f36d8 0x79a02cc 0x79a042b 0x79b3a15 0x7864b7c 0x799073e 0x79907ce 0x7990872 0x73eb37a 0x7409299 0x740c6cb 0x75440fc 0x9e08ac1 0x9d0cb62 0x9dc6e39 0x9e285fb 0x9e29e18 0x9c44517 0x9c44d52 0xecf43f /lib64/libc.so.6+0x2a60f /lib64/libc.so.6+0x2a6bf 0xb46f54 2026-02-23T11:19:50.165 INFO:teuthology.orchestra.run.trial030.stderr:Reactor stalled for 65 ms on shard 3, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0x9f10dfc 0x9d045c3 0x9d40a13 0x9d43fc6 0x9d44234 0x9d4437c 0x9d4483b /lib64/libc.so.6+0x3fc2f /lib64/libasan.so.8+0xc6592 /lib64/libasan.so.8+0x15197 0x2a5304b 0x7d7e215 0x7d9da0c 0x7e39ed3 0x7dbe035 0x78750f3 0x7875de8 0x79b38b2 0x7833a0a 0x78a9398 0x78f345f 0x78f36d8 0x79a02cc 0x79a042b 0x79b3a15 0x7864b7c 0x799073e 0x79907ce 0x7990872 0x73eb37a 0x7409299 0x740c6cb 0x75440fc 0x9e08ac1 0x9d0cb62 0x9dc6e39 0x9e285fb 0x9f988cc 0x9f98f48 0x26ed527 0xa5ce02e /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-23T11:19:50.165 INFO:teuthology.orchestra.run.trial030.stderr:Reactor stalled for 65 ms on shard 2, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0x9f10dfc 0x9d045c3 0x9d40a13 0x9d43fc6 0x9d44234 0x9d4437c 0x9d4483b /lib64/libc.so.6+0x3fc2f 0x7d7e232 0x7d9da0c 0x7e39ed3 0x7dbe035 0x78750f3 0x7875de8 0x79b38b2 0x7833a0a 0x78a9398 0x78f345f 0x78f36d8 0x79a02cc 0x79a042b 0x79b3a15 0x7864b7c 0x799073e 0x79907ce 0x7990872 0x73eb37a 0x7409299 0x740c6cb 0x75440fc 0x9e08ac1 0x9d0cb62 0x9dc6e39 0x9e285fb 0x9f988cc 0x9f98f48 0x26ed527 0xa5ce02e /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-23T11:19:50.165 INFO:teuthology.orchestra.run.trial030.stderr:Reactor stalled for 65 ms on shard 1, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0x9f10dfc 0x9d045c3 0x9d40a13 0x9d43fc6 0x9d44234 0x9d4437c 0x9d4483b /lib64/libc.so.6+0x3fc2f /lib64/libasan.so.8+0xc65bc /lib64/libasan.so.8+0x15197 0x2a5304b 0x7d7e215 0x7d9da0c 0x7e39ed3 0x7dbe035 0x78750f3 0x7875de8 0x79b38b2 0x7833a0a 0x78a9398 0x78f345f 0x78f36d8 0x79a02cc 0x79a042b 0x79b3a15 0x7864b7c 0x799073e 0x79907ce 0x7990872 0x73eb37a 0x7409299 0x740c6cb 0x75440fc 0x9e08ac1 0x9d0cb62 0x9dc6e39 0x9e285fb 0x9f988cc 0x9f98f48 0x26ed527 0xa5ce02e /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-23T11:19:52.302 INFO:teuthology.orchestra.run.trial030.stderr:----------------------------------------------------- 2026-02-23T11:19:52.303 INFO:teuthology.orchestra.run.trial030.stderr:Suppressions used: 2026-02-23T11:19:52.303 INFO:teuthology.orchestra.run.trial030.stderr: count bytes template 2026-02-23T11:19:52.303 INFO:teuthology.orchestra.run.trial030.stderr: 1 8 MallocExtension::Register 2026-02-23T11:19:52.303 INFO:teuthology.orchestra.run.trial030.stderr:----------------------------------------------------- 2026-02-23T11:19:52.303 INFO:teuthology.orchestra.run.trial030.stderr: 2026-02-23T11:19:52.337 DEBUG:teuthology.orchestra.run.trial030:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2026-02-23T11:19:52.363 INFO:tasks.ceph:Reading keys from all nodes... 2026-02-23T11:19:52.363 DEBUG:teuthology.orchestra.run.trial030:> set -ex 2026-02-23T11:19:52.363 DEBUG:teuthology.orchestra.run.trial030:> sudo dd if=/var/lib/ceph/mgr/ceph-x/keyring of=/dev/stdout 2026-02-23T11:19:52.428 DEBUG:teuthology.orchestra.run.trial030:> set -ex 2026-02-23T11:19:52.428 DEBUG:teuthology.orchestra.run.trial030:> sudo dd if=/var/lib/ceph/osd/ceph-0/keyring of=/dev/stdout 2026-02-23T11:19:52.494 DEBUG:teuthology.orchestra.run.trial030:> set -ex 2026-02-23T11:19:52.494 DEBUG:teuthology.orchestra.run.trial030:> sudo dd if=/var/lib/ceph/osd/ceph-1/keyring of=/dev/stdout 2026-02-23T11:19:52.558 DEBUG:teuthology.orchestra.run.trial030:> set -ex 2026-02-23T11:19:52.558 DEBUG:teuthology.orchestra.run.trial030:> sudo dd if=/var/lib/ceph/osd/ceph-2/keyring of=/dev/stdout 2026-02-23T11:19:52.623 DEBUG:teuthology.orchestra.run.trial030:> set -ex 2026-02-23T11:19:52.623 DEBUG:teuthology.orchestra.run.trial030:> dd if=/etc/ceph/ceph.client.0.keyring of=/dev/stdout 2026-02-23T11:19:52.679 INFO:tasks.ceph:Adding keys to all mons... 2026-02-23T11:19:52.679 DEBUG:teuthology.orchestra.run.trial030:> sudo tee -a /etc/ceph/ceph.keyring 2026-02-23T11:19:52.741 INFO:teuthology.orchestra.run.trial030.stdout:[mgr.x] 2026-02-23T11:19:52.742 INFO:teuthology.orchestra.run.trial030.stdout: key = AQDGN5xpd6uNGxAAATfijZP1LUTG/o7nvz9mEQ== 2026-02-23T11:19:52.742 INFO:teuthology.orchestra.run.trial030.stdout:[osd.0] 2026-02-23T11:19:52.742 INFO:teuthology.orchestra.run.trial030.stdout: key = AQDLN5xpsbgRHRAAV2iVjdxItUjLB23xo+NgZQ== 2026-02-23T11:19:52.742 INFO:teuthology.orchestra.run.trial030.stdout:[osd.1] 2026-02-23T11:19:52.742 INFO:teuthology.orchestra.run.trial030.stdout: key = AQDQN5xptNADJRAAihJK4wM+uHLYdnlvhm8Ejg== 2026-02-23T11:19:52.742 INFO:teuthology.orchestra.run.trial030.stdout:[osd.2] 2026-02-23T11:19:52.742 INFO:teuthology.orchestra.run.trial030.stdout: key = AQDVN5xp3/xsKxAAKePS2KCPOYts0CvCYBK9TQ== 2026-02-23T11:19:52.742 INFO:teuthology.orchestra.run.trial030.stdout:[client.0] 2026-02-23T11:19:52.742 INFO:teuthology.orchestra.run.trial030.stdout: key = AQDGN5xpC+M5HhAArBjHb+bNs3Cqv8+4DKsGwQ== 2026-02-23T11:19:52.744 DEBUG:teuthology.orchestra.run.trial030:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2026-02-23T11:19:52.832 DEBUG:teuthology.orchestra.run.trial030:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2026-02-23T11:19:52.880 DEBUG:teuthology.orchestra.run.trial030:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2026-02-23T11:19:52.927 DEBUG:teuthology.orchestra.run.trial030:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2026-02-23T11:19:52.976 DEBUG:teuthology.orchestra.run.trial030:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2026-02-23T11:19:53.025 INFO:tasks.ceph:Running mkfs on mon nodes... 2026-02-23T11:19:53.025 DEBUG:teuthology.orchestra.run.trial030:> sudo mkdir -p /var/lib/ceph/mon/ceph-a 2026-02-23T11:19:53.051 DEBUG:teuthology.orchestra.run.trial030:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i a --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2026-02-23T11:19:53.136 INFO:teuthology.orchestra.run.trial030.stderr:2026-02-23T11:19:53.134+0000 7f4ad7238c00 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T11:19:53.136 INFO:teuthology.orchestra.run.trial030.stderr:2026-02-23T11:19:53.134+0000 7f4ad7238c00 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T11:19:53.148 DEBUG:teuthology.orchestra.run.trial030:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-a 2026-02-23T11:19:53.174 DEBUG:teuthology.orchestra.run.trial030:> rm -- /home/ubuntu/cephtest/ceph.monmap 2026-02-23T11:19:53.229 INFO:tasks.ceph:Starting mon daemons in cluster ceph... 2026-02-23T11:19:53.230 INFO:tasks.ceph.mon.a:Restarting daemon 2026-02-23T11:19:53.230 DEBUG:teuthology.orchestra.run.trial030:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2026-02-23T11:19:53.271 INFO:tasks.ceph.mon.a:Started 2026-02-23T11:19:53.272 INFO:tasks.ceph:Starting mgr daemons in cluster ceph... 2026-02-23T11:19:53.272 INFO:tasks.ceph.mgr.x:Restarting daemon 2026-02-23T11:19:53.272 DEBUG:teuthology.orchestra.run.trial030:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i x 2026-02-23T11:19:53.275 INFO:tasks.ceph.mgr.x:Started 2026-02-23T11:19:53.275 DEBUG:tasks.ceph:set 0 configs 2026-02-23T11:19:53.275 DEBUG:teuthology.orchestra.run.trial030:> sudo ceph --cluster ceph config dump 2026-02-23T11:19:53.337 INFO:tasks.ceph.mon.a.trial030.stderr:2026-02-23T11:19:53.335+0000 7fbfbca96c00 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T11:19:53.337 INFO:tasks.ceph.mon.a.trial030.stderr:2026-02-23T11:19:53.335+0000 7fbfbca96c00 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T11:19:53.337 INFO:tasks.ceph.mon.a.trial030.stderr:2026-02-23T11:19:53.335+0000 7fbfbca96c00 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T11:19:53.339 INFO:tasks.ceph.mgr.x.trial030.stderr:2026-02-23T11:19:53.338+0000 7f5a42a85fc0 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T11:19:53.350 INFO:teuthology.orchestra.run.trial030.stderr:2026-02-23T11:19:53.347+0000 7f0d1730b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T11:19:53.354 INFO:teuthology.orchestra.run.trial030.stderr:2026-02-23T11:19:53.353+0000 7f0d1730b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T11:19:53.386 INFO:teuthology.orchestra.run.trial030.stdout:WHO MASK LEVEL OPTION VALUE RO 2026-02-23T11:19:53.394 INFO:tasks.ceph:Setting crush tunables to default 2026-02-23T11:19:53.395 DEBUG:teuthology.orchestra.run.trial030:> sudo ceph --cluster ceph osd crush tunables default 2026-02-23T11:19:53.469 INFO:teuthology.orchestra.run.trial030.stderr:2026-02-23T11:19:53.464+0000 7f935356f640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T11:19:53.472 INFO:teuthology.orchestra.run.trial030.stderr:2026-02-23T11:19:53.471+0000 7f935356f640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T11:19:53.507 INFO:teuthology.orchestra.run.trial030.stderr:adjusted tunables profile to default 2026-02-23T11:19:53.514 INFO:tasks.ceph:check_enable_crimson: True 2026-02-23T11:19:53.514 INFO:tasks.ceph:check_enable_crimson: setting set-allow-crimson 2026-02-23T11:19:53.515 DEBUG:teuthology.orchestra.run.trial030:> sudo ceph --cluster ceph osd set-allow-crimson --yes-i-really-mean-it 2026-02-23T11:19:53.544 INFO:tasks.ceph.mgr.x.trial030.stderr:2026-02-23T11:19:53.542+0000 7f5a42a85fc0 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T11:19:53.544 INFO:tasks.ceph.mgr.x.trial030.stderr:2026-02-23T11:19:53.543+0000 7f5a42a85fc0 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T11:19:53.587 INFO:teuthology.orchestra.run.trial030.stderr:2026-02-23T11:19:53.583+0000 7f710a41a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T11:19:53.589 INFO:teuthology.orchestra.run.trial030.stderr:2026-02-23T11:19:53.588+0000 7f710a41a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T11:19:53.629 INFO:tasks.ceph.mon.a.trial030.stderr:2026-02-23T11:19:53.628+0000 7fbfb41f1640 -1 WARNING: experimental feature 'crimson' is enabled 2026-02-23T11:19:53.629 INFO:tasks.ceph.mon.a.trial030.stderr:Please be aware that this feature is experimental, untested, 2026-02-23T11:19:53.629 INFO:tasks.ceph.mon.a.trial030.stderr:unsupported, and may result in data corruption, data loss, 2026-02-23T11:19:53.629 INFO:tasks.ceph.mon.a.trial030.stderr:and/or irreparable damage to your cluster. Do not use 2026-02-23T11:19:53.629 INFO:tasks.ceph.mon.a.trial030.stderr:feature with important data. 2026-02-23T11:19:53.629 INFO:tasks.ceph.mon.a.trial030.stderr: 2026-02-23T11:19:54.382 INFO:tasks.ceph.mgr.x.trial030.stderr:/usr/lib64/python3.9/site-packages/scipy/__init__.py:73: UserWarning: NumPy was imported from a Python sub-interpreter but NumPy does not properly support sub-interpreters. This will likely work for most users but might cause hard to track down issues or subtle bugs. A common user of the rare sub-interpreter feature is wsgi which also allows single-interpreter mode. 2026-02-23T11:19:54.382 INFO:tasks.ceph.mgr.x.trial030.stderr:Improvements in the case of bugs are welcome, but is not on the NumPy roadmap, and full support may require significant effort to achieve. 2026-02-23T11:19:54.383 INFO:tasks.ceph.mgr.x.trial030.stderr: from numpy import show_config as show_numpy_config 2026-02-23T11:19:54.518 INFO:tasks.ceph:Starting osd daemons in cluster ceph... 2026-02-23T11:19:54.518 DEBUG:teuthology.orchestra.run.trial030:> set -ex 2026-02-23T11:19:54.518 DEBUG:teuthology.orchestra.run.trial030:> sudo dd if=/var/lib/ceph/osd/ceph-0/fsid of=/dev/stdout 2026-02-23T11:19:54.584 DEBUG:teuthology.orchestra.run.trial030:> set -ex 2026-02-23T11:19:54.584 DEBUG:teuthology.orchestra.run.trial030:> sudo dd if=/var/lib/ceph/osd/ceph-1/fsid of=/dev/stdout 2026-02-23T11:19:54.649 DEBUG:teuthology.orchestra.run.trial030:> set -ex 2026-02-23T11:19:54.649 DEBUG:teuthology.orchestra.run.trial030:> sudo dd if=/var/lib/ceph/osd/ceph-2/fsid of=/dev/stdout 2026-02-23T11:19:54.713 DEBUG:teuthology.orchestra.run.trial030:> sudo ceph --cluster ceph osd new 9b3d3dff-502d-46ad-bb23-c485ba897402 0 2026-02-23T11:19:54.825 INFO:teuthology.orchestra.run.trial030.stderr:2026-02-23T11:19:54.821+0000 7fe33bdec640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T11:19:54.828 INFO:teuthology.orchestra.run.trial030.stderr:2026-02-23T11:19:54.827+0000 7fe33bdec640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T11:19:54.865 INFO:teuthology.orchestra.run.trial030.stdout:0 2026-02-23T11:19:54.873 DEBUG:teuthology.orchestra.run.trial030:> sudo ceph --cluster ceph osd new 5c421ec3-7768-46b0-ace6-d1b2ffc3a840 1 2026-02-23T11:19:54.945 INFO:teuthology.orchestra.run.trial030.stderr:2026-02-23T11:19:54.941+0000 7f0173d5c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T11:19:54.947 INFO:teuthology.orchestra.run.trial030.stderr:2026-02-23T11:19:54.946+0000 7f0173d5c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T11:19:54.985 INFO:teuthology.orchestra.run.trial030.stdout:1 2026-02-23T11:19:54.993 DEBUG:teuthology.orchestra.run.trial030:> sudo ceph --cluster ceph osd new cf71465d-2f79-4740-88c8-000366302513 2 2026-02-23T11:19:55.067 INFO:teuthology.orchestra.run.trial030.stderr:2026-02-23T11:19:55.063+0000 7fa2b9acb640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T11:19:55.069 INFO:teuthology.orchestra.run.trial030.stderr:2026-02-23T11:19:55.068+0000 7fa2b9acb640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T11:19:55.106 INFO:teuthology.orchestra.run.trial030.stdout:2 2026-02-23T11:19:55.114 DEBUG:teuthology.orchestra.run.trial030:> sudo install -b -o ubuntu -g ubuntu /dev/null /var/log/ceph/ceph-osd.0.log 2026-02-23T11:19:55.142 INFO:tasks.ceph.osd.0:Restarting daemon 2026-02-23T11:19:55.143 DEBUG:teuthology.orchestra.run.trial030:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 0 2>> /var/log/ceph/ceph-osd.0.log 2026-02-23T11:19:55.185 INFO:tasks.ceph.osd.0:Started 2026-02-23T11:19:55.185 DEBUG:teuthology.orchestra.run.trial030:> sudo install -b -o ubuntu -g ubuntu /dev/null /var/log/ceph/ceph-osd.1.log 2026-02-23T11:19:55.213 INFO:tasks.ceph.osd.1:Restarting daemon 2026-02-23T11:19:55.214 DEBUG:teuthology.orchestra.run.trial030:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 1 2>> /var/log/ceph/ceph-osd.1.log 2026-02-23T11:19:55.256 INFO:tasks.ceph.osd.1:Started 2026-02-23T11:19:55.256 DEBUG:teuthology.orchestra.run.trial030:> sudo install -b -o ubuntu -g ubuntu /dev/null /var/log/ceph/ceph-osd.2.log 2026-02-23T11:19:55.284 INFO:tasks.ceph.osd.2:Restarting daemon 2026-02-23T11:19:55.284 DEBUG:teuthology.orchestra.run.trial030:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 2 2>> /var/log/ceph/ceph-osd.2.log 2026-02-23T11:19:55.326 INFO:tasks.ceph.osd.2:Started 2026-02-23T11:19:55.327 DEBUG:teuthology.orchestra.run.trial030:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2026-02-23T11:19:55.404 INFO:teuthology.orchestra.run.trial030.stderr:2026-02-23T11:19:55.400+0000 7f9d6050b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T11:19:55.406 INFO:teuthology.orchestra.run.trial030.stderr:2026-02-23T11:19:55.405+0000 7f9d6050b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T11:19:55.440 INFO:teuthology.orchestra.run.trial030.stdout: 2026-02-23T11:19:55.440 INFO:teuthology.orchestra.run.trial030.stdout:{"epoch":6,"fsid":"4c6fca00-8e92-4876-9bd8-e5358ac35226","created":"2026-02-23T11:19:53.347742+0000","modified":"2026-02-23T11:19:55.105654+0000","last_up_change":"0.000000","last_in_change":"2026-02-23T11:19:55.105654+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[],"osds":[{"osd":0,"uuid":"9b3d3dff-502d-46ad-bb23-c485ba897402","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"5c421ec3-7768-46b0-ace6-d1b2ffc3a840","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"cf71465d-2f79-4740-88c8-000366302513","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T11:19:55.448 INFO:tasks.ceph.ceph_manager.ceph:[] 2026-02-23T11:19:55.449 INFO:tasks.ceph:Waiting for OSDs to come up 2026-02-23T11:19:55.750 DEBUG:teuthology.orchestra.run.trial030:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2026-02-23T11:19:55.818 INFO:teuthology.misc.health.trial030.stderr:2026-02-23T11:19:55.815+0000 7f5ac450b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T11:19:55.820 INFO:teuthology.misc.health.trial030.stderr:2026-02-23T11:19:55.819+0000 7f5ac450b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T11:19:55.853 INFO:teuthology.misc.health.trial030.stdout: 2026-02-23T11:19:55.853 INFO:teuthology.misc.health.trial030.stdout:{"epoch":6,"fsid":"4c6fca00-8e92-4876-9bd8-e5358ac35226","created":"2026-02-23T11:19:53.347742+0000","modified":"2026-02-23T11:19:55.105654+0000","last_up_change":"0.000000","last_in_change":"2026-02-23T11:19:55.105654+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[],"osds":[{"osd":0,"uuid":"9b3d3dff-502d-46ad-bb23-c485ba897402","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"5c421ec3-7768-46b0-ace6-d1b2ffc3a840","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"cf71465d-2f79-4740-88c8-000366302513","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T11:19:55.859 DEBUG:teuthology.misc:0 of 3 OSDs are up 2026-02-23T11:19:56.631 INFO:tasks.ceph.mgr.x.trial030.stderr:2026-02-23T11:19:56.630+0000 7f595fee7640 -1 mgr.server handle_report got status from non-daemon mon.a 2026-02-23T11:20:02.162 DEBUG:teuthology.orchestra.run.trial030:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2026-02-23T11:20:02.255 INFO:teuthology.misc.health.trial030.stderr:2026-02-23T11:20:02.251+0000 7fe5883e4640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T11:20:02.259 INFO:teuthology.misc.health.trial030.stderr:2026-02-23T11:20:02.257+0000 7fe5883e4640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T11:20:02.391 INFO:teuthology.misc.health.trial030.stdout: 2026-02-23T11:20:02.392 INFO:teuthology.misc.health.trial030.stdout:{"epoch":7,"fsid":"4c6fca00-8e92-4876-9bd8-e5358ac35226","created":"2026-02-23T11:19:53.347742+0000","modified":"2026-02-23T11:20:01.982702+0000","last_up_change":"0.000000","last_in_change":"2026-02-23T11:19:55.105654+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":3,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[],"osds":[{"osd":0,"uuid":"9b3d3dff-502d-46ad-bb23-c485ba897402","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"5c421ec3-7768-46b0-ace6-d1b2ffc3a840","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"cf71465d-2f79-4740-88c8-000366302513","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T11:20:02.407 DEBUG:teuthology.misc:0 of 3 OSDs are up 2026-02-23T11:20:08.709 DEBUG:teuthology.orchestra.run.trial030:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2026-02-23T11:20:08.781 INFO:teuthology.misc.health.trial030.stderr:2026-02-23T11:20:08.777+0000 7f135d50b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T11:20:08.783 INFO:teuthology.misc.health.trial030.stderr:2026-02-23T11:20:08.782+0000 7f135d50b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T11:20:08.896 INFO:teuthology.misc.health.trial030.stdout: 2026-02-23T11:20:08.896 INFO:teuthology.misc.health.trial030.stdout:{"epoch":13,"fsid":"4c6fca00-8e92-4876-9bd8-e5358ac35226","created":"2026-02-23T11:19:53.347742+0000","modified":"2026-02-23T11:20:08.007140+0000","last_up_change":"2026-02-23T11:20:04.995539+0000","last_in_change":"2026-02-23T11:19:55.105654+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T11:20:05.642161+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"13","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.9900000095367432,"score_stable":2.9900000095367432,"optimal_score":0.67000001668930054,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"9b3d3dff-502d-46ad-bb23-c485ba897402","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":10,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.30:6806","nonce":298998803}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.30:6807","nonce":298998803}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.30:6813","nonce":298998803}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.30:6812","nonce":298998803}]},"public_addr":"10.20.193.30:6806/298998803","cluster_addr":"10.20.193.30:6807/298998803","heartbeat_back_addr":"10.20.193.30:6813/298998803","heartbeat_front_addr":"10.20.193.30:6812/298998803","state":["exists","up"]},{"osd":1,"uuid":"5c421ec3-7768-46b0-ace6-d1b2ffc3a840","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":11,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.30:6804","nonce":947989549}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.30:6805","nonce":947989549}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.30:6811","nonce":947989549}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.30:6809","nonce":947989549}]},"public_addr":"10.20.193.30:6804/947989549","cluster_addr":"10.20.193.30:6805/947989549","heartbeat_back_addr":"10.20.193.30:6811/947989549","heartbeat_front_addr":"10.20.193.30:6809/947989549","state":["exists","up"]},{"osd":2,"uuid":"cf71465d-2f79-4740-88c8-000366302513","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.30:6803","nonce":1688863716}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.30:6802","nonce":1688863716}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.30:6808","nonce":1688863716}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.30:6810","nonce":1688863716}]},"public_addr":"10.20.193.30:6803/1688863716","cluster_addr":"10.20.193.30:6802/1688863716","heartbeat_back_addr":"10.20.193.30:6808/1688863716","heartbeat_front_addr":"10.20.193.30:6810/1688863716","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T11:20:08.904 DEBUG:teuthology.misc:3 of 3 OSDs are up 2026-02-23T11:20:08.905 INFO:tasks.ceph:Creating RBD pool 2026-02-23T11:20:08.905 DEBUG:teuthology.orchestra.run.trial030:> sudo ceph --cluster ceph osd pool create rbd 8 2026-02-23T11:20:08.981 INFO:teuthology.orchestra.run.trial030.stderr:2026-02-23T11:20:08.976+0000 7f6af6d5c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T11:20:08.983 INFO:teuthology.orchestra.run.trial030.stderr:2026-02-23T11:20:08.982+0000 7f6af6d5c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T11:20:10.018 INFO:teuthology.orchestra.run.trial030.stderr:pool 'rbd' created 2026-02-23T11:20:10.031 DEBUG:teuthology.orchestra.run.trial030:> rbd --cluster ceph pool init rbd 2026-02-23T11:20:10.074 INFO:teuthology.orchestra.run.trial030.stderr:2026-02-23T11:20:10.072+0000 7f60664df240 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T11:20:10.074 INFO:teuthology.orchestra.run.trial030.stderr:2026-02-23T11:20:10.073+0000 7f60664df240 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T11:20:10.077 INFO:teuthology.orchestra.run.trial030.stderr:2026-02-23T11:20:10.076+0000 7f60664df240 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T11:20:13.053 INFO:tasks.ceph:Starting mds daemons in cluster ceph... 2026-02-23T11:20:13.055 DEBUG:teuthology.orchestra.run.trial030:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config log 1 --format=json 2026-02-23T11:20:13.056 INFO:tasks.daemonwatchdog.daemon_watchdog:watchdog starting 2026-02-23T11:20:13.171 INFO:teuthology.orchestra.run.trial030.stderr:2026-02-23T11:20:13.167+0000 7f7317d9c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T11:20:13.173 INFO:teuthology.orchestra.run.trial030.stderr:2026-02-23T11:20:13.172+0000 7f7317d9c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T11:20:13.290 INFO:teuthology.orchestra.run.trial030.stdout: 2026-02-23T11:20:13.300 INFO:teuthology.orchestra.run.trial030.stdout:[{"version":1,"timestamp":"0.000000","name":"","changes":[]}] 2026-02-23T11:20:13.301 INFO:tasks.ceph_manager:config epoch is 1 2026-02-23T11:20:13.301 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2026-02-23T11:20:13.301 INFO:tasks.ceph.ceph_manager.ceph:waiting for mgr available 2026-02-23T11:20:13.301 DEBUG:teuthology.orchestra.run.trial030:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr dump --format=json 2026-02-23T11:20:13.378 INFO:teuthology.orchestra.run.trial030.stderr:2026-02-23T11:20:13.375+0000 7f688c8cb640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T11:20:13.380 INFO:teuthology.orchestra.run.trial030.stderr:2026-02-23T11:20:13.379+0000 7f688c8cb640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T11:20:13.523 INFO:teuthology.orchestra.run.trial030.stdout: 2026-02-23T11:20:13.531 INFO:teuthology.orchestra.run.trial030.stdout:{"epoch":5,"flags":0,"active_gid":4103,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.30:6800","nonce":1546595286},{"type":"v1","addr":"10.20.193.30:6801","nonce":1546595286}]},"active_addr":"10.20.193.30:6801/1546595286","active_change":"2026-02-23T11:19:55.625882+0000","active_mgr_features":4544132024016699391,"available":true,"standbys":[],"modules":["iostat","nfs"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to, use commas to separate multiple","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"update_pg_upmap_activity":{"name":"update_pg_upmap_activity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Updates pg_upmap activity stats to be used in `balancer status detail`","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often the agent on each host will gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"The first TCP port the agent will try to bind to. Up to 1000 subsequent ports will be attempted if this port cannot be bound","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this option can facilitate debugging daemons that encounter runtime problems.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"How frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"Fraction of total system memory to divide among autotuned daemons. Converged systems hosting both Ceph and compute should set a lower value; systems with more than 10 GiB of memory per OSD may benefit from a slightly larger value so that the abundance of RAM can be fully utilized","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command persistent logging","long_desc":"","tags":[],"see_also":[]},"certificate_automated_rotation_enabled":{"name":"certificate_automated_rotation_enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"This flag controls whether cephadm automatically rotates certificates upon expiration.","long_desc":"","tags":[],"see_also":[]},"certificate_check_debug_mode":{"name":"certificate_check_debug_mode","type":"bool","level":"dev","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"FOR TESTING ONLY: This flag forces the certificate check instead of waiting for certificate_check_period.","long_desc":"","tags":[],"see_also":[]},"certificate_check_period":{"name":"certificate_check_period","type":"int","level":"advanced","flags":0,"default_value":"1","min":"0","max":"30","enum_allowed":[],"desc":"Specifies how often (in days) the certificate should be checked for validity.","long_desc":"","tags":[],"see_also":[]},"certificate_duration_days":{"name":"certificate_duration_days","type":"int","level":"advanced","flags":0,"default_value":"1095","min":"90","max":"3650","enum_allowed":[],"desc":"Specifies the duration of self-signed certificates generated and signed by the cephadm root CA","long_desc":"","tags":[],"see_also":[]},"certificate_renewal_threshold_days":{"name":"certificate_renewal_threshold_days","type":"int","level":"advanced","flags":0,"default_value":"30","min":"10","max":"90","enum_allowed":[],"desc":"Specifies the lead time in days to initiate certificate renewal before expiration.","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently Podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.28.1","min":"","max":"","enum_allowed":[],"desc":"Alertmanager container image","long_desc":"","tags":[],"see_also":[]},"container_image_alloy":{"name":"container_image_alloy","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/alloy:latest","min":"","max":"","enum_allowed":[],"desc":"Alloy container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"Elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:12.3.1","min":"","max":"","enum_allowed":[],"desc":"Grafana container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"Haproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_nginx":{"name":"container_image_nginx","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nginx:sclorg-nginx-126","min":"","max":"","enum_allowed":[],"desc":"Nginx container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.9.1","min":"","max":"","enum_allowed":[],"desc":"Node exporter container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.6","min":"","max":"","enum_allowed":[],"desc":"Nvmeof container image","long_desc":"","tags":[],"see_also":[]},"container_image_oauth2_proxy":{"name":"container_image_oauth2_proxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/oauth2-proxy/oauth2-proxy:v7.6.0","min":"","max":"","enum_allowed":[],"desc":"Oauth2 proxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v3.6.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba":{"name":"container_image_samba","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-server:devbuilds-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba_metrics":{"name":"container_image_samba_metrics","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-metrics:devbuilds-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba metrics container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"Snmp gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"Seconds for which to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"quay.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"Seconds for which to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"Seconds for which to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"Location of dashboards to include in Grafana","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"How frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether the hardware monitoring daemon be deployed on every host?","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at the debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"Max number of daemons per service per host: guard against mistakes and runaways","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"Max number of OSDs that will be drained simultaneously when OSDs are removed","long_desc":"","tags":[],"see_also":[]},"max_parallel_osd_upgrades":{"name":"max_parallel_osd_upgrades","type":"int","level":"advanced","flags":0,"default_value":"16","min":"","max":"","enum_allowed":[],"desc":"Maximum number of OSD daemons upgraded in parallel.","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"Mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default IP address for RedFish API (OOB management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"Location of alerts to include in Prometheus Alertmanager","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry URL for login purposes. This is not the default registry.","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"Cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Customized SSH config file to distribute to managed hosts","long_desc":"","tags":[],"see_also":[]},"ssh_keepalive_count_max":{"name":"ssh_keepalive_count_max","type":"int","level":"advanced","flags":0,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"How many sequential SSH connection liveness check failures before the host is marked offline","long_desc":"","tags":[],"see_also":[]},"ssh_keepalive_interval":{"name":"ssh_keepalive_interval","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"How often SSH connections are checked for liveness","long_desc":"","tags":[],"see_also":[]},"stray_daemon_check_interval":{"name":"stray_daemon_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"How frequently cephadm should check for the presence of stray daemons","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use the cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest to ensure that all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MANAGED_BY_CLUSTERS":{"name":"MANAGED_BY_CLUSTERS","type":"str","level":"advanced","flags":0,"default_value":"[]","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MULTICLUSTER_CONFIG":{"name":"MULTICLUSTER_CONFIG","type":"str","level":"advanced","flags":0,"default_value":"{}","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROM_ALERT_CREDENTIAL_CACHE_TTL":{"name":"PROM_ALERT_CREDENTIAL_CACHE_TTL","type":"int","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_HOSTNAME_PER_DAEMON":{"name":"RGW_HOSTNAME_PER_DAEMON","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crypto_caller":{"name":"crypto_caller","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sso_oauth2":{"name":"sso_oauth2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"healthcheck_history_max_entries":{"name":"healthcheck_history_max_entries","type":"int","level":"advanced","flags":1,"default_value":"1000","min":"","max":"","enum_allowed":[],"desc":"Maximum number of health check history entries to keep","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"secondary_zone_period_retry_limit":{"name":"secondary_zone_period_retry_limit","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"RGW module period update retry limit for secondary site","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"prometheus_tls_secret_name":{"name":"prometheus_tls_secret_name","type":"str","level":"advanced","flags":0,"default_value":"rook-ceph-prometheus-server-tls","min":"","max":"","enum_allowed":[],"desc":"name of tls secret in k8s for prometheus","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"smb","can_run":true,"error_string":"","module_options":{"internal_store_backend":{"name":"internal_store_backend","type":"str","level":"dev","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"set internal store backend. for develoment and testing only","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"update_orchestration":{"name":"update_orchestration","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically update orchestration when smb resources are changed","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"pause_cloning":{"name":"pause_cloning","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Pause asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"pause_purging":{"name":"pause_purging","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Pause asynchronous subvolume purge threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"squid":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"tentacle":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"force_disabled_modules":{},"last_failure_osd_epoch":0,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"10.20.193.30:0","nonce":3969599300}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"10.20.193.30:0","nonce":3930935151}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"10.20.193.30:0","nonce":3989367179}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"10.20.193.30:0","nonce":664050676}]}]} 2026-02-23T11:20:13.535 INFO:tasks.ceph.ceph_manager.ceph:mgr available! 2026-02-23T11:20:13.535 INFO:tasks.ceph.ceph_manager.ceph:waiting for all up 2026-02-23T11:20:13.535 DEBUG:teuthology.orchestra.run.trial030:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2026-02-23T11:20:13.611 INFO:teuthology.orchestra.run.trial030.stderr:2026-02-23T11:20:13.607+0000 7f8cc78cb640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T11:20:13.613 INFO:teuthology.orchestra.run.trial030.stderr:2026-02-23T11:20:13.612+0000 7f8cc78cb640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T11:20:13.723 INFO:teuthology.orchestra.run.trial030.stdout: 2026-02-23T11:20:13.723 INFO:teuthology.orchestra.run.trial030.stdout:{"epoch":17,"fsid":"4c6fca00-8e92-4876-9bd8-e5358ac35226","created":"2026-02-23T11:19:53.347742+0000","modified":"2026-02-23T11:20:13.024854+0000","last_up_change":"2026-02-23T11:20:04.995539+0000","last_in_change":"2026-02-23T11:19:55.105654+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T11:20:05.642161+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"13","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.9900000095367432,"score_stable":2.9900000095367432,"optimal_score":0.67000001668930054,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2026-02-23T11:20:09.105686+0000","flags":270337,"flags_names":"hashpspool,selfmanaged_snaps,crimson","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":2,"snap_epoch":17,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"9b3d3dff-502d-46ad-bb23-c485ba897402","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":10,"up_thru":14,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.30:6806","nonce":298998803}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.30:6807","nonce":298998803}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.30:6813","nonce":298998803}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.30:6812","nonce":298998803}]},"public_addr":"10.20.193.30:6806/298998803","cluster_addr":"10.20.193.30:6807/298998803","heartbeat_back_addr":"10.20.193.30:6813/298998803","heartbeat_front_addr":"10.20.193.30:6812/298998803","state":["exists","up"]},{"osd":1,"uuid":"5c421ec3-7768-46b0-ace6-d1b2ffc3a840","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":14,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.30:6804","nonce":947989549}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.30:6805","nonce":947989549}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.30:6811","nonce":947989549}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.30:6809","nonce":947989549}]},"public_addr":"10.20.193.30:6804/947989549","cluster_addr":"10.20.193.30:6805/947989549","heartbeat_back_addr":"10.20.193.30:6811/947989549","heartbeat_front_addr":"10.20.193.30:6809/947989549","state":["exists","up"]},{"osd":2,"uuid":"cf71465d-2f79-4740-88c8-000366302513","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":14,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.30:6803","nonce":1688863716}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.30:6802","nonce":1688863716}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.30:6808","nonce":1688863716}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.30:6810","nonce":1688863716}]},"public_addr":"10.20.193.30:6803/1688863716","cluster_addr":"10.20.193.30:6802/1688863716","heartbeat_back_addr":"10.20.193.30:6808/1688863716","heartbeat_front_addr":"10.20.193.30:6810/1688863716","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":2,"snaps":[{"begin":2,"length":1}]}],"new_removed_snaps":[{"pool":2,"snaps":[{"begin":2,"length":1}]}],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T11:20:13.733 INFO:tasks.ceph.ceph_manager.ceph:all up! 2026-02-23T11:20:13.733 DEBUG:teuthology.orchestra.run.trial030:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2026-02-23T11:20:13.809 INFO:teuthology.orchestra.run.trial030.stderr:2026-02-23T11:20:13.806+0000 7f27b108b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T11:20:13.811 INFO:teuthology.orchestra.run.trial030.stderr:2026-02-23T11:20:13.810+0000 7f27b108b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T11:20:13.923 INFO:teuthology.orchestra.run.trial030.stdout: 2026-02-23T11:20:13.924 INFO:teuthology.orchestra.run.trial030.stdout:{"epoch":17,"fsid":"4c6fca00-8e92-4876-9bd8-e5358ac35226","created":"2026-02-23T11:19:53.347742+0000","modified":"2026-02-23T11:20:13.024854+0000","last_up_change":"2026-02-23T11:20:04.995539+0000","last_in_change":"2026-02-23T11:19:55.105654+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T11:20:05.642161+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"13","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.9900000095367432,"score_stable":2.9900000095367432,"optimal_score":0.67000001668930054,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2026-02-23T11:20:09.105686+0000","flags":270337,"flags_names":"hashpspool,selfmanaged_snaps,crimson","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":2,"snap_epoch":17,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"9b3d3dff-502d-46ad-bb23-c485ba897402","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":10,"up_thru":14,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.30:6806","nonce":298998803}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.30:6807","nonce":298998803}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.30:6813","nonce":298998803}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.30:6812","nonce":298998803}]},"public_addr":"10.20.193.30:6806/298998803","cluster_addr":"10.20.193.30:6807/298998803","heartbeat_back_addr":"10.20.193.30:6813/298998803","heartbeat_front_addr":"10.20.193.30:6812/298998803","state":["exists","up"]},{"osd":1,"uuid":"5c421ec3-7768-46b0-ace6-d1b2ffc3a840","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":14,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.30:6804","nonce":947989549}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.30:6805","nonce":947989549}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.30:6811","nonce":947989549}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.30:6809","nonce":947989549}]},"public_addr":"10.20.193.30:6804/947989549","cluster_addr":"10.20.193.30:6805/947989549","heartbeat_back_addr":"10.20.193.30:6811/947989549","heartbeat_front_addr":"10.20.193.30:6809/947989549","state":["exists","up"]},{"osd":2,"uuid":"cf71465d-2f79-4740-88c8-000366302513","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":14,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.30:6803","nonce":1688863716}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.30:6802","nonce":1688863716}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.30:6808","nonce":1688863716}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.30:6810","nonce":1688863716}]},"public_addr":"10.20.193.30:6803/1688863716","cluster_addr":"10.20.193.30:6802/1688863716","heartbeat_back_addr":"10.20.193.30:6808/1688863716","heartbeat_front_addr":"10.20.193.30:6810/1688863716","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":2,"snaps":[{"begin":2,"length":1}]}],"new_removed_snaps":[{"pool":2,"snaps":[{"begin":2,"length":1}]}],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T11:20:13.934 DEBUG:teuthology.orchestra.run.trial030:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2026-02-23T11:20:13.935 DEBUG:teuthology.orchestra.run.trial030:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2026-02-23T11:20:13.935 DEBUG:teuthology.orchestra.run.trial030:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2026-02-23T11:20:14.017 INFO:teuthology.orchestra.run.trial030.stderr:2026-02-23T11:20:14.013+0000 7f141e21a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T11:20:14.017 INFO:teuthology.orchestra.run.trial030.stderr:2026-02-23T11:20:14.013+0000 7ff6bf5cc640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T11:20:14.018 INFO:teuthology.orchestra.run.trial030.stderr:2026-02-23T11:20:14.013+0000 7f1e1d35b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T11:20:14.019 INFO:teuthology.orchestra.run.trial030.stderr:2026-02-23T11:20:14.018+0000 7f141e21a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T11:20:14.019 INFO:teuthology.orchestra.run.trial030.stderr:2026-02-23T11:20:14.018+0000 7ff6bf5cc640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T11:20:14.020 INFO:teuthology.orchestra.run.trial030.stderr:2026-02-23T11:20:14.019+0000 7f1e1d35b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T11:20:14.037 INFO:teuthology.orchestra.run.trial030.stdout:42949672969 2026-02-23T11:20:14.037 INFO:teuthology.orchestra.run.trial030.stdout:38654705674 2026-02-23T11:20:14.038 DEBUG:teuthology.orchestra.run.trial030:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2026-02-23T11:20:14.039 DEBUG:teuthology.orchestra.run.trial030:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2026-02-23T11:20:14.040 INFO:teuthology.orchestra.run.trial030.stdout:38654705674 2026-02-23T11:20:14.040 DEBUG:teuthology.orchestra.run.trial030:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2026-02-23T11:20:14.160 INFO:teuthology.orchestra.run.trial030.stderr:2026-02-23T11:20:14.156+0000 7f3d98d6f640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T11:20:14.160 INFO:teuthology.orchestra.run.trial030.stderr:2026-02-23T11:20:14.155+0000 7f941030b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T11:20:14.161 INFO:teuthology.orchestra.run.trial030.stderr:2026-02-23T11:20:14.156+0000 7f25d82cb640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T11:20:14.162 INFO:teuthology.orchestra.run.trial030.stderr:2026-02-23T11:20:14.161+0000 7f3d98d6f640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T11:20:14.163 INFO:teuthology.orchestra.run.trial030.stderr:2026-02-23T11:20:14.161+0000 7f25d82cb640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T11:20:14.163 INFO:teuthology.orchestra.run.trial030.stderr:2026-02-23T11:20:14.161+0000 7f941030b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T11:20:14.274 INFO:teuthology.orchestra.run.trial030.stdout:38654705674 2026-02-23T11:20:14.277 INFO:teuthology.orchestra.run.trial030.stdout:38654705674 2026-02-23T11:20:14.280 INFO:teuthology.orchestra.run.trial030.stdout:42949672964 2026-02-23T11:20:14.285 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654705674 got 38654705674 for osd.1 2026-02-23T11:20:14.286 DEBUG:teuthology.parallel:result is None 2026-02-23T11:20:14.288 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654705674 got 38654705674 for osd.2 2026-02-23T11:20:14.289 DEBUG:teuthology.parallel:result is None 2026-02-23T11:20:14.291 INFO:tasks.ceph.ceph_manager.ceph:need seq 42949672969 got 42949672964 for osd.0 2026-02-23T11:20:15.292 DEBUG:teuthology.orchestra.run.trial030:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2026-02-23T11:20:15.370 INFO:teuthology.orchestra.run.trial030.stderr:2026-02-23T11:20:15.366+0000 7fded299c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T11:20:15.372 INFO:teuthology.orchestra.run.trial030.stderr:2026-02-23T11:20:15.371+0000 7fded299c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T11:20:15.484 INFO:teuthology.orchestra.run.trial030.stdout:42949672964 2026-02-23T11:20:15.494 INFO:tasks.ceph.ceph_manager.ceph:need seq 42949672969 got 42949672964 for osd.0 2026-02-23T11:20:16.496 DEBUG:teuthology.orchestra.run.trial030:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2026-02-23T11:20:16.576 INFO:teuthology.orchestra.run.trial030.stderr:2026-02-23T11:20:16.572+0000 7eff7aacb640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T11:20:16.578 INFO:teuthology.orchestra.run.trial030.stderr:2026-02-23T11:20:16.577+0000 7eff7aacb640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T11:20:16.690 INFO:teuthology.orchestra.run.trial030.stdout:42949672969 2026-02-23T11:20:16.700 INFO:tasks.ceph.ceph_manager.ceph:need seq 42949672969 got 42949672969 for osd.0 2026-02-23T11:20:16.700 DEBUG:teuthology.parallel:result is None 2026-02-23T11:20:16.700 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2026-02-23T11:20:16.701 DEBUG:teuthology.orchestra.run.trial030:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-23T11:20:16.778 INFO:teuthology.orchestra.run.trial030.stderr:2026-02-23T11:20:16.775+0000 7f95e621a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T11:20:16.780 INFO:teuthology.orchestra.run.trial030.stderr:2026-02-23T11:20:16.779+0000 7f95e621a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T11:20:16.890 INFO:teuthology.orchestra.run.trial030.stdout: 2026-02-23T11:20:16.890 INFO:teuthology.orchestra.run.trial030.stderr:dumped all 2026-02-23T11:20:16.899 INFO:teuthology.orchestra.run.trial030.stdout:{"pg_ready":true,"pg_map":{"version":15,"stamp":"2026-02-23T11:20:15.629893+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":53,"num_read_kb":46,"num_write":81,"num_write_kb":1108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":54,"ondisk_log_size":54,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":3,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":156499968,"kb_used":2540,"kb_used_data":2540,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":156497428,"statfs":{"total":160255967232,"available":160253366272,"internally_reserved":0,"allocated":2600960,"data_stored":2600960,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"1.617454"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":15,"reported_epoch":17,"state":"active+clean","last_fresh":"2026-02-23T11:20:13.037193+0000","last_change":"2026-02-23T11:20:13.041495+0000","last_active":"2026-02-23T11:20:13.037193+0000","last_peered":"2026-02-23T11:20:13.037193+0000","last_clean":"2026-02-23T11:20:13.037193+0000","last_became_active":"2026-02-23T11:20:11.064755+0000","last_became_peered":"2026-02-23T11:20:11.064755+0000","last_unstale":"2026-02-23T11:20:13.037193+0000","last_undegraded":"2026-02-23T11:20:13.037193+0000","last_fullsized":"2026-02-23T11:20:13.037193+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-23T11:20:10.015542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-23T11:20:10.015542+0000","last_clean_scrub_stamp":"2026-02-23T11:20:10.015542+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":15,"reported_epoch":17,"state":"active+clean","last_fresh":"2026-02-23T11:20:13.039613+0000","last_change":"2026-02-23T11:20:13.043746+0000","last_active":"2026-02-23T11:20:13.039613+0000","last_peered":"2026-02-23T11:20:13.039613+0000","last_clean":"2026-02-23T11:20:13.039613+0000","last_became_active":"2026-02-23T11:20:11.063171+0000","last_became_peered":"2026-02-23T11:20:11.063171+0000","last_unstale":"2026-02-23T11:20:13.039613+0000","last_undegraded":"2026-02-23T11:20:13.039613+0000","last_fullsized":"2026-02-23T11:20:13.039613+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-23T11:20:10.015542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-23T11:20:10.015542+0000","last_clean_scrub_stamp":"2026-02-23T11:20:10.015542+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":15,"reported_epoch":17,"state":"active+clean","last_fresh":"2026-02-23T11:20:13.039096+0000","last_change":"2026-02-23T11:20:13.041206+0000","last_active":"2026-02-23T11:20:13.039096+0000","last_peered":"2026-02-23T11:20:13.039096+0000","last_clean":"2026-02-23T11:20:13.039096+0000","last_became_active":"2026-02-23T11:20:11.070677+0000","last_became_peered":"2026-02-23T11:20:11.070677+0000","last_unstale":"2026-02-23T11:20:13.039096+0000","last_undegraded":"2026-02-23T11:20:13.039096+0000","last_fullsized":"2026-02-23T11:20:13.039096+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-23T11:20:10.015542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-23T11:20:10.015542+0000","last_clean_scrub_stamp":"2026-02-23T11:20:10.015542+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":15,"reported_epoch":17,"state":"active+clean","last_fresh":"2026-02-23T11:20:13.039159+0000","last_change":"2026-02-23T11:20:13.043971+0000","last_active":"2026-02-23T11:20:13.039159+0000","last_peered":"2026-02-23T11:20:13.039159+0000","last_clean":"2026-02-23T11:20:13.039159+0000","last_became_active":"2026-02-23T11:20:11.074577+0000","last_became_peered":"2026-02-23T11:20:11.074577+0000","last_unstale":"2026-02-23T11:20:13.039159+0000","last_undegraded":"2026-02-23T11:20:13.039159+0000","last_fullsized":"2026-02-23T11:20:13.039159+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-23T11:20:10.015542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-23T11:20:10.015542+0000","last_clean_scrub_stamp":"2026-02-23T11:20:10.015542+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"16'1","reported_seq":17,"reported_epoch":17,"state":"active+clean","last_fresh":"2026-02-23T11:20:13.039104+0000","last_change":"2026-02-23T11:20:13.039104+0000","last_active":"2026-02-23T11:20:13.039104+0000","last_peered":"2026-02-23T11:20:13.039104+0000","last_clean":"2026-02-23T11:20:13.039104+0000","last_became_active":"2026-02-23T11:20:11.072774+0000","last_became_peered":"2026-02-23T11:20:11.072774+0000","last_unstale":"2026-02-23T11:20:13.039104+0000","last_undegraded":"2026-02-23T11:20:13.039104+0000","last_fullsized":"2026-02-23T11:20:13.039104+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-23T11:20:10.015542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-23T11:20:10.015542+0000","last_clean_scrub_stamp":"2026-02-23T11:20:10.015542+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":15,"reported_epoch":17,"state":"active+clean","last_fresh":"2026-02-23T11:20:13.037209+0000","last_change":"2026-02-23T11:20:13.039858+0000","last_active":"2026-02-23T11:20:13.037209+0000","last_peered":"2026-02-23T11:20:13.037209+0000","last_clean":"2026-02-23T11:20:13.037209+0000","last_became_active":"2026-02-23T11:20:11.083328+0000","last_became_peered":"2026-02-23T11:20:11.083328+0000","last_unstale":"2026-02-23T11:20:13.037209+0000","last_undegraded":"2026-02-23T11:20:13.037209+0000","last_fullsized":"2026-02-23T11:20:13.037209+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-23T11:20:10.015542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-23T11:20:10.015542+0000","last_clean_scrub_stamp":"2026-02-23T11:20:10.015542+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":15,"reported_epoch":17,"state":"active+clean","last_fresh":"2026-02-23T11:20:13.034691+0000","last_change":"2026-02-23T11:20:13.037781+0000","last_active":"2026-02-23T11:20:13.034691+0000","last_peered":"2026-02-23T11:20:13.034691+0000","last_clean":"2026-02-23T11:20:13.034691+0000","last_became_active":"2026-02-23T11:20:11.087641+0000","last_became_peered":"2026-02-23T11:20:11.087641+0000","last_unstale":"2026-02-23T11:20:13.034691+0000","last_undegraded":"2026-02-23T11:20:13.034691+0000","last_fullsized":"2026-02-23T11:20:13.034691+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-23T11:20:10.015542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-23T11:20:10.015542+0000","last_clean_scrub_stamp":"2026-02-23T11:20:10.015542+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"15'1","reported_seq":16,"reported_epoch":17,"state":"active+clean","last_fresh":"2026-02-23T11:20:13.038044+0000","last_change":"2026-02-23T11:20:13.040759+0000","last_active":"2026-02-23T11:20:13.038044+0000","last_peered":"2026-02-23T11:20:13.038044+0000","last_clean":"2026-02-23T11:20:13.038044+0000","last_became_active":"2026-02-23T11:20:11.060769+0000","last_became_peered":"2026-02-23T11:20:11.060769+0000","last_unstale":"2026-02-23T11:20:13.038044+0000","last_undegraded":"2026-02-23T11:20:13.038044+0000","last_fullsized":"2026-02-23T11:20:13.038044+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-23T11:20:10.015542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-23T11:20:10.015542+0000","last_clean_scrub_stamp":"2026-02-23T11:20:10.015542+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"12'52","reported_seq":90,"reported_epoch":17,"state":"active+clean","last_fresh":"2026-02-23T11:20:13.037511+0000","last_change":"2026-02-23T11:20:07.042670+0000","last_active":"2026-02-23T11:20:13.037511+0000","last_peered":"2026-02-23T11:20:13.037511+0000","last_clean":"2026-02-23T11:20:13.037511+0000","last_became_active":"2026-02-23T11:20:07.033110+0000","last_became_peered":"2026-02-23T11:20:07.033110+0000","last_unstale":"2026-02-23T11:20:13.037511+0000","last_undegraded":"2026-02-23T11:20:13.037511+0000","last_fullsized":"2026-02-23T11:20:13.037511+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-23T11:20:06.000692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-23T11:20:06.000692+0000","last_clean_scrub_stamp":"2026-02-23T11:20:06.000692+0000","objects_scrubbed":0,"log_size":52,"log_dups_size":0,"ondisk_log_size":52,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":53,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":160255967232,"available":160253366272,"internally_reserved":0,"allocated":2600960,"data_stored":2600960,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":53,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":53418655744,"available":53417480192,"internally_reserved":0,"allocated":1175552,"data_stored":1175552,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":52,"ondisk_log_size":52,"up":2,"acting":2,"num_store_stats":1}],"osd_stats":[{"osd":2,"up_from":9,"seq":38654705674,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52166656,"kb_used":316,"kb_used_data":316,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":52166340,"statfs":{"total":53418655744,"available":53418332160,"internally_reserved":0,"allocated":323584,"data_stored":323584,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":9,"seq":38654705674,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52166656,"kb_used":1148,"kb_used_data":1148,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":52165508,"statfs":{"total":53418655744,"available":53417480192,"internally_reserved":0,"allocated":1175552,"data_stored":1175552,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":10,"seq":42949672969,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52166656,"kb_used":1076,"kb_used_data":1076,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":52165580,"statfs":{"total":53418655744,"available":53417553920,"internally_reserved":0,"allocated":1101824,"data_stored":1101824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":1,"total":53418655744,"available":53417480192,"internally_reserved":0,"allocated":1175552,"data_stored":1175552,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":53418655744,"available":53417553920,"internally_reserved":0,"allocated":1101824,"data_stored":1101824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":53418655744,"available":53417480192,"internally_reserved":0,"allocated":1175552,"data_stored":1175552,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":53418655744,"available":53418332160,"internally_reserved":0,"allocated":323584,"data_stored":323584,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-23T11:20:16.900 DEBUG:teuthology.orchestra.run.trial030:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-23T11:20:16.976 INFO:teuthology.orchestra.run.trial030.stderr:2026-02-23T11:20:16.973+0000 7f12a7e35640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T11:20:16.979 INFO:teuthology.orchestra.run.trial030.stderr:2026-02-23T11:20:16.978+0000 7f12a7e35640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T11:20:17.093 INFO:teuthology.orchestra.run.trial030.stdout: 2026-02-23T11:20:17.094 INFO:teuthology.orchestra.run.trial030.stderr:dumped all 2026-02-23T11:20:17.103 INFO:teuthology.orchestra.run.trial030.stdout:{"pg_ready":true,"pg_map":{"version":15,"stamp":"2026-02-23T11:20:15.629893+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":53,"num_read_kb":46,"num_write":81,"num_write_kb":1108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":54,"ondisk_log_size":54,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":3,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":156499968,"kb_used":2540,"kb_used_data":2540,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":156497428,"statfs":{"total":160255967232,"available":160253366272,"internally_reserved":0,"allocated":2600960,"data_stored":2600960,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"1.617454"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":15,"reported_epoch":17,"state":"active+clean","last_fresh":"2026-02-23T11:20:13.037193+0000","last_change":"2026-02-23T11:20:13.041495+0000","last_active":"2026-02-23T11:20:13.037193+0000","last_peered":"2026-02-23T11:20:13.037193+0000","last_clean":"2026-02-23T11:20:13.037193+0000","last_became_active":"2026-02-23T11:20:11.064755+0000","last_became_peered":"2026-02-23T11:20:11.064755+0000","last_unstale":"2026-02-23T11:20:13.037193+0000","last_undegraded":"2026-02-23T11:20:13.037193+0000","last_fullsized":"2026-02-23T11:20:13.037193+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-23T11:20:10.015542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-23T11:20:10.015542+0000","last_clean_scrub_stamp":"2026-02-23T11:20:10.015542+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":15,"reported_epoch":17,"state":"active+clean","last_fresh":"2026-02-23T11:20:13.039613+0000","last_change":"2026-02-23T11:20:13.043746+0000","last_active":"2026-02-23T11:20:13.039613+0000","last_peered":"2026-02-23T11:20:13.039613+0000","last_clean":"2026-02-23T11:20:13.039613+0000","last_became_active":"2026-02-23T11:20:11.063171+0000","last_became_peered":"2026-02-23T11:20:11.063171+0000","last_unstale":"2026-02-23T11:20:13.039613+0000","last_undegraded":"2026-02-23T11:20:13.039613+0000","last_fullsized":"2026-02-23T11:20:13.039613+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-23T11:20:10.015542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-23T11:20:10.015542+0000","last_clean_scrub_stamp":"2026-02-23T11:20:10.015542+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":15,"reported_epoch":17,"state":"active+clean","last_fresh":"2026-02-23T11:20:13.039096+0000","last_change":"2026-02-23T11:20:13.041206+0000","last_active":"2026-02-23T11:20:13.039096+0000","last_peered":"2026-02-23T11:20:13.039096+0000","last_clean":"2026-02-23T11:20:13.039096+0000","last_became_active":"2026-02-23T11:20:11.070677+0000","last_became_peered":"2026-02-23T11:20:11.070677+0000","last_unstale":"2026-02-23T11:20:13.039096+0000","last_undegraded":"2026-02-23T11:20:13.039096+0000","last_fullsized":"2026-02-23T11:20:13.039096+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-23T11:20:10.015542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-23T11:20:10.015542+0000","last_clean_scrub_stamp":"2026-02-23T11:20:10.015542+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":15,"reported_epoch":17,"state":"active+clean","last_fresh":"2026-02-23T11:20:13.039159+0000","last_change":"2026-02-23T11:20:13.043971+0000","last_active":"2026-02-23T11:20:13.039159+0000","last_peered":"2026-02-23T11:20:13.039159+0000","last_clean":"2026-02-23T11:20:13.039159+0000","last_became_active":"2026-02-23T11:20:11.074577+0000","last_became_peered":"2026-02-23T11:20:11.074577+0000","last_unstale":"2026-02-23T11:20:13.039159+0000","last_undegraded":"2026-02-23T11:20:13.039159+0000","last_fullsized":"2026-02-23T11:20:13.039159+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-23T11:20:10.015542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-23T11:20:10.015542+0000","last_clean_scrub_stamp":"2026-02-23T11:20:10.015542+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"16'1","reported_seq":17,"reported_epoch":17,"state":"active+clean","last_fresh":"2026-02-23T11:20:13.039104+0000","last_change":"2026-02-23T11:20:13.039104+0000","last_active":"2026-02-23T11:20:13.039104+0000","last_peered":"2026-02-23T11:20:13.039104+0000","last_clean":"2026-02-23T11:20:13.039104+0000","last_became_active":"2026-02-23T11:20:11.072774+0000","last_became_peered":"2026-02-23T11:20:11.072774+0000","last_unstale":"2026-02-23T11:20:13.039104+0000","last_undegraded":"2026-02-23T11:20:13.039104+0000","last_fullsized":"2026-02-23T11:20:13.039104+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-23T11:20:10.015542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-23T11:20:10.015542+0000","last_clean_scrub_stamp":"2026-02-23T11:20:10.015542+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":15,"reported_epoch":17,"state":"active+clean","last_fresh":"2026-02-23T11:20:13.037209+0000","last_change":"2026-02-23T11:20:13.039858+0000","last_active":"2026-02-23T11:20:13.037209+0000","last_peered":"2026-02-23T11:20:13.037209+0000","last_clean":"2026-02-23T11:20:13.037209+0000","last_became_active":"2026-02-23T11:20:11.083328+0000","last_became_peered":"2026-02-23T11:20:11.083328+0000","last_unstale":"2026-02-23T11:20:13.037209+0000","last_undegraded":"2026-02-23T11:20:13.037209+0000","last_fullsized":"2026-02-23T11:20:13.037209+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-23T11:20:10.015542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-23T11:20:10.015542+0000","last_clean_scrub_stamp":"2026-02-23T11:20:10.015542+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":15,"reported_epoch":17,"state":"active+clean","last_fresh":"2026-02-23T11:20:13.034691+0000","last_change":"2026-02-23T11:20:13.037781+0000","last_active":"2026-02-23T11:20:13.034691+0000","last_peered":"2026-02-23T11:20:13.034691+0000","last_clean":"2026-02-23T11:20:13.034691+0000","last_became_active":"2026-02-23T11:20:11.087641+0000","last_became_peered":"2026-02-23T11:20:11.087641+0000","last_unstale":"2026-02-23T11:20:13.034691+0000","last_undegraded":"2026-02-23T11:20:13.034691+0000","last_fullsized":"2026-02-23T11:20:13.034691+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-23T11:20:10.015542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-23T11:20:10.015542+0000","last_clean_scrub_stamp":"2026-02-23T11:20:10.015542+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"15'1","reported_seq":16,"reported_epoch":17,"state":"active+clean","last_fresh":"2026-02-23T11:20:13.038044+0000","last_change":"2026-02-23T11:20:13.040759+0000","last_active":"2026-02-23T11:20:13.038044+0000","last_peered":"2026-02-23T11:20:13.038044+0000","last_clean":"2026-02-23T11:20:13.038044+0000","last_became_active":"2026-02-23T11:20:11.060769+0000","last_became_peered":"2026-02-23T11:20:11.060769+0000","last_unstale":"2026-02-23T11:20:13.038044+0000","last_undegraded":"2026-02-23T11:20:13.038044+0000","last_fullsized":"2026-02-23T11:20:13.038044+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-23T11:20:10.015542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-23T11:20:10.015542+0000","last_clean_scrub_stamp":"2026-02-23T11:20:10.015542+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"12'52","reported_seq":90,"reported_epoch":17,"state":"active+clean","last_fresh":"2026-02-23T11:20:13.037511+0000","last_change":"2026-02-23T11:20:07.042670+0000","last_active":"2026-02-23T11:20:13.037511+0000","last_peered":"2026-02-23T11:20:13.037511+0000","last_clean":"2026-02-23T11:20:13.037511+0000","last_became_active":"2026-02-23T11:20:07.033110+0000","last_became_peered":"2026-02-23T11:20:07.033110+0000","last_unstale":"2026-02-23T11:20:13.037511+0000","last_undegraded":"2026-02-23T11:20:13.037511+0000","last_fullsized":"2026-02-23T11:20:13.037511+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-23T11:20:06.000692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-23T11:20:06.000692+0000","last_clean_scrub_stamp":"2026-02-23T11:20:06.000692+0000","objects_scrubbed":0,"log_size":52,"log_dups_size":0,"ondisk_log_size":52,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":53,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":160255967232,"available":160253366272,"internally_reserved":0,"allocated":2600960,"data_stored":2600960,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":53,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":53418655744,"available":53417480192,"internally_reserved":0,"allocated":1175552,"data_stored":1175552,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":52,"ondisk_log_size":52,"up":2,"acting":2,"num_store_stats":1}],"osd_stats":[{"osd":2,"up_from":9,"seq":38654705674,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52166656,"kb_used":316,"kb_used_data":316,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":52166340,"statfs":{"total":53418655744,"available":53418332160,"internally_reserved":0,"allocated":323584,"data_stored":323584,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":9,"seq":38654705674,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52166656,"kb_used":1148,"kb_used_data":1148,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":52165508,"statfs":{"total":53418655744,"available":53417480192,"internally_reserved":0,"allocated":1175552,"data_stored":1175552,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":10,"seq":42949672969,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52166656,"kb_used":1076,"kb_used_data":1076,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":52165580,"statfs":{"total":53418655744,"available":53417553920,"internally_reserved":0,"allocated":1101824,"data_stored":1101824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":1,"total":53418655744,"available":53417480192,"internally_reserved":0,"allocated":1175552,"data_stored":1175552,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":53418655744,"available":53417553920,"internally_reserved":0,"allocated":1101824,"data_stored":1101824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":53418655744,"available":53417480192,"internally_reserved":0,"allocated":1175552,"data_stored":1175552,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":53418655744,"available":53418332160,"internally_reserved":0,"allocated":323584,"data_stored":323584,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-23T11:20:17.105 INFO:tasks.ceph.ceph_manager.ceph:clean! 2026-02-23T11:20:17.105 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2026-02-23T11:20:17.105 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy 2026-02-23T11:20:17.105 DEBUG:teuthology.orchestra.run.trial030:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2026-02-23T11:20:17.181 INFO:teuthology.orchestra.run.trial030.stderr:2026-02-23T11:20:17.178+0000 7ff510bfd640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T11:20:17.184 INFO:teuthology.orchestra.run.trial030.stderr:2026-02-23T11:20:17.182+0000 7ff510bfd640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T11:20:17.320 INFO:teuthology.orchestra.run.trial030.stdout: 2026-02-23T11:20:17.320 INFO:teuthology.orchestra.run.trial030.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2026-02-23T11:20:17.330 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy done 2026-02-23T11:20:17.330 INFO:teuthology.run_tasks:Running task workunit... 2026-02-23T11:20:17.360 INFO:tasks.workunit:Pulling workunits from ref e371433fcd1390a0b82cfb18effc422c1f5d8c1d 2026-02-23T11:20:17.360 INFO:tasks.workunit:Making a separate scratch dir for every client... 2026-02-23T11:20:17.361 DEBUG:teuthology.orchestra.run.trial030:> stat -- /home/ubuntu/cephtest/mnt.0 2026-02-23T11:20:17.378 DEBUG:teuthology.orchestra.run:got remote process result: 1 2026-02-23T11:20:17.378 INFO:teuthology.orchestra.run.trial030.stderr:stat: cannot statx '/home/ubuntu/cephtest/mnt.0': No such file or directory 2026-02-23T11:20:17.379 DEBUG:teuthology.orchestra.run.trial030:> mkdir -- /home/ubuntu/cephtest/mnt.0 2026-02-23T11:20:17.435 INFO:tasks.workunit:Created dir /home/ubuntu/cephtest/mnt.0 2026-02-23T11:20:17.436 DEBUG:teuthology.orchestra.run.trial030:> cd -- /home/ubuntu/cephtest/mnt.0 && mkdir -- client.0 2026-02-23T11:20:17.492 INFO:tasks.workunit:timeout=3h 2026-02-23T11:20:17.492 INFO:tasks.workunit:cleanup=True 2026-02-23T11:20:17.493 DEBUG:teuthology.orchestra.run.trial030:> rm -rf /home/ubuntu/cephtest/clone.client.0 && git clone https://github.com/shraddhaag/ceph.git /home/ubuntu/cephtest/clone.client.0 && cd /home/ubuntu/cephtest/clone.client.0 && git checkout e371433fcd1390a0b82cfb18effc422c1f5d8c1d 2026-02-23T11:20:17.555 INFO:tasks.workunit.client.0.trial030.stderr:Cloning into '/home/ubuntu/cephtest/clone.client.0'... 2026-02-23T11:20:38.830 INFO:tasks.workunit.client.0.trial030.stderr:Note: switching to 'e371433fcd1390a0b82cfb18effc422c1f5d8c1d'. 2026-02-23T11:20:38.830 INFO:tasks.workunit.client.0.trial030.stderr: 2026-02-23T11:20:38.831 INFO:tasks.workunit.client.0.trial030.stderr:You are in 'detached HEAD' state. You can look around, make experimental 2026-02-23T11:20:38.831 INFO:tasks.workunit.client.0.trial030.stderr:changes and commit them, and you can discard any commits you make in this 2026-02-23T11:20:38.831 INFO:tasks.workunit.client.0.trial030.stderr:state without impacting any branches by switching back to a branch. 2026-02-23T11:20:38.831 INFO:tasks.workunit.client.0.trial030.stderr: 2026-02-23T11:20:38.831 INFO:tasks.workunit.client.0.trial030.stderr:If you want to create a new branch to retain commits you create, you may 2026-02-23T11:20:38.831 INFO:tasks.workunit.client.0.trial030.stderr:do so (now or later) by using -c with the switch command. Example: 2026-02-23T11:20:38.831 INFO:tasks.workunit.client.0.trial030.stderr: 2026-02-23T11:20:38.831 INFO:tasks.workunit.client.0.trial030.stderr: git switch -c 2026-02-23T11:20:38.831 INFO:tasks.workunit.client.0.trial030.stderr: 2026-02-23T11:20:38.831 INFO:tasks.workunit.client.0.trial030.stderr:Or undo this operation with: 2026-02-23T11:20:38.831 INFO:tasks.workunit.client.0.trial030.stderr: 2026-02-23T11:20:38.832 INFO:tasks.workunit.client.0.trial030.stderr: git switch - 2026-02-23T11:20:38.832 INFO:tasks.workunit.client.0.trial030.stderr: 2026-02-23T11:20:38.832 INFO:tasks.workunit.client.0.trial030.stderr:Turn off this advice by setting config variable advice.detachedHead to false 2026-02-23T11:20:38.832 INFO:tasks.workunit.client.0.trial030.stderr: 2026-02-23T11:20:38.832 INFO:tasks.workunit.client.0.trial030.stderr:HEAD is now at e371433fcd1 qa/suites/crimson-rados: enable cephadm tests 2026-02-23T11:20:38.833 DEBUG:teuthology.orchestra.run.trial030:> cd -- /home/ubuntu/cephtest/clone.client.0/qa/workunits && if test -e Makefile ; then make ; fi && find -executable -type f -printf '%P\0' >/home/ubuntu/cephtest/workunits.list.client.0 2026-02-23T11:20:38.892 INFO:tasks.workunit.client.0.trial030.stdout:for d in direct_io fs ; do ( cd $d ; make all ) ; done 2026-02-23T11:20:38.894 INFO:tasks.workunit.client.0.trial030.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2026-02-23T11:20:38.894 INFO:tasks.workunit.client.0.trial030.stdout:cc -Wall -Wextra -D_GNU_SOURCE direct_io_test.c -o direct_io_test 2026-02-23T11:20:38.982 INFO:tasks.workunit.client.0.trial030.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_sync_io.c -o test_sync_io 2026-02-23T11:20:39.013 INFO:tasks.workunit.client.0.trial030.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_short_dio_read.c -o test_short_dio_read 2026-02-23T11:20:39.041 INFO:tasks.workunit.client.0.trial030.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2026-02-23T11:20:39.042 INFO:tasks.workunit.client.0.trial030.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2026-02-23T11:20:39.042 INFO:tasks.workunit.client.0.trial030.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_o_trunc.c -o test_o_trunc 2026-02-23T11:20:39.067 INFO:tasks.workunit.client.0.trial030.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2026-02-23T11:20:39.070 DEBUG:teuthology.orchestra.run.trial030:> set -ex 2026-02-23T11:20:39.070 DEBUG:teuthology.orchestra.run.trial030:> dd if=/home/ubuntu/cephtest/workunits.list.client.0 of=/dev/stdout 2026-02-23T11:20:39.130 INFO:tasks.workunit:Running workunits matching rbd/test_librbd_python.sh on client.0... 2026-02-23T11:20:39.131 INFO:tasks.workunit:Running workunit rbd/test_librbd_python.sh... 2026-02-23T11:20:39.132 DEBUG:teuthology.orchestra.run.trial030:workunit test rbd/test_librbd_python.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=e371433fcd1390a0b82cfb18effc422c1f5d8c1d TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 CRIMSON_COMPAT=1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/rbd/test_librbd_python.sh 2026-02-23T11:20:39.189 INFO:tasks.workunit.client.0.trial030.stderr:++ dirname /home/ubuntu/cephtest/clone.client.0/qa/workunits/rbd/test_librbd_python.sh 2026-02-23T11:20:39.190 INFO:tasks.workunit.client.0.trial030.stderr:+ relpath=/home/ubuntu/cephtest/clone.client.0/qa/workunits/rbd/../../../src/test/pybind 2026-02-23T11:20:39.190 INFO:tasks.workunit.client.0.trial030.stderr:+ '[' -n '' ']' 2026-02-23T11:20:39.190 INFO:tasks.workunit.client.0.trial030.stderr:+ python3 -m pytest -v /home/ubuntu/cephtest/clone.client.0/qa/workunits/rbd/../../../src/test/pybind/test_rbd.py 2026-02-23T11:20:39.265 INFO:tasks.workunit.client.0.trial030.stdout:============================= test session starts ============================== 2026-02-23T11:20:39.265 INFO:tasks.workunit.client.0.trial030.stdout:platform linux -- Python 3.9.25, pytest-6.2.2, py-1.10.0, pluggy-0.13.1 -- /usr/bin/python3 2026-02-23T11:20:39.265 INFO:tasks.workunit.client.0.trial030.stdout:cachedir: .pytest_cache 2026-02-23T11:20:39.265 INFO:tasks.workunit.client.0.trial030.stdout:rootdir: /home/ubuntu/cephtest/clone.client.0/src/test/pybind, configfile: pytest.ini 2026-02-23T11:20:39.562 INFO:tasks.workunit.client.0.trial030.stdout:collecting ... collected 196 items 2026-02-23T11:20:39.562 INFO:tasks.workunit.client.0.trial030.stdout: 2026-02-23T11:20:43.107 INFO:tasks.workunit.client.0.trial030.stdout:../../../clone.client.0/src/test/pybind/test_rbd.py::test_version PASSED [ 0%] 2026-02-23T11:20:43.615 INFO:tasks.workunit.client.0.trial030.stdout:../../../clone.client.0/src/test/pybind/test_rbd.py::test_create PASSED [ 1%] 2026-02-23T11:20:48.066 INFO:tasks.workunit.client.0.trial030.stdout:../../../clone.client.0/src/test/pybind/test_rbd.py::test_create_defaults PASSED [ 1%] 2026-02-23T11:20:48.813 INFO:tasks.workunit.client.0.trial030.stdout:../../../clone.client.0/src/test/pybind/test_rbd.py::test_context_manager PASSED [ 2%] 2026-02-23T11:20:50.919 INFO:tasks.workunit.client.0.trial030.stdout:../../../clone.client.0/src/test/pybind/test_rbd.py::test_open_read_only PASSED [ 2%] 2026-02-23T11:20:52.068 INFO:tasks.workunit.client.0.trial030.stdout:../../../clone.client.0/src/test/pybind/test_rbd.py::test_open_dne PASSED [ 3%] 2026-02-23T11:20:53.219 INFO:tasks.workunit.client.0.trial030.stdout:../../../clone.client.0/src/test/pybind/test_rbd.py::test_open_readonly_dne PASSED [ 3%] 2026-02-23T11:20:53.220 INFO:tasks.workunit.client.0.trial030.stdout:../../../clone.client.0/src/test/pybind/test_rbd.py::test_open_by_id SKIPPED [ 4%] 2026-02-23T11:20:53.835 INFO:tasks.workunit.client.0.trial030.stdout:../../../clone.client.0/src/test/pybind/test_rbd.py::test_aio_open PASSED [ 4%] 2026-02-23T11:20:53.898 INFO:tasks.workunit.client.0.trial030.stdout:../../../clone.client.0/src/test/pybind/test_rbd.py::test_remove_dne PASSED [ 5%] 2026-02-23T11:20:53.907 INFO:tasks.workunit.client.0.trial030.stdout:../../../clone.client.0/src/test/pybind/test_rbd.py::test_list_empty PASSED [ 5%] 2026-02-23T11:20:54.494 INFO:tasks.workunit.client.0.trial030.stdout:../../../clone.client.0/src/test/pybind/test_rbd.py::test_list PASSED [ 6%] 2026-02-23T11:20:54.988 INFO:tasks.workunit.client.0.trial030.stdout:../../../clone.client.0/src/test/pybind/test_rbd.py::test_remove_with_progress PASSED [ 6%] 2026-02-23T11:20:55.632 INFO:tasks.workunit.client.0.trial030.stdout:../../../clone.client.0/src/test/pybind/test_rbd.py::test_remove_canceled PASSED [ 7%] 2026-02-23T11:20:56.145 INFO:tasks.workunit.client.0.trial030.stdout:../../../clone.client.0/src/test/pybind/test_rbd.py::test_remove_with_progress_except PASSED [ 7%] 2026-02-23T11:20:56.955 INFO:tasks.workunit.client.0.trial030.stdout:../../../clone.client.0/src/test/pybind/test_rbd.py::test_rename PASSED [ 8%] 2026-02-23T11:20:58.779 INFO:tasks.workunit.client.0.trial030.stdout:../../../clone.client.0/src/test/pybind/test_rbd.py::test_pool_metadata PASSED [ 8%] 2026-02-23T11:20:58.825 INFO:tasks.workunit.client.0.trial030.stdout:../../../clone.client.0/src/test/pybind/test_rbd.py::test_config_list PASSED [ 9%] 2026-02-23T11:20:58.862 INFO:tasks.workunit.client.0.trial030.stdout:../../../clone.client.0/src/test/pybind/test_rbd.py::test_pool_config_set_and_get_and_remove PASSED [ 9%] 2026-02-23T11:20:59.052 INFO:tasks.workunit.client.0.trial030.stdout:../../../clone.client.0/src/test/pybind/test_rbd.py::test_namespaces PASSED [ 10%] 2026-02-23T11:20:59.053 INFO:tasks.workunit.client.0.trial030.stdout:../../../clone.client.0/src/test/pybind/test_rbd.py::test_pool_stats SKIPPED [ 10%] 2026-02-23T11:20:59.053 INFO:tasks.workunit.client.0.trial030.stdout:../../../clone.client.0/src/test/pybind/test_rbd.py::test_features_to_string SKIPPED [ 11%] 2026-02-23T11:20:59.053 INFO:tasks.workunit.client.0.trial030.stdout:../../../clone.client.0/src/test/pybind/test_rbd.py::test_features_from_string SKIPPED [ 11%] 2026-02-23T11:20:59.618 INFO:tasks.workunit.client.0.trial030.stdout:../../../clone.client.0/src/test/pybind/test_rbd.py::TestImage::test_update_features SKIPPED [ 12%] 2026-02-23T11:21:00.193 INFO:tasks.workunit.client.0.trial030.stdout:../../../clone.client.0/src/test/pybind/test_rbd.py::TestImage::test_create_with_params SKIPPED [ 12%] 2026-02-23T11:21:00.772 INFO:tasks.workunit.client.0.trial030.stdout:../../../clone.client.0/src/test/pybind/test_rbd.py::TestImage::test_id SKIPPED [ 13%] 2026-02-23T11:21:01.344 INFO:tasks.workunit.client.0.trial030.stdout:../../../clone.client.0/src/test/pybind/test_rbd.py::TestImage::test_block_name_prefix PASSED [ 13%] 2026-02-23T11:21:01.909 INFO:tasks.workunit.client.0.trial030.stdout:../../../clone.client.0/src/test/pybind/test_rbd.py::TestImage::test_data_pool_id PASSED [ 14%] 2026-02-23T11:21:02.495 INFO:tasks.workunit.client.0.trial030.stdout:../../../clone.client.0/src/test/pybind/test_rbd.py::TestImage::test_create_timestamp PASSED [ 14%] 2026-02-23T11:21:03.080 INFO:tasks.workunit.client.0.trial030.stdout:../../../clone.client.0/src/test/pybind/test_rbd.py::TestImage::test_access_timestamp PASSED [ 15%] 2026-02-23T11:21:03.656 INFO:tasks.workunit.client.0.trial030.stdout:../../../clone.client.0/src/test/pybind/test_rbd.py::TestImage::test_modify_timestamp PASSED [ 15%] 2026-02-23T11:21:04.375 INFO:tasks.workunit.client.0.trial030.stdout:../../../clone.client.0/src/test/pybind/test_rbd.py::TestImage::test_invalidate_cache PASSED [ 16%] 2026-02-23T11:21:04.946 INFO:tasks.workunit.client.0.trial030.stdout:../../../clone.client.0/src/test/pybind/test_rbd.py::TestImage::test_stat PASSED [ 16%] 2026-02-23T11:21:05.535 INFO:tasks.workunit.client.0.trial030.stdout:../../../clone.client.0/src/test/pybind/test_rbd.py::TestImage::test_flags PASSED [ 17%] 2026-02-23T11:21:06.166 INFO:tasks.workunit.client.0.trial030.stdout:../../../clone.client.0/src/test/pybind/test_rbd.py::TestImage::test_image_auto_close PASSED [ 17%] 2026-02-23T11:21:06.731 INFO:tasks.workunit.client.0.trial030.stdout:../../../clone.client.0/src/test/pybind/test_rbd.py::TestImage::test_use_after_close PASSED [ 18%] 2026-02-23T11:21:07.431 INFO:tasks.workunit.client.0.trial030.stdout:../../../clone.client.0/src/test/pybind/test_rbd.py::TestImage::test_write PASSED [ 18%] 2026-02-23T11:21:08.130 INFO:tasks.workunit.client.0.trial030.stdout:../../../clone.client.0/src/test/pybind/test_rbd.py::TestImage::test_write_with_fadvise_flags PASSED [ 19%] 2026-02-23T11:21:08.797 INFO:tasks.workunit.client.0.trial030.stdout:../../../clone.client.0/src/test/pybind/test_rbd.py::TestImage::test_write_zeroes PASSED [ 19%] 2026-02-23T11:21:09.501 INFO:tasks.workunit.client.0.trial030.stdout:../../../clone.client.0/src/test/pybind/test_rbd.py::TestImage::test_write_zeroes_thick_provision PASSED [ 20%] 2026-02-23T11:21:10.078 INFO:tasks.workunit.client.0.trial030.stdout:../../../clone.client.0/src/test/pybind/test_rbd.py::TestImage::test_read PASSED [ 20%] 2026-02-23T11:21:10.667 INFO:tasks.workunit.client.0.trial030.stdout:../../../clone.client.0/src/test/pybind/test_rbd.py::TestImage::test_read_with_fadvise_flags PASSED [ 21%] 2026-02-23T11:21:11.386 INFO:tasks.workunit.client.0.trial030.stdout:../../../clone.client.0/src/test/pybind/test_rbd.py::TestImage::test_large_write PASSED [ 21%] 2026-02-23T11:21:11.986 INFO:tasks.workunit.client.0.trial030.stdout:../../../clone.client.0/src/test/pybind/test_rbd.py::TestImage::test_large_read PASSED [ 22%] 2026-02-23T11:21:12.682 INFO:tasks.workunit.client.0.trial030.stdout:../../../clone.client.0/src/test/pybind/test_rbd.py::TestImage::test_write_read PASSED [ 22%] 2026-02-23T11:21:13.278 INFO:tasks.workunit.client.0.trial030.stdout:../../../clone.client.0/src/test/pybind/test_rbd.py::TestImage::test_read_bad_offset PASSED [ 23%] 2026-02-23T11:21:13.950 INFO:tasks.workunit.client.0.trial030.stdout:../../../clone.client.0/src/test/pybind/test_rbd.py::TestImage::test_resize PASSED [ 23%] 2026-02-23T11:21:14.638 INFO:tasks.workunit.client.0.trial030.stdout:../../../clone.client.0/src/test/pybind/test_rbd.py::TestImage::test_resize_allow_shrink_False PASSED [ 24%] 2026-02-23T11:21:18.912 INFO:tasks.workunit.client.0.trial030.stdout:../../../clone.client.0/src/test/pybind/test_rbd.py::TestImage::test_size PASSED [ 25%] 2026-02-23T11:21:19.719 INFO:tasks.workunit.client.0.trial030.stdout:../../../clone.client.0/src/test/pybind/test_rbd.py::TestImage::test_resize_down PASSED [ 25%] 2026-02-23T11:21:20.581 INFO:tasks.workunit.client.0.trial030.stdout:../../../clone.client.0/src/test/pybind/test_rbd.py::TestImage::test_resize_bytes PASSED [ 26%] 2026-02-23T11:21:22.077 INFO:tasks.workunit.client.0.trial030.stdout:../../../clone.client.0/src/test/pybind/test_rbd.py::TestImage::test_copy PASSED [ 26%] 2026-02-23T11:21:23.539 INFO:tasks.workunit.client.0.trial030.stdout:../../../clone.client.0/src/test/pybind/test_rbd.py::TestImage::test_copy2 PASSED [ 27%] 2026-02-23T11:21:24.109 INFO:tasks.workunit.client.0.trial030.stdout:../../../clone.client.0/src/test/pybind/test_rbd.py::TestImage::test_copy3 SKIPPED [ 27%] 2026-02-23T11:21:28.536 INFO:tasks.workunit.client.0.trial030.stdout:../../../clone.client.0/src/test/pybind/test_rbd.py::TestImage::test_deep_copy PASSED [ 28%] 2026-02-23T11:21:29.134 INFO:tasks.workunit.client.0.trial030.stdout:../../../clone.client.0/src/test/pybind/test_rbd.py::TestImage::test_deep_copy_clone_v1_to_v1 SKIPPED [ 28%] 2026-02-23T11:21:29.740 INFO:tasks.workunit.client.0.trial030.stdout:../../../clone.client.0/src/test/pybind/test_rbd.py::TestImage::test_deep_copy_clone_v1_to_v2 SKIPPED [ 29%] 2026-02-23T11:21:30.308 INFO:tasks.workunit.client.0.trial030.stdout:../../../clone.client.0/src/test/pybind/test_rbd.py::TestImage::test_deep_copy_clone_v2_to_v1 SKIPPED [ 29%] 2026-02-23T11:21:30.904 INFO:tasks.workunit.client.0.trial030.stdout:../../../clone.client.0/src/test/pybind/test_rbd.py::TestImage::test_deep_copy_clone_v2_to_v2 SKIPPED [ 30%] 2026-02-23T11:21:31.501 INFO:tasks.workunit.client.0.trial030.stdout:../../../clone.client.0/src/test/pybind/test_rbd.py::TestImage::test_deep_copy_clone_v1_flatten SKIPPED [ 30%] 2026-02-23T11:21:32.080 INFO:tasks.workunit.client.0.trial030.stdout:../../../clone.client.0/src/test/pybind/test_rbd.py::TestImage::test_deep_copy_clone_v2_flatten SKIPPED [ 31%] 2026-02-23T11:21:34.006 INFO:tasks.workunit.client.0.trial030.stdout:../../../clone.client.0/src/test/pybind/test_rbd.py::TestImage::test_create_snap PASSED [ 31%] 2026-02-23T11:21:36.000 INFO:tasks.workunit.client.0.trial030.stdout:../../../clone.client.0/src/test/pybind/test_rbd.py::TestImage::test_create_snap_exists PASSED [ 32%] 2026-02-23T11:21:42.026 INFO:tasks.workunit.client.0.trial030.stdout:../../../clone.client.0/src/test/pybind/test_rbd.py::TestImage::test_create_snap_flags PASSED [ 32%] 2026-02-23T11:21:46.002 INFO:tasks.workunit.client.0.trial030.stdout:../../../clone.client.0/src/test/pybind/test_rbd.py::TestImage::test_list_snaps PASSED [ 33%] 2026-02-23T11:21:48.035 INFO:tasks.workunit.client.0.trial030.stdout:../../../clone.client.0/src/test/pybind/test_rbd.py::TestImage::test_list_snaps_iterator_auto_close PASSED [ 33%] 2026-02-23T11:21:50.079 INFO:tasks.workunit.client.0.trial030.stdout:../../../clone.client.0/src/test/pybind/test_rbd.py::TestImage::test_remove_snap PASSED [ 34%] 2026-02-23T11:21:50.683 INFO:tasks.workunit.client.0.trial030.stdout:../../../clone.client.0/src/test/pybind/test_rbd.py::TestImage::test_remove_snap_not_found PASSED [ 34%] 2026-02-23T11:21:51.320 INFO:tasks.workunit.client.0.trial030.stdout:../../../clone.client.0/src/test/pybind/test_rbd.py::TestImage::test_remove_snap2 SKIPPED [ 35%] 2026-02-23T11:21:53.064 INFO:tasks.workunit.client.0.trial030.stdout:../../../clone.client.0/src/test/pybind/test_rbd.py::TestImage::test_remove_snap_by_id PASSED [ 35%] 2026-02-23T11:21:55.097 INFO:tasks.workunit.client.0.trial030.stdout:../../../clone.client.0/src/test/pybind/test_rbd.py::TestImage::test_rename_snap PASSED [ 36%] 2026-02-23T11:21:55.709 INFO:tasks.workunit.client.0.trial030.stdout:../../../clone.client.0/src/test/pybind/test_rbd.py::TestImage::test_protect_snap SKIPPED [ 36%] 2026-02-23T11:21:58.092 INFO:tasks.workunit.client.0.trial030.stdout:../../../clone.client.0/src/test/pybind/test_rbd.py::TestImage::test_snap_exists PASSED [ 37%] 2026-02-23T11:22:00.084 INFO:tasks.workunit.client.0.trial030.stdout:../../../clone.client.0/src/test/pybind/test_rbd.py::TestImage::test_snap_timestamp PASSED [ 37%] 2026-02-23T11:22:08.102 INFO:tasks.workunit.client.0.trial030.stdout:../../../clone.client.0/src/test/pybind/test_rbd.py::TestImage::test_limit_snaps PASSED [ 38%] 2026-02-23T11:22:08.691 INFO:tasks.workunit.client.0.trial030.stdout:../../../clone.client.0/src/test/pybind/test_rbd.py::TestImage::test_remove_with_exclusive_lock SKIPPED [ 38%] 2026-02-23T11:22:11.118 INFO:tasks.workunit.client.0.trial030.stdout:../../../clone.client.0/src/test/pybind/test_rbd.py::TestImage::test_remove_with_snap PASSED [ 39%] 2026-02-23T11:22:11.948 INFO:tasks.workunit.client.0.trial030.stdout:../../../clone.client.0/src/test/pybind/test_rbd.py::TestImage::test_remove_with_watcher PASSED [ 39%] 2026-02-23T11:22:13.966 INFO:tasks.workunit.client.0.trial030.stdout:../../../clone.client.0/src/test/pybind/test_rbd.py::TestImage::test_rollback_to_snap PASSED [ 40%] 2026-02-23T11:22:16.149 INFO:tasks.workunit.client.0.trial030.stdout:../../../clone.client.0/src/test/pybind/test_rbd.py::TestImage::test_rollback_to_snap_sparse PASSED [ 40%] 2026-02-23T11:22:19.966 INFO:tasks.workunit.client.0.trial030.stdout:../../../clone.client.0/src/test/pybind/test_rbd.py::TestImage::test_rollback_with_resize PASSED [ 41%] 2026-02-23T11:22:21.927 INFO:tasks.workunit.client.0.trial030.stdout:../../../clone.client.0/src/test/pybind/test_rbd.py::TestImage::test_set_snap PASSED [ 41%] 2026-02-23T11:22:23.969 INFO:tasks.workunit.client.0.trial030.stdout:../../../clone.client.0/src/test/pybind/test_rbd.py::TestImage::test_set_no_snap PASSED [ 42%] 2026-02-23T11:22:25.990 INFO:tasks.workunit.client.0.trial030.stdout:../../../clone.client.0/src/test/pybind/test_rbd.py::TestImage::test_set_snap_by_id PASSED [ 42%] 2026-02-23T11:22:31.995 INFO:tasks.workunit.client.0.trial030.stdout:../../../clone.client.0/src/test/pybind/test_rbd.py::TestImage::test_snap_get_name PASSED [ 43%] 2026-02-23T11:22:38.016 INFO:tasks.workunit.client.0.trial030.stdout:../../../clone.client.0/src/test/pybind/test_rbd.py::TestImage::test_snap_get_id PASSED [ 43%] 2026-02-23T11:22:40.003 INFO:tasks.workunit.client.0.trial030.stdout:../../../clone.client.0/src/test/pybind/test_rbd.py::TestImage::test_set_snap_sparse PASSED [ 44%] 2026-02-23T11:31:14.113 INFO:tasks.workunit.client.0.trial030.stdout:../../../clone.client.0/src/test/pybind/test_rbd.py::TestImage::test_many_snaps PASSED [ 44%] 2026-02-23T11:31:16.141 INFO:tasks.workunit.client.0.trial030.stdout:../../../clone.client.0/src/test/pybind/test_rbd.py::TestImage::test_set_snap_deleted PASSED [ 45%] 2026-02-23T11:31:20.184 INFO:tasks.workunit.client.0.trial030.stdout:../../../clone.client.0/src/test/pybind/test_rbd.py::TestImage::test_set_snap_recreated PASSED [ 45%] 2026-02-23T11:31:20.896 INFO:tasks.workunit.client.0.trial030.stdout:../../../clone.client.0/src/test/pybind/test_rbd.py::TestImage::test_lock_unlock PASSED [ 46%] 2026-02-23T11:31:22.416 INFO:tasks.workunit.client.0.trial030.stdout:../../../clone.client.0/src/test/pybind/test_rbd.py::TestImage::test_list_lockers PASSED [ 46%] 2026-02-23T11:31:29.207 INFO:tasks.workunit.client.0.trial030.stdout:../../../clone.client.0/src/test/pybind/test_rbd.py::TestImage::test_diff_iterate PASSED [ 47%] 2026-02-23T11:31:29.836 INFO:tasks.workunit.client.0.trial030.stdout:../../../clone.client.0/src/test/pybind/test_rbd.py::TestImage::test_diff_iterate_from_trash_snap SKIPPED [ 47%] 2026-02-23T11:31:30.453 INFO:tasks.workunit.client.0.trial030.stdout:../../../clone.client.0/src/test/pybind/test_rbd.py::TestImage::test_diff_iterate_exclude_parent SKIPPED [ 48%] 2026-02-23T11:31:31.063 INFO:tasks.workunit.client.0.trial030.stdout:../../../clone.client.0/src/test/pybind/test_rbd.py::TestImage::test_aio_read PASSED [ 48%] 2026-02-23T11:31:31.819 INFO:tasks.workunit.client.0.trial030.stdout:../../../clone.client.0/src/test/pybind/test_rbd.py::TestImage::test_aio_write PASSED [ 49%] 2026-02-23T11:31:32.610 INFO:tasks.workunit.client.0.trial030.stdout:../../../clone.client.0/src/test/pybind/test_rbd.py::TestImage::test_aio_discard PASSED [ 50%] 2026-02-23T11:31:33.347 INFO:tasks.workunit.client.0.trial030.stdout:../../../clone.client.0/src/test/pybind/test_rbd.py::TestImage::test_aio_write_zeroes PASSED [ 50%] 2026-02-23T11:31:34.053 INFO:tasks.workunit.client.0.trial030.stdout:../../../clone.client.0/src/test/pybind/test_rbd.py::TestImage::test_aio_flush PASSED [ 51%] 2026-02-23T11:31:40.568 INFO:tasks.workunit.client.0.trial030.stdout:../../../clone.client.0/src/test/pybind/test_rbd.py::TestImage::test_metadata PASSED [ 51%] 2026-02-23T11:31:41.188 INFO:tasks.workunit.client.0.trial030.stdout:../../../clone.client.0/src/test/pybind/test_rbd.py::TestImage::test_watchers_list PASSED [ 52%] 2026-02-23T11:31:42.084 INFO:tasks.workunit.client.0.trial030.stdout:../../../clone.client.0/src/test/pybind/test_rbd.py::TestImage::test_config_list PASSED [ 52%] 2026-02-23T11:31:42.935 INFO:tasks.workunit.client.0.trial030.stdout:../../../clone.client.0/src/test/pybind/test_rbd.py::TestImage::test_image_config_set_and_get_and_remove PASSED [ 53%] 2026-02-23T11:31:43.596 INFO:tasks.workunit.client.0.trial030.stdout:../../../clone.client.0/src/test/pybind/test_rbd.py::TestImage::test_sparsify PASSED [ 53%] 2026-02-23T11:31:52.863 INFO:tasks.workunit.client.0.trial030.stdout:../../../clone.client.0/src/test/pybind/test_rbd.py::TestImage::test_encryption_luks1 PASSED [ 54%] 2026-02-23T11:32:05.477 INFO:tasks.workunit.client.0.trial030.stdout:../../../clone.client.0/src/test/pybind/test_rbd.py::TestImage::test_encryption_luks2 PASSED [ 54%] 2026-02-23T11:32:06.202 INFO:tasks.workunit.client.0.trial030.stdout:../../../clone.client.0/src/test/pybind/test_rbd.py::TestImageId::test_read PASSED [ 55%] 2026-02-23T11:32:07.015 INFO:tasks.workunit.client.0.trial030.stdout:../../../clone.client.0/src/test/pybind/test_rbd.py::TestImageId::test_write PASSED [ 55%] 2026-02-23T11:32:07.803 INFO:tasks.workunit.client.0.trial030.stdout:../../../clone.client.0/src/test/pybind/test_rbd.py::TestImageId::test_resize PASSED [ 56%] 2026-02-23T11:32:07.804 INFO:tasks.workunit.client.0.trial030.stdout:../../../clone.client.0/src/test/pybind/test_rbd.py::TestClone::test_with_params SKIPPED [ 56%] 2026-02-23T11:32:07.804 INFO:tasks.workunit.client.0.trial030.stdout:../../../clone.client.0/src/test/pybind/test_rbd.py::TestClone::test_with_params2 SKIPPED [ 57%] 2026-02-23T11:32:07.804 INFO:tasks.workunit.client.0.trial030.stdout:../../../clone.client.0/src/test/pybind/test_rbd.py::TestClone::test_with_params3 SKIPPED [ 57%] 2026-02-23T11:32:07.805 INFO:tasks.workunit.client.0.trial030.stdout:../../../clone.client.0/src/test/pybind/test_rbd.py::TestClone::test_stripe_unit_and_count SKIPPED [ 58%] 2026-02-23T11:32:07.805 INFO:tasks.workunit.client.0.trial030.stdout:../../../clone.client.0/src/test/pybind/test_rbd.py::TestClone::test_clone_format SKIPPED [ 58%] 2026-02-23T11:32:07.805 INFO:tasks.workunit.client.0.trial030.stdout:../../../clone.client.0/src/test/pybind/test_rbd.py::TestClone::test_unprotected SKIPPED [ 59%] 2026-02-23T11:32:07.806 INFO:tasks.workunit.client.0.trial030.stdout:../../../clone.client.0/src/test/pybind/test_rbd.py::TestClone::test_unprotect_with_children SKIPPED [ 59%] 2026-02-23T11:32:07.807 INFO:tasks.workunit.client.0.trial030.stdout:../../../clone.client.0/src/test/pybind/test_rbd.py::TestClone::test_clone_by_snap_id SKIPPED [ 60%] 2026-02-23T11:32:07.807 INFO:tasks.workunit.client.0.trial030.stdout:../../../clone.client.0/src/test/pybind/test_rbd.py::TestClone::test_stat SKIPPED [ 60%] 2026-02-23T11:32:07.807 INFO:tasks.workunit.client.0.trial030.stdout:../../../clone.client.0/src/test/pybind/test_rbd.py::TestClone::test_resize_stat SKIPPED [ 61%] 2026-02-23T11:32:07.808 INFO:tasks.workunit.client.0.trial030.stdout:../../../clone.client.0/src/test/pybind/test_rbd.py::TestClone::test_resize_io SKIPPED [ 61%] 2026-02-23T11:32:07.808 INFO:tasks.workunit.client.0.trial030.stdout:../../../clone.client.0/src/test/pybind/test_rbd.py::TestClone::test_read SKIPPED [ 62%] 2026-02-23T11:32:07.808 INFO:tasks.workunit.client.0.trial030.stdout:../../../clone.client.0/src/test/pybind/test_rbd.py::TestClone::test_write SKIPPED [ 62%] 2026-02-23T11:32:07.808 INFO:tasks.workunit.client.0.trial030.stdout:../../../clone.client.0/src/test/pybind/test_rbd.py::TestClone::test_list_children SKIPPED [ 63%] 2026-02-23T11:32:07.809 INFO:tasks.workunit.client.0.trial030.stdout:../../../clone.client.0/src/test/pybind/test_rbd.py::TestClone::test_flatten_errors SKIPPED [ 63%] 2026-02-23T11:32:07.809 INFO:tasks.workunit.client.0.trial030.stdout:../../../clone.client.0/src/test/pybind/test_rbd.py::TestClone::test_flatten_basic SKIPPED [ 64%] 2026-02-23T11:32:07.809 INFO:tasks.workunit.client.0.trial030.stdout:../../../clone.client.0/src/test/pybind/test_rbd.py::TestClone::test_flatten_smaller_order SKIPPED [ 64%] 2026-02-23T11:32:07.809 INFO:tasks.workunit.client.0.trial030.stdout:../../../clone.client.0/src/test/pybind/test_rbd.py::TestClone::test_flatten_larger_order SKIPPED [ 65%] 2026-02-23T11:32:07.810 INFO:tasks.workunit.client.0.trial030.stdout:../../../clone.client.0/src/test/pybind/test_rbd.py::TestClone::test_flatten_drops_cache SKIPPED [ 65%] 2026-02-23T11:32:07.810 INFO:tasks.workunit.client.0.trial030.stdout:../../../clone.client.0/src/test/pybind/test_rbd.py::TestClone::test_flatten_multi_level SKIPPED [ 66%] 2026-02-23T11:32:07.811 INFO:tasks.workunit.client.0.trial030.stdout:../../../clone.client.0/src/test/pybind/test_rbd.py::TestClone::test_flatten_with_progress SKIPPED [ 66%] 2026-02-23T11:32:07.812 INFO:tasks.workunit.client.0.trial030.stdout:../../../clone.client.0/src/test/pybind/test_rbd.py::TestClone::test_resize_flatten_multi_level SKIPPED [ 67%] 2026-02-23T11:32:07.812 INFO:tasks.workunit.client.0.trial030.stdout:../../../clone.client.0/src/test/pybind/test_rbd.py::TestClone::test_trash_snapshot SKIPPED [ 67%] 2026-02-23T11:32:07.812 INFO:tasks.workunit.client.0.trial030.stdout:../../../clone.client.0/src/test/pybind/test_rbd.py::TestClone::test_encryption_luks1 SKIPPED [ 68%] 2026-02-23T11:32:07.813 INFO:tasks.workunit.client.0.trial030.stdout:../../../clone.client.0/src/test/pybind/test_rbd.py::TestClone::test_encryption_luks2 SKIPPED [ 68%] 2026-02-23T11:32:07.813 INFO:tasks.workunit.client.0.trial030.stdout:../../../clone.client.0/src/test/pybind/test_rbd.py::TestExclusiveLock::test_ownership SKIPPED [ 69%] 2026-02-23T11:32:07.814 INFO:tasks.workunit.client.0.trial030.stdout:../../../clone.client.0/src/test/pybind/test_rbd.py::TestExclusiveLock::test_snapshot_leadership SKIPPED [ 69%] 2026-02-23T11:32:07.814 INFO:tasks.workunit.client.0.trial030.stdout:../../../clone.client.0/src/test/pybind/test_rbd.py::TestExclusiveLock::test_read_only_leadership SKIPPED [ 70%] 2026-02-23T11:32:07.814 INFO:tasks.workunit.client.0.trial030.stdout:../../../clone.client.0/src/test/pybind/test_rbd.py::TestExclusiveLock::test_follower_flatten SKIPPED [ 70%] 2026-02-23T11:32:07.815 INFO:tasks.workunit.client.0.trial030.stdout:../../../clone.client.0/src/test/pybind/test_rbd.py::TestExclusiveLock::test_follower_resize SKIPPED [ 71%] 2026-02-23T11:32:07.815 INFO:tasks.workunit.client.0.trial030.stdout:../../../clone.client.0/src/test/pybind/test_rbd.py::TestExclusiveLock::test_follower_snap_create SKIPPED [ 71%] 2026-02-23T11:32:07.815 INFO:tasks.workunit.client.0.trial030.stdout:../../../clone.client.0/src/test/pybind/test_rbd.py::TestExclusiveLock::test_follower_snap_rollback SKIPPED [ 72%] 2026-02-23T11:32:07.816 INFO:tasks.workunit.client.0.trial030.stdout:../../../clone.client.0/src/test/pybind/test_rbd.py::TestExclusiveLock::test_follower_discard SKIPPED [ 72%] 2026-02-23T11:32:07.816 INFO:tasks.workunit.client.0.trial030.stdout:../../../clone.client.0/src/test/pybind/test_rbd.py::TestExclusiveLock::test_follower_write SKIPPED [ 73%] 2026-02-23T11:32:07.817 INFO:tasks.workunit.client.0.trial030.stdout:../../../clone.client.0/src/test/pybind/test_rbd.py::TestExclusiveLock::test_acquire_release_lock SKIPPED [ 73%] 2026-02-23T11:32:07.817 INFO:tasks.workunit.client.0.trial030.stdout:../../../clone.client.0/src/test/pybind/test_rbd.py::TestExclusiveLock::test_break_lock SKIPPED [ 74%] 2026-02-23T11:32:08.524 INFO:tasks.workunit.client.0.trial030.stdout:../../../clone.client.0/src/test/pybind/test_rbd.py::TestMirroring::test_uuid PASSED [ 75%] 2026-02-23T11:32:09.256 INFO:tasks.workunit.client.0.trial030.stdout:../../../clone.client.0/src/test/pybind/test_rbd.py::TestMirroring::test_site_name PASSED [ 75%] 2026-02-23T11:32:10.587 INFO:tasks.workunit.client.0.trial030.stdout:../../../clone.client.0/src/test/pybind/test_rbd.py::TestMirroring::test_mirror_remote_namespace PASSED [ 76%] 2026-02-23T11:32:11.373 INFO:tasks.workunit.client.0.trial030.stdout:../../../clone.client.0/src/test/pybind/test_rbd.py::TestMirroring::test_mirror_peer_bootstrap PASSED [ 76%] 2026-02-23T11:32:12.157 INFO:tasks.workunit.client.0.trial030.stdout:../../../clone.client.0/src/test/pybind/test_rbd.py::TestMirroring::test_mirror_peer PASSED [ 77%] 2026-02-23T11:32:12.853 INFO:tasks.workunit.client.0.trial030.stdout:../../../clone.client.0/src/test/pybind/test_rbd.py::TestMirroring::test_mirror_image SKIPPED [ 77%] 2026-02-23T11:32:13.553 INFO:tasks.workunit.client.0.trial030.stdout:../../../clone.client.0/src/test/pybind/test_rbd.py::TestMirroring::test_mirror_image_status SKIPPED [ 78%] 2026-02-23T11:32:22.522 INFO:tasks.workunit.client.0.trial030.stdout:../../../clone.client.0/src/test/pybind/test_rbd.py::TestMirroring::test_mirror_image_create_snapshot PASSED [ 78%] 2026-02-23T11:32:25.977 INFO:tasks.workunit.client.0.trial030.stdout:../../../clone.client.0/src/test/pybind/test_rbd.py::TestMirroring::test_aio_mirror_image_create_snapshot PASSED [ 79%] 2026-02-23T11:32:26.494 INFO:tasks.workunit.client.0.trial030.stdout:../../../clone.client.0/src/test/pybind/test_rbd.py::TestTrash::test_move PASSED [ 79%] 2026-02-23T11:32:27.514 INFO:tasks.workunit.client.0.trial030.stdout:../../../clone.client.0/src/test/pybind/test_rbd.py::TestTrash::test_purge PASSED [ 80%] 2026-02-23T11:32:28.033 INFO:tasks.workunit.client.0.trial030.stdout:../../../clone.client.0/src/test/pybind/test_rbd.py::TestTrash::test_remove_denied PASSED [ 80%] 2026-02-23T11:32:28.546 INFO:tasks.workunit.client.0.trial030.stdout:../../../clone.client.0/src/test/pybind/test_rbd.py::TestTrash::test_remove PASSED [ 81%] 2026-02-23T11:32:29.061 INFO:tasks.workunit.client.0.trial030.stdout:../../../clone.client.0/src/test/pybind/test_rbd.py::TestTrash::test_remove_with_progress PASSED [ 81%] 2026-02-23T11:32:29.573 INFO:tasks.workunit.client.0.trial030.stdout:../../../clone.client.0/src/test/pybind/test_rbd.py::TestTrash::test_get PASSED [ 82%] 2026-02-23T11:32:30.600 INFO:tasks.workunit.client.0.trial030.stdout:../../../clone.client.0/src/test/pybind/test_rbd.py::TestTrash::test_list PASSED [ 82%] 2026-02-23T11:32:31.479 INFO:tasks.workunit.client.0.trial030.stdout:../../../clone.client.0/src/test/pybind/test_rbd.py::TestTrash::test_restore PASSED [ 83%] 2026-02-23T11:32:31.535 INFO:tasks.workunit.client.0.trial030.stdout:../../../clone.client.0/src/test/pybind/test_rbd.py::test_create_group PASSED [ 83%] 2026-02-23T11:32:31.608 INFO:tasks.workunit.client.0.trial030.stdout:../../../clone.client.0/src/test/pybind/test_rbd.py::test_rename_group PASSED [ 84%] 2026-02-23T11:32:31.612 INFO:tasks.workunit.client.0.trial030.stdout:../../../clone.client.0/src/test/pybind/test_rbd.py::test_list_groups_empty PASSED [ 84%] 2026-02-23T11:32:31.679 INFO:tasks.workunit.client.0.trial030.stdout:../../../clone.client.0/src/test/pybind/test_rbd.py::test_list_groups PASSED [ 85%] 2026-02-23T11:32:31.738 INFO:tasks.workunit.client.0.trial030.stdout:../../../clone.client.0/src/test/pybind/test_rbd.py::test_list_groups_after_removed PASSED [ 85%] 2026-02-23T11:32:32.511 INFO:tasks.workunit.client.0.trial030.stdout:../../../clone.client.0/src/test/pybind/test_rbd.py::TestGroups::test_group_image_add PASSED [ 86%] 2026-02-23T11:32:33.259 INFO:tasks.workunit.client.0.trial030.stdout:../../../clone.client.0/src/test/pybind/test_rbd.py::TestGroups::test_group_image_list_empty PASSED [ 86%] 2026-02-23T11:32:34.069 INFO:tasks.workunit.client.0.trial030.stdout:../../../clone.client.0/src/test/pybind/test_rbd.py::TestGroups::test_group_image_list PASSED [ 87%] 2026-02-23T11:32:35.082 INFO:tasks.workunit.client.0.trial030.stdout:../../../clone.client.0/src/test/pybind/test_rbd.py::TestGroups::test_group_image_list_move_to_trash PASSED [ 87%] 2026-02-23T11:32:36.618 INFO:tasks.workunit.client.0.trial030.stdout:../../../clone.client.0/src/test/pybind/test_rbd.py::TestGroups::test_group_image_list_remove PASSED [ 88%] 2026-02-23T11:32:37.296 INFO:tasks.workunit.client.0.trial030.stdout:../../../clone.client.0/src/test/pybind/test_rbd.py::TestGroups::test_group_get_id PASSED [ 88%] 2026-02-23T11:32:51.181 INFO:tasks.workunit.client.0.trial030.stdout:../../../clone.client.0/src/test/pybind/test_rbd.py::TestGroups::test_group_image_many_images PASSED [ 89%] 2026-02-23T11:32:52.103 INFO:tasks.workunit.client.0.trial030.stdout:../../../clone.client.0/src/test/pybind/test_rbd.py::TestGroups::test_group_image_remove PASSED [ 89%] 2026-02-23T11:32:54.764 INFO:tasks.workunit.client.0.trial030.stdout:../../../clone.client.0/src/test/pybind/test_rbd.py::TestGroups::test_group_snap_get_info PASSED [ 90%] 2026-02-23T11:32:55.517 INFO:tasks.workunit.client.0.trial030.stdout:../../../clone.client.0/src/test/pybind/test_rbd.py::TestGroups::test_group_snap_get_info_no_member_images PASSED [ 90%] 2026-02-23T11:32:56.292 INFO:tasks.workunit.client.0.trial030.stdout:../../../clone.client.0/src/test/pybind/test_rbd.py::TestGroups::test_group_snap PASSED [ 91%] 2026-02-23T11:32:57.185 INFO:tasks.workunit.client.0.trial030.stdout:../../../clone.client.0/src/test/pybind/test_rbd.py::TestGroups::test_group_snap_flags PASSED [ 91%] 2026-02-23T11:33:39.107 INFO:tasks.workunit.client.0.trial030.stdout:../../../clone.client.0/src/test/pybind/test_rbd.py::TestGroups::test_group_snap_list_many PASSED [ 92%] 2026-02-23T11:33:41.436 INFO:tasks.workunit.client.0.trial030.stdout:../../../clone.client.0/src/test/pybind/test_rbd.py::TestGroups::test_group_snap_namespace PASSED [ 92%] 2026-02-23T11:33:42.235 INFO:tasks.workunit.client.0.trial030.stdout:../../../clone.client.0/src/test/pybind/test_rbd.py::TestGroups::test_group_snap_rename PASSED [ 93%] 2026-02-23T11:33:42.945 INFO:tasks.workunit.client.0.trial030.stdout:../../../clone.client.0/src/test/pybind/test_rbd.py::TestGroups::test_group_snap_clone SKIPPED [ 93%] 2026-02-23T11:33:43.698 INFO:tasks.workunit.client.0.trial030.stdout:../../../clone.client.0/src/test/pybind/test_rbd.py::TestGroups::test_group_snap_clone_flatten SKIPPED [ 94%] 2026-02-23T11:33:46.245 INFO:tasks.workunit.client.0.trial030.stdout:../../../clone.client.0/src/test/pybind/test_rbd.py::TestGroups::test_group_snap_diff_iterate PASSED [ 94%] 2026-02-23T11:33:59.488 INFO:tasks.workunit.client.0.trial030.stdout:../../../clone.client.0/src/test/pybind/test_rbd.py::TestGroups::test_group_snap_rollback PASSED [ 95%] 2026-02-23T11:34:01.864 INFO:tasks.workunit.client.0.trial030.stdout:../../../clone.client.0/src/test/pybind/test_rbd.py::TestMigration::test_migration PASSED [ 95%] 2026-02-23T11:34:01.865 INFO:tasks.workunit.client.0.trial030.stdout:../../../clone.client.0/src/test/pybind/test_rbd.py::TestMigration::test_migration_clone_v1_to_v1 SKIPPED [ 96%] 2026-02-23T11:34:01.865 INFO:tasks.workunit.client.0.trial030.stdout:../../../clone.client.0/src/test/pybind/test_rbd.py::TestMigration::test_migration_clone_v1_to_v2 SKIPPED [ 96%] 2026-02-23T11:34:01.866 INFO:tasks.workunit.client.0.trial030.stdout:../../../clone.client.0/src/test/pybind/test_rbd.py::TestMigration::test_migration_clone_v2_to_v1 SKIPPED [ 97%] 2026-02-23T11:34:01.866 INFO:tasks.workunit.client.0.trial030.stdout:../../../clone.client.0/src/test/pybind/test_rbd.py::TestMigration::test_migration_clone_v2_to_v2 SKIPPED [ 97%] 2026-02-23T11:34:07.643 INFO:tasks.workunit.client.0.trial030.stdout:../../../clone.client.0/src/test/pybind/test_rbd.py::TestMigration::test_migration_import PASSED [ 98%] 2026-02-23T11:34:09.628 INFO:tasks.workunit.client.0.trial030.stdout:../../../clone.client.0/src/test/pybind/test_rbd.py::TestMigration::test_migration_with_progress PASSED [ 98%] 2026-02-23T11:34:11.264 INFO:tasks.workunit.client.0.trial030.stdout:../../../clone.client.0/src/test/pybind/test_rbd.py::TestMigration::test_migrate_abort PASSED [ 99%] 2026-02-23T11:34:13.541 INFO:tasks.workunit.client.0.trial030.stdout:../../../clone.client.0/src/test/pybind/test_rbd.py::TestMigration::test_migrate_abort_with_progress PASSED [100%] 2026-02-23T11:34:13.541 INFO:tasks.workunit.client.0.trial030.stdout: 2026-02-23T11:34:13.541 INFO:tasks.workunit.client.0.trial030.stdout:=============================== warnings summary =============================== 2026-02-23T11:34:13.541 INFO:tasks.workunit.client.0.trial030.stdout:test_rbd.py::test_remove_with_progress_except 2026-02-23T11:34:13.541 INFO:tasks.workunit.client.0.trial030.stdout: /usr/lib/python3.9/site-packages/_pytest/unraisableexception.py:78: PytestUnraisableExceptionWarning: Exception ignored in: 'rbd.progress_callback' 2026-02-23T11:34:13.541 INFO:tasks.workunit.client.0.trial030.stdout: 2026-02-23T11:34:13.541 INFO:tasks.workunit.client.0.trial030.stdout: Traceback (most recent call last): 2026-02-23T11:34:13.541 INFO:tasks.workunit.client.0.trial030.stdout: File "/home/ubuntu/cephtest/clone.client.0/src/test/pybind/test_rbd.py", line 430, in progress_cb 2026-02-23T11:34:13.542 INFO:tasks.workunit.client.0.trial030.stdout: raise Exception() 2026-02-23T11:34:13.542 INFO:tasks.workunit.client.0.trial030.stdout: Exception 2026-02-23T11:34:13.542 INFO:tasks.workunit.client.0.trial030.stdout: 2026-02-23T11:34:13.542 INFO:tasks.workunit.client.0.trial030.stdout: warnings.warn(pytest.PytestUnraisableExceptionWarning(msg)) 2026-02-23T11:34:13.542 INFO:tasks.workunit.client.0.trial030.stdout: 2026-02-23T11:34:13.542 INFO:tasks.workunit.client.0.trial030.stdout:-- Docs: https://docs.pytest.org/en/stable/warnings.html 2026-02-23T11:34:13.542 INFO:tasks.workunit.client.0.trial030.stdout:============ 133 passed, 63 skipped, 1 warning in 814.28s (0:13:34) ============ 2026-02-23T11:34:13.596 INFO:tasks.workunit.client.0.trial030.stderr:+ exit 0 2026-02-23T11:34:13.597 INFO:teuthology.orchestra.run:Running command with timeout 3600 2026-02-23T11:34:13.597 DEBUG:teuthology.orchestra.run.trial030:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2026-02-23T11:34:13.669 INFO:tasks.workunit:Stopping ['rbd/test_librbd_python.sh'] on client.0... 2026-02-23T11:34:13.669 DEBUG:teuthology.orchestra.run.trial030:> sudo rm -rf -- /home/ubuntu/cephtest/workunits.list.client.0 /home/ubuntu/cephtest/clone.client.0 2026-02-23T11:34:14.065 DEBUG:teuthology.parallel:result is None 2026-02-23T11:34:14.065 DEBUG:teuthology.orchestra.run.trial030:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0 2026-02-23T11:34:14.091 INFO:tasks.workunit:Deleted dir /home/ubuntu/cephtest/mnt.0/client.0 2026-02-23T11:34:14.091 DEBUG:teuthology.orchestra.run.trial030:> rmdir -- /home/ubuntu/cephtest/mnt.0 2026-02-23T11:34:14.149 INFO:tasks.workunit:Deleted artificial mount point /home/ubuntu/cephtest/mnt.0/client.0 2026-02-23T11:34:14.149 DEBUG:teuthology.run_tasks:Unwinding manager ceph 2026-02-23T11:34:14.165 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2026-02-23T11:34:14.166 DEBUG:teuthology.orchestra.run.trial030:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-23T11:34:14.267 INFO:teuthology.orchestra.run.trial030.stderr:2026-02-23T11:34:14.263+0000 7f0f6d593640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T11:34:14.269 INFO:teuthology.orchestra.run.trial030.stderr:2026-02-23T11:34:14.268+0000 7f0f6d593640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T11:34:14.380 INFO:teuthology.orchestra.run.trial030.stdout: 2026-02-23T11:34:14.380 INFO:teuthology.orchestra.run.trial030.stderr:dumped all 2026-02-23T11:34:14.389 INFO:teuthology.orchestra.run.trial030.stdout:{"pg_ready":true,"pg_map":{"version":1009,"stamp":"2026-02-23T11:34:13.759969+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":53,"num_read_kb":46,"num_write":81,"num_write_kb":1108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":55,"ondisk_log_size":55,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":3,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":156499968,"kb_used":19872,"kb_used_data":19872,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":156480096,"statfs":{"total":160255967232,"available":160235618304,"internally_reserved":0,"allocated":20348928,"data_stored":20348928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-164,"num_objects":-13,"num_object_clones":0,"num_object_copies":-26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":-1,"num_read":-339685,"num_read_kb":-37685,"num_write":-8133,"num_write_kb":-41425,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":-8,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001323"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":1737,"reported_epoch":591,"state":"active+clean","last_fresh":"2026-02-23T11:34:06.542900+0000","last_change":"2026-02-23T11:20:13.041495+0000","last_active":"2026-02-23T11:34:06.542900+0000","last_peered":"2026-02-23T11:34:06.542900+0000","last_clean":"2026-02-23T11:34:06.542900+0000","last_became_active":"2026-02-23T11:20:11.064755+0000","last_became_peered":"2026-02-23T11:20:11.064755+0000","last_unstale":"2026-02-23T11:34:06.542900+0000","last_undegraded":"2026-02-23T11:34:06.542900+0000","last_fullsized":"2026-02-23T11:34:06.542900+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-23T11:20:10.015542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-23T11:20:10.015542+0000","last_clean_scrub_stamp":"2026-02-23T11:20:10.015542+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":1737,"reported_epoch":591,"state":"active+clean","last_fresh":"2026-02-23T11:34:06.548421+0000","last_change":"2026-02-23T11:20:13.043746+0000","last_active":"2026-02-23T11:34:06.548421+0000","last_peered":"2026-02-23T11:34:06.548421+0000","last_clean":"2026-02-23T11:34:06.548421+0000","last_became_active":"2026-02-23T11:20:11.063171+0000","last_became_peered":"2026-02-23T11:20:11.063171+0000","last_unstale":"2026-02-23T11:34:06.548421+0000","last_undegraded":"2026-02-23T11:34:06.548421+0000","last_fullsized":"2026-02-23T11:34:06.548421+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-23T11:20:10.015542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-23T11:20:10.015542+0000","last_clean_scrub_stamp":"2026-02-23T11:20:10.015542+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":1737,"reported_epoch":591,"state":"active+clean","last_fresh":"2026-02-23T11:34:06.545945+0000","last_change":"2026-02-23T11:20:13.041206+0000","last_active":"2026-02-23T11:34:06.545945+0000","last_peered":"2026-02-23T11:34:06.545945+0000","last_clean":"2026-02-23T11:34:06.545945+0000","last_became_active":"2026-02-23T11:20:11.070677+0000","last_became_peered":"2026-02-23T11:20:11.070677+0000","last_unstale":"2026-02-23T11:34:06.545945+0000","last_undegraded":"2026-02-23T11:34:06.545945+0000","last_fullsized":"2026-02-23T11:34:06.545945+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-23T11:20:10.015542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-23T11:20:10.015542+0000","last_clean_scrub_stamp":"2026-02-23T11:20:10.015542+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":1737,"reported_epoch":591,"state":"active+clean","last_fresh":"2026-02-23T11:34:06.547339+0000","last_change":"2026-02-23T11:20:13.043971+0000","last_active":"2026-02-23T11:34:06.547339+0000","last_peered":"2026-02-23T11:34:06.547339+0000","last_clean":"2026-02-23T11:34:06.547339+0000","last_became_active":"2026-02-23T11:20:11.074577+0000","last_became_peered":"2026-02-23T11:20:11.074577+0000","last_unstale":"2026-02-23T11:34:06.547339+0000","last_undegraded":"2026-02-23T11:34:06.547339+0000","last_fullsized":"2026-02-23T11:34:06.547339+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-23T11:20:10.015542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-23T11:20:10.015542+0000","last_clean_scrub_stamp":"2026-02-23T11:20:10.015542+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"17'2","reported_seq":1739,"reported_epoch":591,"state":"active+clean","last_fresh":"2026-02-23T11:34:06.544968+0000","last_change":"2026-02-23T11:20:13.039104+0000","last_active":"2026-02-23T11:34:06.544968+0000","last_peered":"2026-02-23T11:34:06.544968+0000","last_clean":"2026-02-23T11:34:06.544968+0000","last_became_active":"2026-02-23T11:20:11.072774+0000","last_became_peered":"2026-02-23T11:20:11.072774+0000","last_unstale":"2026-02-23T11:34:06.544968+0000","last_undegraded":"2026-02-23T11:34:06.544968+0000","last_fullsized":"2026-02-23T11:34:06.544968+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-23T11:20:10.015542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-23T11:20:10.015542+0000","last_clean_scrub_stamp":"2026-02-23T11:20:10.015542+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":1737,"reported_epoch":591,"state":"active+clean","last_fresh":"2026-02-23T11:34:06.546672+0000","last_change":"2026-02-23T11:20:13.039858+0000","last_active":"2026-02-23T11:34:06.546672+0000","last_peered":"2026-02-23T11:34:06.546672+0000","last_clean":"2026-02-23T11:34:06.546672+0000","last_became_active":"2026-02-23T11:20:11.083328+0000","last_became_peered":"2026-02-23T11:20:11.083328+0000","last_unstale":"2026-02-23T11:34:06.546672+0000","last_undegraded":"2026-02-23T11:34:06.546672+0000","last_fullsized":"2026-02-23T11:34:06.546672+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-23T11:20:10.015542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-23T11:20:10.015542+0000","last_clean_scrub_stamp":"2026-02-23T11:20:10.015542+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":1737,"reported_epoch":591,"state":"active+clean","last_fresh":"2026-02-23T11:34:06.544713+0000","last_change":"2026-02-23T11:20:13.037781+0000","last_active":"2026-02-23T11:34:06.544713+0000","last_peered":"2026-02-23T11:34:06.544713+0000","last_clean":"2026-02-23T11:34:06.544713+0000","last_became_active":"2026-02-23T11:20:11.087641+0000","last_became_peered":"2026-02-23T11:20:11.087641+0000","last_unstale":"2026-02-23T11:34:06.544713+0000","last_undegraded":"2026-02-23T11:34:06.544713+0000","last_fullsized":"2026-02-23T11:34:06.544713+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-23T11:20:10.015542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-23T11:20:10.015542+0000","last_clean_scrub_stamp":"2026-02-23T11:20:10.015542+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"15'1","reported_seq":1738,"reported_epoch":591,"state":"active+clean","last_fresh":"2026-02-23T11:34:06.546618+0000","last_change":"2026-02-23T11:20:13.040759+0000","last_active":"2026-02-23T11:34:06.546618+0000","last_peered":"2026-02-23T11:34:06.546618+0000","last_clean":"2026-02-23T11:34:06.546618+0000","last_became_active":"2026-02-23T11:20:11.060769+0000","last_became_peered":"2026-02-23T11:20:11.060769+0000","last_unstale":"2026-02-23T11:34:06.546618+0000","last_undegraded":"2026-02-23T11:34:06.546618+0000","last_fullsized":"2026-02-23T11:34:06.546618+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-23T11:20:10.015542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-23T11:20:10.015542+0000","last_clean_scrub_stamp":"2026-02-23T11:20:10.015542+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"12'52","reported_seq":1812,"reported_epoch":591,"state":"active+clean","last_fresh":"2026-02-23T11:34:06.545848+0000","last_change":"2026-02-23T11:20:07.042670+0000","last_active":"2026-02-23T11:34:06.545848+0000","last_peered":"2026-02-23T11:34:06.545848+0000","last_clean":"2026-02-23T11:34:06.545848+0000","last_became_active":"2026-02-23T11:20:07.033110+0000","last_became_peered":"2026-02-23T11:20:07.033110+0000","last_unstale":"2026-02-23T11:34:06.545848+0000","last_undegraded":"2026-02-23T11:34:06.545848+0000","last_fullsized":"2026-02-23T11:34:06.545848+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-23T11:20:06.000692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-23T11:20:06.000692+0000","last_clean_scrub_stamp":"2026-02-23T11:20:06.000692+0000","objects_scrubbed":0,"log_size":52,"log_dups_size":0,"ondisk_log_size":52,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":53,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":160255967232,"available":160235790336,"internally_reserved":0,"allocated":20176896,"data_stored":20176896,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":53,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":53418655744,"available":53411295232,"internally_reserved":0,"allocated":7360512,"data_stored":7360512,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":52,"ondisk_log_size":52,"up":2,"acting":2,"num_store_stats":1}],"osd_stats":[{"osd":2,"up_from":9,"seq":38654706508,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52166656,"kb_used":5792,"kb_used_data":5792,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":52160864,"statfs":{"total":53418655744,"available":53412724736,"internally_reserved":0,"allocated":5931008,"data_stored":5931008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":9,"seq":38654706508,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52166656,"kb_used":7240,"kb_used_data":7240,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":52159416,"statfs":{"total":53418655744,"available":53411241984,"internally_reserved":0,"allocated":7413760,"data_stored":7413760,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":10,"seq":42949673804,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52166656,"kb_used":6840,"kb_used_data":6840,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":52159816,"statfs":{"total":53418655744,"available":53411651584,"internally_reserved":0,"allocated":7004160,"data_stored":7004160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":1,"total":53418655744,"available":53411295232,"internally_reserved":0,"allocated":7360512,"data_stored":7360512,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":53418655744,"available":53411586048,"internally_reserved":0,"allocated":7069696,"data_stored":7069696,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":53418655744,"available":53411295232,"internally_reserved":0,"allocated":7360512,"data_stored":7360512,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":53418655744,"available":53412909056,"internally_reserved":0,"allocated":5746688,"data_stored":5746688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-23T11:34:14.392 DEBUG:teuthology.orchestra.run.trial030:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-23T11:34:14.468 INFO:teuthology.orchestra.run.trial030.stderr:2026-02-23T11:34:14.465+0000 7f46f990b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T11:34:14.471 INFO:teuthology.orchestra.run.trial030.stderr:2026-02-23T11:34:14.469+0000 7f46f990b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T11:34:14.580 INFO:teuthology.orchestra.run.trial030.stdout: 2026-02-23T11:34:14.580 INFO:teuthology.orchestra.run.trial030.stderr:dumped all 2026-02-23T11:34:14.589 INFO:teuthology.orchestra.run.trial030.stdout:{"pg_ready":true,"pg_map":{"version":1009,"stamp":"2026-02-23T11:34:13.759969+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":53,"num_read_kb":46,"num_write":81,"num_write_kb":1108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":55,"ondisk_log_size":55,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":3,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":156499968,"kb_used":19872,"kb_used_data":19872,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":156480096,"statfs":{"total":160255967232,"available":160235618304,"internally_reserved":0,"allocated":20348928,"data_stored":20348928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-164,"num_objects":-13,"num_object_clones":0,"num_object_copies":-26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":-1,"num_read":-339685,"num_read_kb":-37685,"num_write":-8133,"num_write_kb":-41425,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":-8,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001323"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":1737,"reported_epoch":591,"state":"active+clean","last_fresh":"2026-02-23T11:34:06.542900+0000","last_change":"2026-02-23T11:20:13.041495+0000","last_active":"2026-02-23T11:34:06.542900+0000","last_peered":"2026-02-23T11:34:06.542900+0000","last_clean":"2026-02-23T11:34:06.542900+0000","last_became_active":"2026-02-23T11:20:11.064755+0000","last_became_peered":"2026-02-23T11:20:11.064755+0000","last_unstale":"2026-02-23T11:34:06.542900+0000","last_undegraded":"2026-02-23T11:34:06.542900+0000","last_fullsized":"2026-02-23T11:34:06.542900+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-23T11:20:10.015542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-23T11:20:10.015542+0000","last_clean_scrub_stamp":"2026-02-23T11:20:10.015542+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":1737,"reported_epoch":591,"state":"active+clean","last_fresh":"2026-02-23T11:34:06.548421+0000","last_change":"2026-02-23T11:20:13.043746+0000","last_active":"2026-02-23T11:34:06.548421+0000","last_peered":"2026-02-23T11:34:06.548421+0000","last_clean":"2026-02-23T11:34:06.548421+0000","last_became_active":"2026-02-23T11:20:11.063171+0000","last_became_peered":"2026-02-23T11:20:11.063171+0000","last_unstale":"2026-02-23T11:34:06.548421+0000","last_undegraded":"2026-02-23T11:34:06.548421+0000","last_fullsized":"2026-02-23T11:34:06.548421+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-23T11:20:10.015542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-23T11:20:10.015542+0000","last_clean_scrub_stamp":"2026-02-23T11:20:10.015542+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":1737,"reported_epoch":591,"state":"active+clean","last_fresh":"2026-02-23T11:34:06.545945+0000","last_change":"2026-02-23T11:20:13.041206+0000","last_active":"2026-02-23T11:34:06.545945+0000","last_peered":"2026-02-23T11:34:06.545945+0000","last_clean":"2026-02-23T11:34:06.545945+0000","last_became_active":"2026-02-23T11:20:11.070677+0000","last_became_peered":"2026-02-23T11:20:11.070677+0000","last_unstale":"2026-02-23T11:34:06.545945+0000","last_undegraded":"2026-02-23T11:34:06.545945+0000","last_fullsized":"2026-02-23T11:34:06.545945+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-23T11:20:10.015542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-23T11:20:10.015542+0000","last_clean_scrub_stamp":"2026-02-23T11:20:10.015542+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":1737,"reported_epoch":591,"state":"active+clean","last_fresh":"2026-02-23T11:34:06.547339+0000","last_change":"2026-02-23T11:20:13.043971+0000","last_active":"2026-02-23T11:34:06.547339+0000","last_peered":"2026-02-23T11:34:06.547339+0000","last_clean":"2026-02-23T11:34:06.547339+0000","last_became_active":"2026-02-23T11:20:11.074577+0000","last_became_peered":"2026-02-23T11:20:11.074577+0000","last_unstale":"2026-02-23T11:34:06.547339+0000","last_undegraded":"2026-02-23T11:34:06.547339+0000","last_fullsized":"2026-02-23T11:34:06.547339+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-23T11:20:10.015542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-23T11:20:10.015542+0000","last_clean_scrub_stamp":"2026-02-23T11:20:10.015542+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"17'2","reported_seq":1739,"reported_epoch":591,"state":"active+clean","last_fresh":"2026-02-23T11:34:06.544968+0000","last_change":"2026-02-23T11:20:13.039104+0000","last_active":"2026-02-23T11:34:06.544968+0000","last_peered":"2026-02-23T11:34:06.544968+0000","last_clean":"2026-02-23T11:34:06.544968+0000","last_became_active":"2026-02-23T11:20:11.072774+0000","last_became_peered":"2026-02-23T11:20:11.072774+0000","last_unstale":"2026-02-23T11:34:06.544968+0000","last_undegraded":"2026-02-23T11:34:06.544968+0000","last_fullsized":"2026-02-23T11:34:06.544968+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-23T11:20:10.015542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-23T11:20:10.015542+0000","last_clean_scrub_stamp":"2026-02-23T11:20:10.015542+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":1737,"reported_epoch":591,"state":"active+clean","last_fresh":"2026-02-23T11:34:06.546672+0000","last_change":"2026-02-23T11:20:13.039858+0000","last_active":"2026-02-23T11:34:06.546672+0000","last_peered":"2026-02-23T11:34:06.546672+0000","last_clean":"2026-02-23T11:34:06.546672+0000","last_became_active":"2026-02-23T11:20:11.083328+0000","last_became_peered":"2026-02-23T11:20:11.083328+0000","last_unstale":"2026-02-23T11:34:06.546672+0000","last_undegraded":"2026-02-23T11:34:06.546672+0000","last_fullsized":"2026-02-23T11:34:06.546672+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-23T11:20:10.015542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-23T11:20:10.015542+0000","last_clean_scrub_stamp":"2026-02-23T11:20:10.015542+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":1737,"reported_epoch":591,"state":"active+clean","last_fresh":"2026-02-23T11:34:06.544713+0000","last_change":"2026-02-23T11:20:13.037781+0000","last_active":"2026-02-23T11:34:06.544713+0000","last_peered":"2026-02-23T11:34:06.544713+0000","last_clean":"2026-02-23T11:34:06.544713+0000","last_became_active":"2026-02-23T11:20:11.087641+0000","last_became_peered":"2026-02-23T11:20:11.087641+0000","last_unstale":"2026-02-23T11:34:06.544713+0000","last_undegraded":"2026-02-23T11:34:06.544713+0000","last_fullsized":"2026-02-23T11:34:06.544713+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-23T11:20:10.015542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-23T11:20:10.015542+0000","last_clean_scrub_stamp":"2026-02-23T11:20:10.015542+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"15'1","reported_seq":1738,"reported_epoch":591,"state":"active+clean","last_fresh":"2026-02-23T11:34:06.546618+0000","last_change":"2026-02-23T11:20:13.040759+0000","last_active":"2026-02-23T11:34:06.546618+0000","last_peered":"2026-02-23T11:34:06.546618+0000","last_clean":"2026-02-23T11:34:06.546618+0000","last_became_active":"2026-02-23T11:20:11.060769+0000","last_became_peered":"2026-02-23T11:20:11.060769+0000","last_unstale":"2026-02-23T11:34:06.546618+0000","last_undegraded":"2026-02-23T11:34:06.546618+0000","last_fullsized":"2026-02-23T11:34:06.546618+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-23T11:20:10.015542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-23T11:20:10.015542+0000","last_clean_scrub_stamp":"2026-02-23T11:20:10.015542+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"12'52","reported_seq":1812,"reported_epoch":591,"state":"active+clean","last_fresh":"2026-02-23T11:34:06.545848+0000","last_change":"2026-02-23T11:20:07.042670+0000","last_active":"2026-02-23T11:34:06.545848+0000","last_peered":"2026-02-23T11:34:06.545848+0000","last_clean":"2026-02-23T11:34:06.545848+0000","last_became_active":"2026-02-23T11:20:07.033110+0000","last_became_peered":"2026-02-23T11:20:07.033110+0000","last_unstale":"2026-02-23T11:34:06.545848+0000","last_undegraded":"2026-02-23T11:34:06.545848+0000","last_fullsized":"2026-02-23T11:34:06.545848+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-23T11:20:06.000692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-23T11:20:06.000692+0000","last_clean_scrub_stamp":"2026-02-23T11:20:06.000692+0000","objects_scrubbed":0,"log_size":52,"log_dups_size":0,"ondisk_log_size":52,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":53,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":160255967232,"available":160235790336,"internally_reserved":0,"allocated":20176896,"data_stored":20176896,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":53,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":53418655744,"available":53411295232,"internally_reserved":0,"allocated":7360512,"data_stored":7360512,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":52,"ondisk_log_size":52,"up":2,"acting":2,"num_store_stats":1}],"osd_stats":[{"osd":2,"up_from":9,"seq":38654706508,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52166656,"kb_used":5792,"kb_used_data":5792,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":52160864,"statfs":{"total":53418655744,"available":53412724736,"internally_reserved":0,"allocated":5931008,"data_stored":5931008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":9,"seq":38654706508,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52166656,"kb_used":7240,"kb_used_data":7240,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":52159416,"statfs":{"total":53418655744,"available":53411241984,"internally_reserved":0,"allocated":7413760,"data_stored":7413760,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":10,"seq":42949673804,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52166656,"kb_used":6840,"kb_used_data":6840,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":52159816,"statfs":{"total":53418655744,"available":53411651584,"internally_reserved":0,"allocated":7004160,"data_stored":7004160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":1,"total":53418655744,"available":53411295232,"internally_reserved":0,"allocated":7360512,"data_stored":7360512,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":53418655744,"available":53411586048,"internally_reserved":0,"allocated":7069696,"data_stored":7069696,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":53418655744,"available":53411295232,"internally_reserved":0,"allocated":7360512,"data_stored":7360512,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":53418655744,"available":53412909056,"internally_reserved":0,"allocated":5746688,"data_stored":5746688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-23T11:34:14.590 INFO:tasks.ceph.ceph_manager.ceph:clean! 2026-02-23T11:34:14.590 DEBUG:teuthology.orchestra.run.trial030:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-23T11:34:14.667 INFO:teuthology.orchestra.run.trial030.stderr:2026-02-23T11:34:14.664+0000 7fa2d9e11640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T11:34:14.670 INFO:teuthology.orchestra.run.trial030.stderr:2026-02-23T11:34:14.668+0000 7fa2d9e11640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T11:34:14.784 INFO:teuthology.orchestra.run.trial030.stdout: 2026-02-23T11:34:14.785 INFO:teuthology.orchestra.run.trial030.stderr:dumped all 2026-02-23T11:34:14.794 INFO:teuthology.orchestra.run.trial030.stdout:{"pg_ready":true,"pg_map":{"version":1009,"stamp":"2026-02-23T11:34:13.759969+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":53,"num_read_kb":46,"num_write":81,"num_write_kb":1108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":55,"ondisk_log_size":55,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":3,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":156499968,"kb_used":19872,"kb_used_data":19872,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":156480096,"statfs":{"total":160255967232,"available":160235618304,"internally_reserved":0,"allocated":20348928,"data_stored":20348928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-164,"num_objects":-13,"num_object_clones":0,"num_object_copies":-26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":-1,"num_read":-339685,"num_read_kb":-37685,"num_write":-8133,"num_write_kb":-41425,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":-8,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001323"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":1737,"reported_epoch":591,"state":"active+clean","last_fresh":"2026-02-23T11:34:06.542900+0000","last_change":"2026-02-23T11:20:13.041495+0000","last_active":"2026-02-23T11:34:06.542900+0000","last_peered":"2026-02-23T11:34:06.542900+0000","last_clean":"2026-02-23T11:34:06.542900+0000","last_became_active":"2026-02-23T11:20:11.064755+0000","last_became_peered":"2026-02-23T11:20:11.064755+0000","last_unstale":"2026-02-23T11:34:06.542900+0000","last_undegraded":"2026-02-23T11:34:06.542900+0000","last_fullsized":"2026-02-23T11:34:06.542900+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-23T11:20:10.015542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-23T11:20:10.015542+0000","last_clean_scrub_stamp":"2026-02-23T11:20:10.015542+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":1737,"reported_epoch":591,"state":"active+clean","last_fresh":"2026-02-23T11:34:06.548421+0000","last_change":"2026-02-23T11:20:13.043746+0000","last_active":"2026-02-23T11:34:06.548421+0000","last_peered":"2026-02-23T11:34:06.548421+0000","last_clean":"2026-02-23T11:34:06.548421+0000","last_became_active":"2026-02-23T11:20:11.063171+0000","last_became_peered":"2026-02-23T11:20:11.063171+0000","last_unstale":"2026-02-23T11:34:06.548421+0000","last_undegraded":"2026-02-23T11:34:06.548421+0000","last_fullsized":"2026-02-23T11:34:06.548421+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-23T11:20:10.015542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-23T11:20:10.015542+0000","last_clean_scrub_stamp":"2026-02-23T11:20:10.015542+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":1737,"reported_epoch":591,"state":"active+clean","last_fresh":"2026-02-23T11:34:06.545945+0000","last_change":"2026-02-23T11:20:13.041206+0000","last_active":"2026-02-23T11:34:06.545945+0000","last_peered":"2026-02-23T11:34:06.545945+0000","last_clean":"2026-02-23T11:34:06.545945+0000","last_became_active":"2026-02-23T11:20:11.070677+0000","last_became_peered":"2026-02-23T11:20:11.070677+0000","last_unstale":"2026-02-23T11:34:06.545945+0000","last_undegraded":"2026-02-23T11:34:06.545945+0000","last_fullsized":"2026-02-23T11:34:06.545945+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-23T11:20:10.015542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-23T11:20:10.015542+0000","last_clean_scrub_stamp":"2026-02-23T11:20:10.015542+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":1737,"reported_epoch":591,"state":"active+clean","last_fresh":"2026-02-23T11:34:06.547339+0000","last_change":"2026-02-23T11:20:13.043971+0000","last_active":"2026-02-23T11:34:06.547339+0000","last_peered":"2026-02-23T11:34:06.547339+0000","last_clean":"2026-02-23T11:34:06.547339+0000","last_became_active":"2026-02-23T11:20:11.074577+0000","last_became_peered":"2026-02-23T11:20:11.074577+0000","last_unstale":"2026-02-23T11:34:06.547339+0000","last_undegraded":"2026-02-23T11:34:06.547339+0000","last_fullsized":"2026-02-23T11:34:06.547339+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-23T11:20:10.015542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-23T11:20:10.015542+0000","last_clean_scrub_stamp":"2026-02-23T11:20:10.015542+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"17'2","reported_seq":1739,"reported_epoch":591,"state":"active+clean","last_fresh":"2026-02-23T11:34:06.544968+0000","last_change":"2026-02-23T11:20:13.039104+0000","last_active":"2026-02-23T11:34:06.544968+0000","last_peered":"2026-02-23T11:34:06.544968+0000","last_clean":"2026-02-23T11:34:06.544968+0000","last_became_active":"2026-02-23T11:20:11.072774+0000","last_became_peered":"2026-02-23T11:20:11.072774+0000","last_unstale":"2026-02-23T11:34:06.544968+0000","last_undegraded":"2026-02-23T11:34:06.544968+0000","last_fullsized":"2026-02-23T11:34:06.544968+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-23T11:20:10.015542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-23T11:20:10.015542+0000","last_clean_scrub_stamp":"2026-02-23T11:20:10.015542+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":1737,"reported_epoch":591,"state":"active+clean","last_fresh":"2026-02-23T11:34:06.546672+0000","last_change":"2026-02-23T11:20:13.039858+0000","last_active":"2026-02-23T11:34:06.546672+0000","last_peered":"2026-02-23T11:34:06.546672+0000","last_clean":"2026-02-23T11:34:06.546672+0000","last_became_active":"2026-02-23T11:20:11.083328+0000","last_became_peered":"2026-02-23T11:20:11.083328+0000","last_unstale":"2026-02-23T11:34:06.546672+0000","last_undegraded":"2026-02-23T11:34:06.546672+0000","last_fullsized":"2026-02-23T11:34:06.546672+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-23T11:20:10.015542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-23T11:20:10.015542+0000","last_clean_scrub_stamp":"2026-02-23T11:20:10.015542+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":1737,"reported_epoch":591,"state":"active+clean","last_fresh":"2026-02-23T11:34:06.544713+0000","last_change":"2026-02-23T11:20:13.037781+0000","last_active":"2026-02-23T11:34:06.544713+0000","last_peered":"2026-02-23T11:34:06.544713+0000","last_clean":"2026-02-23T11:34:06.544713+0000","last_became_active":"2026-02-23T11:20:11.087641+0000","last_became_peered":"2026-02-23T11:20:11.087641+0000","last_unstale":"2026-02-23T11:34:06.544713+0000","last_undegraded":"2026-02-23T11:34:06.544713+0000","last_fullsized":"2026-02-23T11:34:06.544713+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-23T11:20:10.015542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-23T11:20:10.015542+0000","last_clean_scrub_stamp":"2026-02-23T11:20:10.015542+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"15'1","reported_seq":1738,"reported_epoch":591,"state":"active+clean","last_fresh":"2026-02-23T11:34:06.546618+0000","last_change":"2026-02-23T11:20:13.040759+0000","last_active":"2026-02-23T11:34:06.546618+0000","last_peered":"2026-02-23T11:34:06.546618+0000","last_clean":"2026-02-23T11:34:06.546618+0000","last_became_active":"2026-02-23T11:20:11.060769+0000","last_became_peered":"2026-02-23T11:20:11.060769+0000","last_unstale":"2026-02-23T11:34:06.546618+0000","last_undegraded":"2026-02-23T11:34:06.546618+0000","last_fullsized":"2026-02-23T11:34:06.546618+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-23T11:20:10.015542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-23T11:20:10.015542+0000","last_clean_scrub_stamp":"2026-02-23T11:20:10.015542+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"12'52","reported_seq":1812,"reported_epoch":591,"state":"active+clean","last_fresh":"2026-02-23T11:34:06.545848+0000","last_change":"2026-02-23T11:20:07.042670+0000","last_active":"2026-02-23T11:34:06.545848+0000","last_peered":"2026-02-23T11:34:06.545848+0000","last_clean":"2026-02-23T11:34:06.545848+0000","last_became_active":"2026-02-23T11:20:07.033110+0000","last_became_peered":"2026-02-23T11:20:07.033110+0000","last_unstale":"2026-02-23T11:34:06.545848+0000","last_undegraded":"2026-02-23T11:34:06.545848+0000","last_fullsized":"2026-02-23T11:34:06.545848+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-23T11:20:06.000692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-23T11:20:06.000692+0000","last_clean_scrub_stamp":"2026-02-23T11:20:06.000692+0000","objects_scrubbed":0,"log_size":52,"log_dups_size":0,"ondisk_log_size":52,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":53,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":160255967232,"available":160235790336,"internally_reserved":0,"allocated":20176896,"data_stored":20176896,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":53,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":53418655744,"available":53411295232,"internally_reserved":0,"allocated":7360512,"data_stored":7360512,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":52,"ondisk_log_size":52,"up":2,"acting":2,"num_store_stats":1}],"osd_stats":[{"osd":2,"up_from":9,"seq":38654706508,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52166656,"kb_used":5792,"kb_used_data":5792,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":52160864,"statfs":{"total":53418655744,"available":53412724736,"internally_reserved":0,"allocated":5931008,"data_stored":5931008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":9,"seq":38654706508,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52166656,"kb_used":7240,"kb_used_data":7240,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":52159416,"statfs":{"total":53418655744,"available":53411241984,"internally_reserved":0,"allocated":7413760,"data_stored":7413760,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":10,"seq":42949673804,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52166656,"kb_used":6840,"kb_used_data":6840,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":52159816,"statfs":{"total":53418655744,"available":53411651584,"internally_reserved":0,"allocated":7004160,"data_stored":7004160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":1,"total":53418655744,"available":53411295232,"internally_reserved":0,"allocated":7360512,"data_stored":7360512,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":53418655744,"available":53411586048,"internally_reserved":0,"allocated":7069696,"data_stored":7069696,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":53418655744,"available":53411295232,"internally_reserved":0,"allocated":7360512,"data_stored":7360512,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":53418655744,"available":53412909056,"internally_reserved":0,"allocated":5746688,"data_stored":5746688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-23T11:34:14.796 DEBUG:teuthology.orchestra.run.trial030:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2026-02-23T11:34:14.873 INFO:teuthology.orchestra.run.trial030.stderr:2026-02-23T11:34:14.869+0000 7f9950a5a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T11:34:14.875 INFO:teuthology.orchestra.run.trial030.stderr:2026-02-23T11:34:14.873+0000 7f9950a5a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T11:34:14.986 INFO:teuthology.orchestra.run.trial030.stdout: 2026-02-23T11:34:14.986 INFO:teuthology.orchestra.run.trial030.stdout:{"epoch":592,"fsid":"4c6fca00-8e92-4876-9bd8-e5358ac35226","created":"2026-02-23T11:19:53.347742+0000","modified":"2026-02-23T11:34:13.532967+0000","last_up_change":"2026-02-23T11:20:04.995539+0000","last_in_change":"2026-02-23T11:19:55.105654+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T11:20:05.642161+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"13","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.9900000095367432,"score_stable":2.9900000095367432,"optimal_score":0.67000001668930054,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2026-02-23T11:20:09.105686+0000","flags":270337,"flags_names":"hashpspool,selfmanaged_snaps,crimson","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":2,"snap_epoch":17,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"9b3d3dff-502d-46ad-bb23-c485ba897402","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":10,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.30:6806","nonce":298998803}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.30:6807","nonce":298998803}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.30:6813","nonce":298998803}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.30:6812","nonce":298998803}]},"public_addr":"10.20.193.30:6806/298998803","cluster_addr":"10.20.193.30:6807/298998803","heartbeat_back_addr":"10.20.193.30:6813/298998803","heartbeat_front_addr":"10.20.193.30:6812/298998803","state":["exists","up"]},{"osd":1,"uuid":"5c421ec3-7768-46b0-ace6-d1b2ffc3a840","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.30:6804","nonce":947989549}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.30:6805","nonce":947989549}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.30:6811","nonce":947989549}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.30:6809","nonce":947989549}]},"public_addr":"10.20.193.30:6804/947989549","cluster_addr":"10.20.193.30:6805/947989549","heartbeat_back_addr":"10.20.193.30:6811/947989549","heartbeat_front_addr":"10.20.193.30:6809/947989549","state":["exists","up"]},{"osd":2,"uuid":"cf71465d-2f79-4740-88c8-000366302513","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.30:6803","nonce":1688863716}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.30:6802","nonce":1688863716}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.30:6808","nonce":1688863716}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.30:6810","nonce":1688863716}]},"public_addr":"10.20.193.30:6803/1688863716","cluster_addr":"10.20.193.30:6802/1688863716","heartbeat_back_addr":"10.20.193.30:6808/1688863716","heartbeat_front_addr":"10.20.193.30:6810/1688863716","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":2,"snaps":[{"begin":2,"length":1}]},{"pool":3,"snaps":[{"begin":2,"length":302}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T11:34:15.997 INFO:tasks.ceph:Scrubbing osd.0 2026-02-23T11:34:15.997 DEBUG:teuthology.orchestra.run.trial030:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 config set osd_debug_deep_scrub_sleep 0 2026-02-23T11:34:16.077 INFO:teuthology.orchestra.run.trial030.stderr:2026-02-23T11:34:16.071+0000 7f3b88393640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T11:34:16.079 INFO:teuthology.orchestra.run.trial030.stderr:2026-02-23T11:34:16.078+0000 7f3b88393640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T11:34:16.089 INFO:teuthology.orchestra.run.trial030.stdout:{ 2026-02-23T11:34:16.089 INFO:teuthology.orchestra.run.trial030.stdout: "success": "" 2026-02-23T11:34:16.089 INFO:teuthology.orchestra.run.trial030.stdout:} 2026-02-23T11:34:16.097 DEBUG:teuthology.orchestra.run.trial030:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 0 2026-02-23T11:34:16.174 INFO:teuthology.orchestra.run.trial030.stderr:2026-02-23T11:34:16.170+0000 7f7bc365a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T11:34:16.176 INFO:teuthology.orchestra.run.trial030.stderr:2026-02-23T11:34:16.175+0000 7f7bc365a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T11:34:16.290 INFO:teuthology.orchestra.run.trial030.stderr:instructed osd(s) 0 to deep-scrub 2026-02-23T11:34:16.301 INFO:tasks.ceph:Scrubbing osd.1 2026-02-23T11:34:16.302 DEBUG:teuthology.orchestra.run.trial030:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 config set osd_debug_deep_scrub_sleep 0 2026-02-23T11:34:16.381 INFO:teuthology.orchestra.run.trial030.stderr:2026-02-23T11:34:16.375+0000 7f064d1d3640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T11:34:16.383 INFO:teuthology.orchestra.run.trial030.stderr:2026-02-23T11:34:16.382+0000 7f064d1d3640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T11:34:16.394 INFO:teuthology.orchestra.run.trial030.stdout:{ 2026-02-23T11:34:16.394 INFO:teuthology.orchestra.run.trial030.stdout: "success": "" 2026-02-23T11:34:16.394 INFO:teuthology.orchestra.run.trial030.stdout:} 2026-02-23T11:34:16.401 DEBUG:teuthology.orchestra.run.trial030:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 1 2026-02-23T11:34:16.479 INFO:teuthology.orchestra.run.trial030.stderr:2026-02-23T11:34:16.475+0000 7fbcdcd74640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T11:34:16.482 INFO:teuthology.orchestra.run.trial030.stderr:2026-02-23T11:34:16.480+0000 7fbcdcd74640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T11:34:16.593 INFO:teuthology.orchestra.run.trial030.stderr:instructed osd(s) 1 to deep-scrub 2026-02-23T11:34:16.606 INFO:tasks.ceph:Scrubbing osd.2 2026-02-23T11:34:16.606 DEBUG:teuthology.orchestra.run.trial030:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 config set osd_debug_deep_scrub_sleep 0 2026-02-23T11:34:16.686 INFO:teuthology.orchestra.run.trial030.stderr:2026-02-23T11:34:16.679+0000 7f90f459c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T11:34:16.688 INFO:teuthology.orchestra.run.trial030.stderr:2026-02-23T11:34:16.686+0000 7f90f459c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T11:34:16.697 INFO:teuthology.orchestra.run.trial030.stdout:{ 2026-02-23T11:34:16.697 INFO:teuthology.orchestra.run.trial030.stdout: "success": "" 2026-02-23T11:34:16.697 INFO:teuthology.orchestra.run.trial030.stdout:} 2026-02-23T11:34:16.704 DEBUG:teuthology.orchestra.run.trial030:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 2 2026-02-23T11:34:16.782 INFO:teuthology.orchestra.run.trial030.stderr:2026-02-23T11:34:16.777+0000 7fb2f29f5640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T11:34:16.784 INFO:teuthology.orchestra.run.trial030.stderr:2026-02-23T11:34:16.782+0000 7fb2f29f5640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T11:34:16.895 INFO:teuthology.orchestra.run.trial030.stderr:instructed osd(s) 2 to deep-scrub 2026-02-23T11:34:16.907 DEBUG:teuthology.orchestra.run.trial030:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-23T11:34:16.983 INFO:teuthology.orchestra.run.trial030.stderr:2026-02-23T11:34:16.980+0000 7f0adc90b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T11:34:16.985 INFO:teuthology.orchestra.run.trial030.stderr:2026-02-23T11:34:16.984+0000 7f0adc90b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T11:34:17.095 INFO:teuthology.orchestra.run.trial030.stdout: 2026-02-23T11:34:17.096 INFO:teuthology.orchestra.run.trial030.stderr:dumped all 2026-02-23T11:34:17.104 INFO:teuthology.orchestra.run.trial030.stdout:{"pg_ready":true,"pg_map":{"version":1010,"stamp":"2026-02-23T11:34:15.760245+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":53,"num_read_kb":46,"num_write":81,"num_write_kb":1108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":55,"ondisk_log_size":55,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":3,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":156499968,"kb_used":15084,"kb_used_data":15084,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":156484884,"statfs":{"total":160255967232,"available":160240521216,"internally_reserved":0,"allocated":15446016,"data_stored":15446016,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-164,"num_objects":-13,"num_object_clones":0,"num_object_copies":-26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":-1,"num_read":-339685,"num_read_kb":-37685,"num_write":-8133,"num_write_kb":-41425,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":-8,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"9.230335"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":1740,"reported_epoch":592,"state":"active+clean","last_fresh":"2026-02-23T11:34:13.562046+0000","last_change":"2026-02-23T11:20:13.041495+0000","last_active":"2026-02-23T11:34:13.562046+0000","last_peered":"2026-02-23T11:34:13.562046+0000","last_clean":"2026-02-23T11:34:13.562046+0000","last_became_active":"2026-02-23T11:20:11.064755+0000","last_became_peered":"2026-02-23T11:20:11.064755+0000","last_unstale":"2026-02-23T11:34:13.562046+0000","last_undegraded":"2026-02-23T11:34:13.562046+0000","last_fullsized":"2026-02-23T11:34:13.562046+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-23T11:20:10.015542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-23T11:20:10.015542+0000","last_clean_scrub_stamp":"2026-02-23T11:20:10.015542+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":1740,"reported_epoch":592,"state":"active+clean","last_fresh":"2026-02-23T11:34:13.560376+0000","last_change":"2026-02-23T11:20:13.043746+0000","last_active":"2026-02-23T11:34:13.560376+0000","last_peered":"2026-02-23T11:34:13.560376+0000","last_clean":"2026-02-23T11:34:13.560376+0000","last_became_active":"2026-02-23T11:20:11.063171+0000","last_became_peered":"2026-02-23T11:20:11.063171+0000","last_unstale":"2026-02-23T11:34:13.560376+0000","last_undegraded":"2026-02-23T11:34:13.560376+0000","last_fullsized":"2026-02-23T11:34:13.560376+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-23T11:20:10.015542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-23T11:20:10.015542+0000","last_clean_scrub_stamp":"2026-02-23T11:20:10.015542+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":1740,"reported_epoch":592,"state":"active+clean","last_fresh":"2026-02-23T11:34:13.563543+0000","last_change":"2026-02-23T11:20:13.041206+0000","last_active":"2026-02-23T11:34:13.563543+0000","last_peered":"2026-02-23T11:34:13.563543+0000","last_clean":"2026-02-23T11:34:13.563543+0000","last_became_active":"2026-02-23T11:20:11.070677+0000","last_became_peered":"2026-02-23T11:20:11.070677+0000","last_unstale":"2026-02-23T11:34:13.563543+0000","last_undegraded":"2026-02-23T11:34:13.563543+0000","last_fullsized":"2026-02-23T11:34:13.563543+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-23T11:20:10.015542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-23T11:20:10.015542+0000","last_clean_scrub_stamp":"2026-02-23T11:20:10.015542+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":1740,"reported_epoch":592,"state":"active+clean","last_fresh":"2026-02-23T11:34:13.563365+0000","last_change":"2026-02-23T11:20:13.043971+0000","last_active":"2026-02-23T11:34:13.563365+0000","last_peered":"2026-02-23T11:34:13.563365+0000","last_clean":"2026-02-23T11:34:13.563365+0000","last_became_active":"2026-02-23T11:20:11.074577+0000","last_became_peered":"2026-02-23T11:20:11.074577+0000","last_unstale":"2026-02-23T11:34:13.563365+0000","last_undegraded":"2026-02-23T11:34:13.563365+0000","last_fullsized":"2026-02-23T11:34:13.563365+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-23T11:20:10.015542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-23T11:20:10.015542+0000","last_clean_scrub_stamp":"2026-02-23T11:20:10.015542+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"17'2","reported_seq":1742,"reported_epoch":592,"state":"active+clean","last_fresh":"2026-02-23T11:34:13.557831+0000","last_change":"2026-02-23T11:20:13.039104+0000","last_active":"2026-02-23T11:34:13.557831+0000","last_peered":"2026-02-23T11:34:13.557831+0000","last_clean":"2026-02-23T11:34:13.557831+0000","last_became_active":"2026-02-23T11:20:11.072774+0000","last_became_peered":"2026-02-23T11:20:11.072774+0000","last_unstale":"2026-02-23T11:34:13.557831+0000","last_undegraded":"2026-02-23T11:34:13.557831+0000","last_fullsized":"2026-02-23T11:34:13.557831+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-23T11:20:10.015542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-23T11:20:10.015542+0000","last_clean_scrub_stamp":"2026-02-23T11:20:10.015542+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":1740,"reported_epoch":592,"state":"active+clean","last_fresh":"2026-02-23T11:34:13.562458+0000","last_change":"2026-02-23T11:20:13.039858+0000","last_active":"2026-02-23T11:34:13.562458+0000","last_peered":"2026-02-23T11:34:13.562458+0000","last_clean":"2026-02-23T11:34:13.562458+0000","last_became_active":"2026-02-23T11:20:11.083328+0000","last_became_peered":"2026-02-23T11:20:11.083328+0000","last_unstale":"2026-02-23T11:34:13.562458+0000","last_undegraded":"2026-02-23T11:34:13.562458+0000","last_fullsized":"2026-02-23T11:34:13.562458+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-23T11:20:10.015542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-23T11:20:10.015542+0000","last_clean_scrub_stamp":"2026-02-23T11:20:10.015542+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":1740,"reported_epoch":592,"state":"active+clean","last_fresh":"2026-02-23T11:34:13.560580+0000","last_change":"2026-02-23T11:20:13.037781+0000","last_active":"2026-02-23T11:34:13.560580+0000","last_peered":"2026-02-23T11:34:13.560580+0000","last_clean":"2026-02-23T11:34:13.560580+0000","last_became_active":"2026-02-23T11:20:11.087641+0000","last_became_peered":"2026-02-23T11:20:11.087641+0000","last_unstale":"2026-02-23T11:34:13.560580+0000","last_undegraded":"2026-02-23T11:34:13.560580+0000","last_fullsized":"2026-02-23T11:34:13.560580+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-23T11:20:10.015542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-23T11:20:10.015542+0000","last_clean_scrub_stamp":"2026-02-23T11:20:10.015542+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"15'1","reported_seq":1741,"reported_epoch":592,"state":"active+clean","last_fresh":"2026-02-23T11:34:13.563108+0000","last_change":"2026-02-23T11:20:13.040759+0000","last_active":"2026-02-23T11:34:13.563108+0000","last_peered":"2026-02-23T11:34:13.563108+0000","last_clean":"2026-02-23T11:34:13.563108+0000","last_became_active":"2026-02-23T11:20:11.060769+0000","last_became_peered":"2026-02-23T11:20:11.060769+0000","last_unstale":"2026-02-23T11:34:13.563108+0000","last_undegraded":"2026-02-23T11:34:13.563108+0000","last_fullsized":"2026-02-23T11:34:13.563108+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-23T11:20:10.015542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-23T11:20:10.015542+0000","last_clean_scrub_stamp":"2026-02-23T11:20:10.015542+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"12'52","reported_seq":1815,"reported_epoch":592,"state":"active+clean","last_fresh":"2026-02-23T11:34:13.557764+0000","last_change":"2026-02-23T11:20:07.042670+0000","last_active":"2026-02-23T11:34:13.557764+0000","last_peered":"2026-02-23T11:34:13.557764+0000","last_clean":"2026-02-23T11:34:13.557764+0000","last_became_active":"2026-02-23T11:20:07.033110+0000","last_became_peered":"2026-02-23T11:20:07.033110+0000","last_unstale":"2026-02-23T11:34:13.557764+0000","last_undegraded":"2026-02-23T11:34:13.557764+0000","last_fullsized":"2026-02-23T11:34:13.557764+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-23T11:20:06.000692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-23T11:20:06.000692+0000","last_clean_scrub_stamp":"2026-02-23T11:20:06.000692+0000","objects_scrubbed":0,"log_size":52,"log_dups_size":0,"ondisk_log_size":52,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":53,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":160255967232,"available":160248803328,"internally_reserved":0,"allocated":7163904,"data_stored":7163904,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":53,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":53418655744,"available":53415796736,"internally_reserved":0,"allocated":2859008,"data_stored":2859008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":52,"ondisk_log_size":52,"up":2,"acting":2,"num_store_stats":1}],"osd_stats":[{"osd":2,"up_from":9,"seq":38654706513,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52166656,"kb_used":5480,"kb_used_data":5480,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":52161176,"statfs":{"total":53418655744,"available":53413044224,"internally_reserved":0,"allocated":5611520,"data_stored":5611520,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":9,"seq":38654706513,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52166656,"kb_used":7108,"kb_used_data":7108,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":52159548,"statfs":{"total":53418655744,"available":53411377152,"internally_reserved":0,"allocated":7278592,"data_stored":7278592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":10,"seq":42949673809,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52166656,"kb_used":2496,"kb_used_data":2496,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":52164160,"statfs":{"total":53418655744,"available":53416099840,"internally_reserved":0,"allocated":2555904,"data_stored":2555904,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":1,"total":53418655744,"available":53415796736,"internally_reserved":0,"allocated":2859008,"data_stored":2859008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":53418655744,"available":53416099840,"internally_reserved":0,"allocated":2555904,"data_stored":2555904,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":53418655744,"available":53415796736,"internally_reserved":0,"allocated":2859008,"data_stored":2859008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":53418655744,"available":53416906752,"internally_reserved":0,"allocated":1748992,"data_stored":1748992,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-23T11:34:17.109 INFO:tasks.ceph:pgid 2.7 last_scrub_stamp 2026-02-23T11:20:10.015542+0000 time.struct_time(tm_year=2026, tm_mon=2, tm_mday=23, tm_hour=11, tm_min=20, tm_sec=10, tm_wday=0, tm_yday=54, tm_isdst=-1) <= time.struct_time(tm_year=2026, tm_mon=2, tm_mday=23, tm_hour=11, tm_min=34, tm_sec=14, tm_wday=0, tm_yday=54, tm_isdst=0) 2026-02-23T11:34:17.109 INFO:tasks.ceph:pgid 2.6 last_scrub_stamp 2026-02-23T11:20:10.015542+0000 time.struct_time(tm_year=2026, tm_mon=2, tm_mday=23, tm_hour=11, tm_min=20, tm_sec=10, tm_wday=0, tm_yday=54, tm_isdst=-1) <= time.struct_time(tm_year=2026, tm_mon=2, tm_mday=23, tm_hour=11, tm_min=34, tm_sec=14, tm_wday=0, tm_yday=54, tm_isdst=0) 2026-02-23T11:34:17.109 INFO:tasks.ceph:pgid 2.5 last_scrub_stamp 2026-02-23T11:20:10.015542+0000 time.struct_time(tm_year=2026, tm_mon=2, tm_mday=23, tm_hour=11, tm_min=20, tm_sec=10, tm_wday=0, tm_yday=54, tm_isdst=-1) <= time.struct_time(tm_year=2026, tm_mon=2, tm_mday=23, tm_hour=11, tm_min=34, tm_sec=14, tm_wday=0, tm_yday=54, tm_isdst=0) 2026-02-23T11:34:17.109 INFO:tasks.ceph:pgid 2.4 last_scrub_stamp 2026-02-23T11:20:10.015542+0000 time.struct_time(tm_year=2026, tm_mon=2, tm_mday=23, tm_hour=11, tm_min=20, tm_sec=10, tm_wday=0, tm_yday=54, tm_isdst=-1) <= time.struct_time(tm_year=2026, tm_mon=2, tm_mday=23, tm_hour=11, tm_min=34, tm_sec=14, tm_wday=0, tm_yday=54, tm_isdst=0) 2026-02-23T11:34:17.110 INFO:tasks.ceph:pgid 2.2 last_scrub_stamp 2026-02-23T11:20:10.015542+0000 time.struct_time(tm_year=2026, tm_mon=2, tm_mday=23, tm_hour=11, tm_min=20, tm_sec=10, tm_wday=0, tm_yday=54, tm_isdst=-1) <= time.struct_time(tm_year=2026, tm_mon=2, tm_mday=23, tm_hour=11, tm_min=34, tm_sec=14, tm_wday=0, tm_yday=54, tm_isdst=0) 2026-02-23T11:34:17.110 INFO:tasks.ceph:pgid 2.1 last_scrub_stamp 2026-02-23T11:20:10.015542+0000 time.struct_time(tm_year=2026, tm_mon=2, tm_mday=23, tm_hour=11, tm_min=20, tm_sec=10, tm_wday=0, tm_yday=54, tm_isdst=-1) <= time.struct_time(tm_year=2026, tm_mon=2, tm_mday=23, tm_hour=11, tm_min=34, tm_sec=14, tm_wday=0, tm_yday=54, tm_isdst=0) 2026-02-23T11:34:17.110 INFO:tasks.ceph:pgid 2.0 last_scrub_stamp 2026-02-23T11:20:10.015542+0000 time.struct_time(tm_year=2026, tm_mon=2, tm_mday=23, tm_hour=11, tm_min=20, tm_sec=10, tm_wday=0, tm_yday=54, tm_isdst=-1) <= time.struct_time(tm_year=2026, tm_mon=2, tm_mday=23, tm_hour=11, tm_min=34, tm_sec=14, tm_wday=0, tm_yday=54, tm_isdst=0) 2026-02-23T11:34:17.110 INFO:tasks.ceph:pgid 2.3 last_scrub_stamp 2026-02-23T11:20:10.015542+0000 time.struct_time(tm_year=2026, tm_mon=2, tm_mday=23, tm_hour=11, tm_min=20, tm_sec=10, tm_wday=0, tm_yday=54, tm_isdst=-1) <= time.struct_time(tm_year=2026, tm_mon=2, tm_mday=23, tm_hour=11, tm_min=34, tm_sec=14, tm_wday=0, tm_yday=54, tm_isdst=0) 2026-02-23T11:34:17.110 INFO:tasks.ceph:pgid 1.0 last_scrub_stamp 2026-02-23T11:20:06.000692+0000 time.struct_time(tm_year=2026, tm_mon=2, tm_mday=23, tm_hour=11, tm_min=20, tm_sec=6, tm_wday=0, tm_yday=54, tm_isdst=-1) <= time.struct_time(tm_year=2026, tm_mon=2, tm_mday=23, tm_hour=11, tm_min=34, tm_sec=14, tm_wday=0, tm_yday=54, tm_isdst=0) 2026-02-23T11:34:17.110 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2026-02-23T11:34:37.111 DEBUG:teuthology.orchestra.run.trial030:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-23T11:34:37.191 INFO:teuthology.orchestra.run.trial030.stderr:2026-02-23T11:34:37.187+0000 7f3d7eacb640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T11:34:37.194 INFO:teuthology.orchestra.run.trial030.stderr:2026-02-23T11:34:37.192+0000 7f3d7eacb640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T11:34:37.305 INFO:teuthology.orchestra.run.trial030.stdout: 2026-02-23T11:34:37.305 INFO:teuthology.orchestra.run.trial030.stderr:dumped all 2026-02-23T11:34:37.315 INFO:teuthology.orchestra.run.trial030.stdout:{"pg_ready":true,"pg_map":{"version":1021,"stamp":"2026-02-23T11:34:35.762941+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":53,"num_read_kb":46,"num_write":81,"num_write_kb":1108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":55,"ondisk_log_size":55,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":3,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":156499968,"kb_used":6780,"kb_used_data":6780,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":156493188,"statfs":{"total":160255967232,"available":160249024512,"internally_reserved":0,"allocated":6942720,"data_stored":6942720,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001667"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":1745,"reported_epoch":593,"state":"active+clean","last_fresh":"2026-02-23T11:34:18.492563+0000","last_change":"2026-02-23T11:34:16.608910+0000","last_active":"2026-02-23T11:34:18.492563+0000","last_peered":"2026-02-23T11:34:18.492563+0000","last_clean":"2026-02-23T11:34:18.492563+0000","last_became_active":"2026-02-23T11:20:11.064755+0000","last_became_peered":"2026-02-23T11:20:11.064755+0000","last_unstale":"2026-02-23T11:34:18.492563+0000","last_undegraded":"2026-02-23T11:34:18.492563+0000","last_fullsized":"2026-02-23T11:34:18.492563+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-23T11:34:16.608854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-23T11:34:16.608854+0000","last_clean_scrub_stamp":"2026-02-23T11:20:10.015542+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":1745,"reported_epoch":593,"state":"active+clean","last_fresh":"2026-02-23T11:34:18.494190+0000","last_change":"2026-02-23T11:34:16.608181+0000","last_active":"2026-02-23T11:34:18.494190+0000","last_peered":"2026-02-23T11:34:18.494190+0000","last_clean":"2026-02-23T11:34:18.494190+0000","last_became_active":"2026-02-23T11:20:11.063171+0000","last_became_peered":"2026-02-23T11:20:11.063171+0000","last_unstale":"2026-02-23T11:34:18.494190+0000","last_undegraded":"2026-02-23T11:34:18.494190+0000","last_fullsized":"2026-02-23T11:34:18.494190+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-23T11:34:16.608126+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-23T11:34:16.608126+0000","last_clean_scrub_stamp":"2026-02-23T11:20:10.015542+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":1745,"reported_epoch":593,"state":"active+clean","last_fresh":"2026-02-23T11:34:18.493836+0000","last_change":"2026-02-23T11:34:16.604295+0000","last_active":"2026-02-23T11:34:18.493836+0000","last_peered":"2026-02-23T11:34:18.493836+0000","last_clean":"2026-02-23T11:34:18.493836+0000","last_became_active":"2026-02-23T11:20:11.070677+0000","last_became_peered":"2026-02-23T11:20:11.070677+0000","last_unstale":"2026-02-23T11:34:18.493836+0000","last_undegraded":"2026-02-23T11:34:18.493836+0000","last_fullsized":"2026-02-23T11:34:18.493836+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-23T11:34:16.604219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-23T11:34:16.604219+0000","last_clean_scrub_stamp":"2026-02-23T11:20:10.015542+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":1745,"reported_epoch":593,"state":"active+clean","last_fresh":"2026-02-23T11:34:18.494570+0000","last_change":"2026-02-23T11:34:16.607253+0000","last_active":"2026-02-23T11:34:18.494570+0000","last_peered":"2026-02-23T11:34:18.494570+0000","last_clean":"2026-02-23T11:34:18.494570+0000","last_became_active":"2026-02-23T11:20:11.074577+0000","last_became_peered":"2026-02-23T11:20:11.074577+0000","last_unstale":"2026-02-23T11:34:18.494570+0000","last_undegraded":"2026-02-23T11:34:18.494570+0000","last_fullsized":"2026-02-23T11:34:18.494570+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-23T11:34:16.607196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-23T11:34:16.607196+0000","last_clean_scrub_stamp":"2026-02-23T11:20:10.015542+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"17'2","reported_seq":1747,"reported_epoch":593,"state":"active+clean","last_fresh":"2026-02-23T11:34:18.493779+0000","last_change":"2026-02-23T11:34:16.301362+0000","last_active":"2026-02-23T11:34:18.493779+0000","last_peered":"2026-02-23T11:34:18.493779+0000","last_clean":"2026-02-23T11:34:18.493779+0000","last_became_active":"2026-02-23T11:20:11.072774+0000","last_became_peered":"2026-02-23T11:20:11.072774+0000","last_unstale":"2026-02-23T11:34:18.493779+0000","last_undegraded":"2026-02-23T11:34:18.493779+0000","last_fullsized":"2026-02-23T11:34:18.493779+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"17'2","last_scrub_stamp":"2026-02-23T11:34:16.301262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-23T11:34:16.301262+0000","last_clean_scrub_stamp":"2026-02-23T11:20:10.015542+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":1745,"reported_epoch":593,"state":"active+clean","last_fresh":"2026-02-23T11:34:18.493260+0000","last_change":"2026-02-23T11:34:16.902895+0000","last_active":"2026-02-23T11:34:18.493260+0000","last_peered":"2026-02-23T11:34:18.493260+0000","last_clean":"2026-02-23T11:34:18.493260+0000","last_became_active":"2026-02-23T11:20:11.083328+0000","last_became_peered":"2026-02-23T11:20:11.083328+0000","last_unstale":"2026-02-23T11:34:18.493260+0000","last_undegraded":"2026-02-23T11:34:18.493260+0000","last_fullsized":"2026-02-23T11:34:18.493260+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-23T11:34:16.902810+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-23T11:34:16.902810+0000","last_clean_scrub_stamp":"2026-02-23T11:20:10.015542+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":1745,"reported_epoch":593,"state":"active+clean","last_fresh":"2026-02-23T11:34:18.492344+0000","last_change":"2026-02-23T11:34:16.903777+0000","last_active":"2026-02-23T11:34:18.492344+0000","last_peered":"2026-02-23T11:34:18.492344+0000","last_clean":"2026-02-23T11:34:18.492344+0000","last_became_active":"2026-02-23T11:20:11.087641+0000","last_became_peered":"2026-02-23T11:20:11.087641+0000","last_unstale":"2026-02-23T11:34:18.492344+0000","last_undegraded":"2026-02-23T11:34:18.492344+0000","last_fullsized":"2026-02-23T11:34:18.492344+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-23T11:34:16.903714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-23T11:34:16.903714+0000","last_clean_scrub_stamp":"2026-02-23T11:20:10.015542+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"15'1","reported_seq":1746,"reported_epoch":593,"state":"active+clean","last_fresh":"2026-02-23T11:34:18.494276+0000","last_change":"2026-02-23T11:34:16.606887+0000","last_active":"2026-02-23T11:34:18.494276+0000","last_peered":"2026-02-23T11:34:18.494276+0000","last_clean":"2026-02-23T11:34:18.494276+0000","last_became_active":"2026-02-23T11:20:11.060769+0000","last_became_peered":"2026-02-23T11:20:11.060769+0000","last_unstale":"2026-02-23T11:34:18.494276+0000","last_undegraded":"2026-02-23T11:34:18.494276+0000","last_fullsized":"2026-02-23T11:34:18.494276+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"15'1","last_scrub_stamp":"2026-02-23T11:34:16.606820+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-23T11:34:16.606820+0000","last_clean_scrub_stamp":"2026-02-23T11:20:10.015542+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"12'52","reported_seq":1820,"reported_epoch":593,"state":"active+clean","last_fresh":"2026-02-23T11:34:18.493326+0000","last_change":"2026-02-23T11:34:16.619245+0000","last_active":"2026-02-23T11:34:18.493326+0000","last_peered":"2026-02-23T11:34:18.493326+0000","last_clean":"2026-02-23T11:34:18.493326+0000","last_became_active":"2026-02-23T11:20:07.033110+0000","last_became_peered":"2026-02-23T11:20:07.033110+0000","last_unstale":"2026-02-23T11:34:18.493326+0000","last_undegraded":"2026-02-23T11:34:18.493326+0000","last_fullsized":"2026-02-23T11:34:18.493326+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"12'52","last_scrub_stamp":"2026-02-23T11:34:16.619180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-23T11:34:16.619180+0000","last_clean_scrub_stamp":"2026-02-23T11:20:06.000692+0000","objects_scrubbed":0,"log_size":52,"log_dups_size":0,"ondisk_log_size":52,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":53,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":160255967232,"available":160249024512,"internally_reserved":0,"allocated":6942720,"data_stored":6942720,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":53,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":53418655744,"available":53416034304,"internally_reserved":0,"allocated":2621440,"data_stored":2621440,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":52,"ondisk_log_size":52,"up":2,"acting":2,"num_store_stats":1}],"osd_stats":[{"osd":2,"up_from":9,"seq":38654706533,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52166656,"kb_used":1716,"kb_used_data":1716,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":52164940,"statfs":{"total":53418655744,"available":53416898560,"internally_reserved":0,"allocated":1757184,"data_stored":1757184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":9,"seq":38654706533,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52166656,"kb_used":2560,"kb_used_data":2560,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":52164096,"statfs":{"total":53418655744,"available":53416034304,"internally_reserved":0,"allocated":2621440,"data_stored":2621440,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":10,"seq":42949673829,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52166656,"kb_used":2504,"kb_used_data":2504,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":52164152,"statfs":{"total":53418655744,"available":53416091648,"internally_reserved":0,"allocated":2564096,"data_stored":2564096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":1,"total":53418655744,"available":53416034304,"internally_reserved":0,"allocated":2621440,"data_stored":2621440,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":53418655744,"available":53416091648,"internally_reserved":0,"allocated":2564096,"data_stored":2564096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":53418655744,"available":53416034304,"internally_reserved":0,"allocated":2621440,"data_stored":2621440,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":53418655744,"available":53416898560,"internally_reserved":0,"allocated":1757184,"data_stored":1757184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-23T11:34:37.318 DEBUG:teuthology.orchestra.run.trial030:> sudo ceph --cluster ceph config set global mon_health_to_clog false 2026-02-23T11:34:37.395 INFO:teuthology.orchestra.run.trial030.stderr:2026-02-23T11:34:37.389+0000 7f179661a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T11:34:37.397 INFO:teuthology.orchestra.run.trial030.stderr:2026-02-23T11:34:37.395+0000 7f179661a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T11:34:37.521 INFO:teuthology.misc:Shutting down mds daemons... 2026-02-23T11:34:37.522 INFO:teuthology.misc:Shutting down osd daemons... 2026-02-23T11:34:37.522 DEBUG:tasks.ceph.osd.0:waiting for process to exit 2026-02-23T11:34:37.522 INFO:teuthology.orchestra.run:waiting for 300 2026-02-23T11:34:37.588 INFO:tasks.ceph.osd.0:Stopped 2026-02-23T11:34:37.589 DEBUG:tasks.ceph.osd.1:waiting for process to exit 2026-02-23T11:34:37.589 INFO:teuthology.orchestra.run:waiting for 300 2026-02-23T11:34:37.654 INFO:tasks.ceph.osd.1:Stopped 2026-02-23T11:34:37.654 DEBUG:tasks.ceph.osd.2:waiting for process to exit 2026-02-23T11:34:37.654 INFO:teuthology.orchestra.run:waiting for 300 2026-02-23T11:34:37.733 INFO:tasks.ceph.osd.2:Stopped 2026-02-23T11:34:37.733 INFO:teuthology.misc:Shutting down mgr daemons... 2026-02-23T11:34:37.733 DEBUG:tasks.ceph.mgr.x:waiting for process to exit 2026-02-23T11:34:37.733 INFO:teuthology.orchestra.run:waiting for 300 2026-02-23T11:34:37.763 INFO:tasks.ceph.mgr.x:Stopped 2026-02-23T11:34:37.764 INFO:teuthology.misc:Shutting down mon daemons... 2026-02-23T11:34:37.764 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2026-02-23T11:34:37.764 INFO:teuthology.orchestra.run:waiting for 300 2026-02-23T11:34:37.816 INFO:tasks.ceph.mon.a:Stopped 2026-02-23T11:34:37.816 INFO:tasks.ceph:Checking cluster log for badness... 2026-02-23T11:34:37.816 DEBUG:teuthology.orchestra.run.trial030:> sudo grep -E '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/ceph.log | grep -E -v '\(MDS_ALL_DOWN\)' | grep -E -v '\(MDS_UP_LESS_THAN_MAX\)' | grep -E -v '\(POOL_APP_NOT_ENABLED\)' | grep -E -v '\(SLOW_OPS\)' | grep -E -v 'slow request' | head -n 1 2026-02-23T11:34:37.882 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-0 on ubuntu@trial030.front.sepia.ceph.com 2026-02-23T11:34:37.883 DEBUG:teuthology.orchestra.run.trial030:> sync && sudo umount -f /var/lib/ceph/osd/ceph-0 2026-02-23T11:34:37.950 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-1 on ubuntu@trial030.front.sepia.ceph.com 2026-02-23T11:34:37.950 DEBUG:teuthology.orchestra.run.trial030:> sync && sudo umount -f /var/lib/ceph/osd/ceph-1 2026-02-23T11:34:38.017 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-2 on ubuntu@trial030.front.sepia.ceph.com 2026-02-23T11:34:38.017 DEBUG:teuthology.orchestra.run.trial030:> sync && sudo umount -f /var/lib/ceph/osd/ceph-2 2026-02-23T11:34:38.084 INFO:tasks.ceph:Archiving mon data... 2026-02-23T11:34:38.087 DEBUG:teuthology.misc:Transferring archived files from trial030:/var/lib/ceph/mon/ceph-a to /home/teuthworker/mnt/teuthology/shraddhaag-2026-02-23_09:58:48-crimson-rados-main-distro-debug-trial/66207/data/mon.a.tgz 2026-02-23T11:34:38.087 DEBUG:teuthology.orchestra.run.trial030:> mktemp 2026-02-23T11:34:38.139 INFO:teuthology.orchestra.run.trial030.stdout:/tmp/tmp.NOmZLg7Y0H 2026-02-23T11:34:38.139 DEBUG:teuthology.orchestra.run.trial030:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-a -- . > /tmp/tmp.NOmZLg7Y0H 2026-02-23T11:34:38.235 DEBUG:teuthology.orchestra.run.trial030:> sudo chmod 0666 /tmp/tmp.NOmZLg7Y0H 2026-02-23T11:34:38.319 DEBUG:teuthology.orchestra.remote:trial030:/tmp/tmp.NOmZLg7Y0H is 253KB 2026-02-23T11:34:38.413 DEBUG:teuthology.orchestra.run.trial030:> rm -fr /tmp/tmp.NOmZLg7Y0H 2026-02-23T11:34:38.430 INFO:tasks.ceph:Cleaning ceph cluster... 2026-02-23T11:34:38.430 DEBUG:teuthology.orchestra.run.trial030:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2026-02-23T11:34:38.550 INFO:teuthology.util.scanner:summary_data or yaml_file is empty! 2026-02-23T11:34:38.550 INFO:tasks.ceph:Archiving crash dumps... 2026-02-23T11:34:38.553 DEBUG:teuthology.misc:Transferring archived files from trial030:/var/lib/ceph/crash to /home/teuthworker/mnt/teuthology/shraddhaag-2026-02-23_09:58:48-crimson-rados-main-distro-debug-trial/66207/remote/trial030/crash 2026-02-23T11:34:38.554 DEBUG:teuthology.orchestra.run.trial030:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2026-02-23T11:34:38.613 INFO:tasks.ceph:Compressing logs... 2026-02-23T11:34:38.613 DEBUG:teuthology.orchestra.run.trial030:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2026-02-23T11:34:38.636 INFO:teuthology.orchestra.run.trial030.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.2.log 2026-02-23T11:34:38.636 INFO:teuthology.orchestra.run.trial030.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mon.a.log 2026-02-23T11:34:38.636 INFO:teuthology.orchestra.run.trial030.stderr:/var/log/ceph/ceph-osd.2.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.0.log 2026-02-23T11:34:38.637 INFO:teuthology.orchestra.run.trial030.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37700.log 2026-02-23T11:34:38.637 INFO:teuthology.orchestra.run.trial030.stderr:/var/log/ceph/ceph-mon.a.log: /var/log/ceph/ceph-osd.0.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37308.log 2026-02-23T11:34:38.637 INFO:teuthology.orchestra.run.trial030.stderr:/var/log/ceph/ceph-client.admin.37700.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40284.log 2026-02-23T11:34:38.637 INFO:teuthology.orchestra.run.trial030.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.37700.log.gz 2026-02-23T11:34:38.637 INFO:teuthology.orchestra.run.trial030.stderr:/var/log/ceph/ceph-client.admin.37308.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37605.log 2026-02-23T11:34:38.638 INFO:teuthology.orchestra.run.trial030.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.37308.log.gz 2026-02-23T11:34:38.638 INFO:teuthology.orchestra.run.trial030.stderr:/var/log/ceph/ceph-client.admin.40284.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37357.log 2026-02-23T11:34:38.638 INFO:teuthology.orchestra.run.trial030.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.40284.log.gz 2026-02-23T11:34:38.638 INFO:teuthology.orchestra.run.trial030.stderr:/var/log/ceph/ceph-client.admin.37605.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37749.log 2026-02-23T11:34:38.638 INFO:teuthology.orchestra.run.trial030.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.37605.log.gz 2026-02-23T11:34:38.638 INFO:teuthology.orchestra.run.trial030.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39892.log 2026-02-23T11:34:38.638 INFO:teuthology.orchestra.run.trial030.stderr:/var/log/ceph/ceph-client.admin.37357.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.37357.log.gz 2026-02-23T11:34:38.639 INFO:teuthology.orchestra.run.trial030.stderr:/var/log/ceph/ceph-client.admin.37749.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36476.log 2026-02-23T11:34:38.639 INFO:teuthology.orchestra.run.trial030.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.37749.log.gz 2026-02-23T11:34:38.639 INFO:teuthology.orchestra.run.trial030.stderr:/var/log/ceph/ceph-client.admin.39892.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36339.log 2026-02-23T11:34:38.639 INFO:teuthology.orchestra.run.trial030.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.39892.log.gz 2026-02-23T11:34:38.639 INFO:teuthology.orchestra.run.trial030.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37259.log 2026-02-23T11:34:38.639 INFO:teuthology.orchestra.run.trial030.stderr:/var/log/ceph/ceph-client.admin.36476.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.36476.log.gz 2026-02-23T11:34:38.639 INFO:teuthology.orchestra.run.trial030.stderr:/var/log/ceph/ceph-client.admin.36339.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40382.log 2026-02-23T11:34:38.639 INFO:teuthology.orchestra.run.trial030.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.36339.log.gz 2026-02-23T11:34:38.639 INFO:teuthology.orchestra.run.trial030.stderr:/var/log/ceph/ceph-client.admin.37259.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.37259.log.gz 2026-02-23T11:34:38.640 INFO:teuthology.orchestra.run.trial030.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37459.log 2026-02-23T11:34:38.640 INFO:teuthology.orchestra.run.trial030.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.tmp-client.admin.34910.log 2026-02-23T11:34:38.640 INFO:teuthology.orchestra.run.trial030.stderr:/var/log/ceph/ceph-client.admin.40382.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.40382.log.gz 2026-02-23T11:34:38.640 INFO:teuthology.orchestra.run.trial030.stderr:/var/log/ceph/ceph-client.admin.37459.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40088.log 2026-02-23T11:34:38.640 INFO:teuthology.orchestra.run.trial030.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.37459.log.gz 2026-02-23T11:34:38.640 INFO:teuthology.orchestra.run.trial030.stderr:/var/log/ceph/ceph.tmp-client.admin.34910.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36789.log 2026-02-23T11:34:38.640 INFO:teuthology.orchestra.run.trial030.stderr: 67.5% -- replaced with /var/log/ceph/ceph.tmp-client.admin.34910.log.gz 2026-02-23T11:34:38.640 INFO:teuthology.orchestra.run.trial030.stderr:/var/log/ceph/ceph-client.admin.40088.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39990.log 2026-02-23T11:34:38.640 INFO:teuthology.orchestra.run.trial030.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.40088.log.gz 2026-02-23T11:34:38.641 INFO:teuthology.orchestra.run.trial030.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37210.log 2026-02-23T11:34:38.641 INFO:teuthology.orchestra.run.trial030.stderr:/var/log/ceph/ceph-client.admin.36789.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.36789.log.gz 2026-02-23T11:34:38.641 INFO:teuthology.orchestra.run.trial030.stderr:/var/log/ceph/ceph-client.admin.39990.log: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.39990.log.gz 2026-02-23T11:34:38.641 INFO:teuthology.orchestra.run.trial030.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36292.log 2026-02-23T11:34:38.641 INFO:teuthology.orchestra.run.trial030.stderr:/var/log/ceph/ceph-client.admin.37210.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.37210.log.gz 2026-02-23T11:34:38.641 INFO:teuthology.orchestra.run.trial030.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37050.log 2026-02-23T11:34:38.641 INFO:teuthology.orchestra.run.trial030.stderr:/var/log/ceph/ceph-client.admin.36292.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40431.log 2026-02-23T11:34:38.641 INFO:teuthology.orchestra.run.trial030.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.36292.log.gz 2026-02-23T11:34:38.642 INFO:teuthology.orchestra.run.trial030.stderr:/var/log/ceph/ceph-client.admin.37050.log: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.37050.log.gz 2026-02-23T11:34:38.642 INFO:teuthology.orchestra.run.trial030.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40333.log 2026-02-23T11:34:38.642 INFO:teuthology.orchestra.run.trial030.stderr:/var/log/ceph/ceph-client.admin.40431.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38171.log 2026-02-23T11:34:38.642 INFO:teuthology.orchestra.run.trial030.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.40431.log.gz 2026-02-23T11:34:38.642 INFO:teuthology.orchestra.run.trial030.stderr:/var/log/ceph/ceph-client.admin.40333.log: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.40333.log.gz 2026-02-23T11:34:38.642 INFO:teuthology.orchestra.run.trial030.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40186.log 2026-02-23T11:34:38.642 INFO:teuthology.orchestra.run.trial030.stderr:/var/log/ceph/ceph-client.admin.38171.log: gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2026-02-23T11:34:38.643 INFO:teuthology.orchestra.run.trial030.stderr:/var/log/ceph/ceph-client.admin.40186.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.38171.log.gz 2026-02-23T11:34:38.643 INFO:teuthology.orchestra.run.trial030.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.40186.log.gz 2026-02-23T11:34:38.643 INFO:teuthology.orchestra.run.trial030.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40137.log 2026-02-23T11:34:38.643 INFO:teuthology.orchestra.run.trial030.stderr:/var/log/ceph/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37458.log 2026-02-23T11:34:38.643 INFO:teuthology.orchestra.run.trial030.stderr:/var/log/ceph/ceph-client.admin.40137.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39941.log 2026-02-23T11:34:38.643 INFO:teuthology.orchestra.run.trial030.stderr: 86.8% -- replaced with /var/log/ceph/ceph.audit.log.gz 2026-02-23T11:34:38.643 INFO:teuthology.orchestra.run.trial030.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.40137.log.gz 2026-02-23T11:34:38.644 INFO:teuthology.orchestra.run.trial030.stderr:/var/log/ceph/ceph-client.admin.37458.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.37458.log.gz 2026-02-23T11:34:38.644 INFO:teuthology.orchestra.run.trial030.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37896.log 2026-02-23T11:34:38.644 INFO:teuthology.orchestra.run.trial030.stderr:/var/log/ceph/ceph-client.admin.39941.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.1.log 2026-02-23T11:34:38.644 INFO:teuthology.orchestra.run.trial030.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.39941.log.gz 2026-02-23T11:34:38.644 INFO:teuthology.orchestra.run.trial030.stderr:/var/log/ceph/ceph-client.admin.37896.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40039.log 2026-02-23T11:34:38.644 INFO:teuthology.orchestra.run.trial030.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.37896.log.gz 2026-02-23T11:34:38.644 INFO:teuthology.orchestra.run.trial030.stderr:/var/log/ceph/ceph-osd.1.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37145.log 2026-02-23T11:34:38.645 INFO:teuthology.orchestra.run.trial030.stderr:/var/log/ceph/ceph-client.admin.40039.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36197.log 2026-02-23T11:34:38.645 INFO:teuthology.orchestra.run.trial030.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.40039.log.gz 2026-02-23T11:34:38.645 INFO:teuthology.orchestra.run.trial030.stderr:/var/log/ceph/ceph-client.admin.37145.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36570.log 2026-02-23T11:34:38.645 INFO:teuthology.orchestra.run.trial030.stderr: 66.1% -- replaced with /var/log/ceph/ceph-client.admin.37145.log.gz 2026-02-23T11:34:38.645 INFO:teuthology.orchestra.run.trial030.stderr:/var/log/ceph/ceph-client.admin.36197.log: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.36197.log.gz 2026-02-23T11:34:38.645 INFO:teuthology.orchestra.run.trial030.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37120.log 2026-02-23T11:34:38.646 INFO:teuthology.orchestra.run.trial030.stderr:/var/log/ceph/ceph-client.admin.36570.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37607.log 2026-02-23T11:34:38.646 INFO:teuthology.orchestra.run.trial030.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.36570.log.gz 2026-02-23T11:34:38.646 INFO:teuthology.orchestra.run.trial030.stderr:/var/log/ceph/ceph-client.admin.37120.log: gzip -5 --verbose -- /var/log/ceph/ceph-mgr.x.log 2026-02-23T11:34:38.646 INFO:teuthology.orchestra.run.trial030.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.37120.log.gz 2026-02-23T11:34:38.646 INFO:teuthology.orchestra.run.trial030.stderr:/var/log/ceph/ceph-client.admin.37607.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37460.log 2026-02-23T11:34:38.646 INFO:teuthology.orchestra.run.trial030.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.37607.log.gz 2026-02-23T11:34:38.646 INFO:teuthology.orchestra.run.trial030.stderr:/var/log/ceph/ceph-mgr.x.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36523.log 2026-02-23T11:34:38.647 INFO:teuthology.orchestra.run.trial030.stderr:/var/log/ceph/ceph-client.admin.37460.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.37460.log.gz 2026-02-23T11:34:38.647 INFO:teuthology.orchestra.run.trial030.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37798.log 2026-02-23T11:34:38.647 INFO:teuthology.orchestra.run.trial030.stderr:/var/log/ceph/ceph-client.admin.36523.log: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.36523.log.gz 2026-02-23T11:34:38.647 INFO:teuthology.orchestra.run.trial030.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.log 2026-02-23T11:34:38.647 INFO:teuthology.orchestra.run.trial030.stderr:/var/log/ceph/ceph-client.admin.37798.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36897.log 2026-02-23T11:34:38.647 INFO:teuthology.orchestra.run.trial030.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.37798.log.gz 2026-02-23T11:34:38.647 INFO:teuthology.orchestra.run.trial030.stderr:/var/log/ceph/ceph.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36975.log 2026-02-23T11:34:38.648 INFO:teuthology.orchestra.run.trial030.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37606.log 2026-02-23T11:34:38.648 INFO:teuthology.orchestra.run.trial030.stderr:/var/log/ceph/ceph-client.admin.36897.log: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.36897.log.gz 2026-02-23T11:34:38.648 INFO:teuthology.orchestra.run.trial030.stderr:/var/log/ceph/ceph-client.admin.36975.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37847.log 2026-02-23T11:34:38.648 INFO:teuthology.orchestra.run.trial030.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.36975.log.gz 2026-02-23T11:34:38.648 INFO:teuthology.orchestra.run.trial030.stderr:/var/log/ceph/ceph-client.admin.37606.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40235.log 2026-02-23T11:34:38.648 INFO:teuthology.orchestra.run.trial030.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.37606.log.gz 2026-02-23T11:34:38.648 INFO:teuthology.orchestra.run.trial030.stderr:/var/log/ceph/ceph-client.admin.37847.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40478.log 2026-02-23T11:34:38.649 INFO:teuthology.orchestra.run.trial030.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.37847.log.gz 2026-02-23T11:34:38.649 INFO:teuthology.orchestra.run.trial030.stderr:/var/log/ceph/ceph-client.admin.40235.log: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.40235.log.gz 2026-02-23T11:34:38.649 INFO:teuthology.orchestra.run.trial030.stderr:/var/log/ceph/ceph-client.admin.40478.log: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.40478.log.gz 2026-02-23T11:34:38.649 INFO:teuthology.orchestra.run.trial030.stderr: 86.8% -- replaced with /var/log/ceph/ceph.log.gz 2026-02-23T11:34:38.690 INFO:teuthology.orchestra.run.trial030.stderr: 91.1% -- replaced with /var/log/ceph/ceph-mgr.x.log.gz 2026-02-23T11:34:38.889 INFO:teuthology.orchestra.run.trial030.stderr: 89.9% -- replaced with /var/log/ceph/ceph-mon.a.log.gz 2026-02-23T11:34:42.849 INFO:teuthology.orchestra.run.trial030.stderr: 93.1% -- replaced with /var/log/ceph/ceph-osd.2.log.gz 2026-02-23T11:34:44.423 INFO:teuthology.orchestra.run.trial030.stderr: 93.2% -- replaced with /var/log/ceph/ceph-osd.1.log.gz 2026-02-23T11:34:49.871 INFO:teuthology.orchestra.run.trial030.stderr: 95.3% -- replaced with /var/log/ceph/ceph-osd.0.log.gz 2026-02-23T11:34:49.873 INFO:teuthology.orchestra.run.trial030.stderr: 2026-02-23T11:34:49.873 INFO:teuthology.orchestra.run.trial030.stderr:real 0m11.244s 2026-02-23T11:34:49.873 INFO:teuthology.orchestra.run.trial030.stderr:user 0m20.240s 2026-02-23T11:34:49.873 INFO:teuthology.orchestra.run.trial030.stderr:sys 0m1.280s 2026-02-23T11:34:49.873 INFO:tasks.ceph:Archiving logs... 2026-02-23T11:34:49.874 DEBUG:teuthology.misc:Transferring archived files from trial030:/var/log/ceph to /home/teuthworker/mnt/teuthology/shraddhaag-2026-02-23_09:58:48-crimson-rados-main-distro-debug-trial/66207/remote/trial030/log 2026-02-23T11:34:49.875 DEBUG:teuthology.orchestra.run.trial030:> sudo tar c -f - -C /var/log/ceph -- . 2026-02-23T11:34:51.550 DEBUG:teuthology.run_tasks:Unwinding manager install 2026-02-23T11:34:51.618 DEBUG:teuthology.run_tasks:Unwinding manager install 2026-02-23T11:34:51.628 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2026-02-23T11:34:51.628 DEBUG:teuthology.orchestra.run.trial030:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2026-02-23T11:34:51.654 DEBUG:teuthology.run_tasks:Unwinding manager clock 2026-02-23T11:34:51.663 INFO:teuthology.task.clock:Checking final clock skew... 2026-02-23T11:34:51.663 DEBUG:teuthology.orchestra.run.trial030:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2026-02-23T11:34:51.708 INFO:teuthology.orchestra.run.trial030.stderr:bash: line 1: ntpq: command not found 2026-02-23T11:34:51.740 INFO:teuthology.orchestra.run.trial030.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2026-02-23T11:34:51.740 INFO:teuthology.orchestra.run.trial030.stdout:=============================================================================== 2026-02-23T11:34:51.740 INFO:teuthology.orchestra.run.trial030.stdout:^- ntp2.zeus.frumentum.media 2 6 377 40 -106ms[ -106ms] +/- 129ms 2026-02-23T11:34:51.740 INFO:teuthology.orchestra.run.trial030.stdout:^* nyc2.us.ntp.li 2 6 377 39 -70us[ -98us] +/- 2707us 2026-02-23T11:34:51.740 INFO:teuthology.orchestra.run.trial030.stdout:^- ntp6.kernfusion.at 2 6 377 37 +735us[ +735us] +/- 46ms 2026-02-23T11:34:51.740 INFO:teuthology.orchestra.run.trial030.stdout:^- s2-a.time.mci1.us.rozint> 2 7 377 105 +1294us[+1269us] +/- 23ms 2026-02-23T11:34:51.741 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2026-02-23T11:34:51.753 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2026-02-23T11:34:51.754 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2026-02-23T11:34:51.766 DEBUG:teuthology.orchestra.run.trial030:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2026-02-23T11:34:51.832 DEBUG:teuthology.orchestra.run.trial030:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 2026-02-23T11:34:51.858 DEBUG:teuthology.orchestra.run:got remote process result: 1 2026-02-23T11:34:51.859 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2026-02-23T11:34:51.878 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2026-02-23T11:34:51.893 INFO:teuthology.task.internal:Duration was 1500.351567 seconds 2026-02-23T11:34:51.894 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2026-02-23T11:34:51.909 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2026-02-23T11:34:51.909 DEBUG:teuthology.orchestra.run.trial030:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2026-02-23T11:34:51.944 INFO:teuthology.orchestra.run.trial030.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2026-02-23T11:34:52.293 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2026-02-23T11:34:52.293 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@trial030.front.sepia.ceph.com 2026-02-23T11:34:52.294 DEBUG:teuthology.orchestra.run.trial030:> grep -E --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | grep -E -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | grep -E -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2026-02-23T11:34:52.352 INFO:teuthology.task.internal.syslog:Gathering journactl... 2026-02-23T11:34:52.353 DEBUG:teuthology.orchestra.run.trial030:> sudo journalctl > /home/ubuntu/cephtest/archive/syslog/journalctl.log 2026-02-23T11:34:52.453 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2026-02-23T11:34:52.453 DEBUG:teuthology.orchestra.run.trial030:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --max-args=1 --max-procs=0 --verbose --no-run-if-empty -- gzip -5 --verbose -- 2026-02-23T11:34:52.476 INFO:teuthology.orchestra.run.trial030.stderr:gzip -5 --verbose -- /home/ubuntu/cephtest/archive/syslog/misc.log 2026-02-23T11:34:52.478 INFO:teuthology.orchestra.run.trial030.stderr:gzip -5 --verbose -- /home/ubuntu/cephtest/archive/syslog/kern.log 2026-02-23T11:34:52.478 INFO:teuthology.orchestra.run.trial030.stderr:/home/ubuntu/cephtest/archive/syslog/misc.log: 0.0% -- replaced with /home/ubuntu/cephtest/archive/syslog/misc.log.gz 2026-02-23T11:34:52.478 INFO:teuthology.orchestra.run.trial030.stderr:gzip -5 --verbose -- /home/ubuntu/cephtest/archive/syslog/journalctl.log 2026-02-23T11:34:52.478 INFO:teuthology.orchestra.run.trial030.stderr:/home/ubuntu/cephtest/archive/syslog/kern.log: 0.0% -- replaced with /home/ubuntu/cephtest/archive/syslog/kern.log.gz 2026-02-23T11:34:52.488 INFO:teuthology.orchestra.run.trial030.stderr:/home/ubuntu/cephtest/archive/syslog/journalctl.log: 81.8% -- replaced with /home/ubuntu/cephtest/archive/syslog/journalctl.log.gz 2026-02-23T11:34:52.490 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2026-02-23T11:34:52.507 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2026-02-23T11:34:52.507 DEBUG:teuthology.orchestra.run.trial030:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2026-02-23T11:34:52.554 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2026-02-23T11:34:52.572 DEBUG:teuthology.orchestra.run.trial030:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2026-02-23T11:34:52.616 INFO:teuthology.orchestra.run.trial030.stdout:kernel.core_pattern = core 2026-02-23T11:34:52.628 DEBUG:teuthology.orchestra.run.trial030:> test -e /home/ubuntu/cephtest/archive/coredump 2026-02-23T11:34:52.683 DEBUG:teuthology.orchestra.run:got remote process result: 1 2026-02-23T11:34:52.684 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2026-02-23T11:34:52.703 INFO:teuthology.task.internal:Transferring archived files... 2026-02-23T11:34:52.704 DEBUG:teuthology.misc:Transferring archived files from trial030:/home/ubuntu/cephtest/archive to /home/teuthworker/mnt/teuthology/shraddhaag-2026-02-23_09:58:48-crimson-rados-main-distro-debug-trial/66207/remote/trial030 2026-02-23T11:34:52.704 DEBUG:teuthology.orchestra.run.trial030:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2026-02-23T11:34:52.760 INFO:teuthology.task.internal:Removing archive directory... 2026-02-23T11:34:52.760 DEBUG:teuthology.orchestra.run.trial030:> rm -rf -- /home/ubuntu/cephtest/archive 2026-02-23T11:34:52.808 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2026-02-23T11:34:52.826 INFO:teuthology.task.internal:Not uploading archives. 2026-02-23T11:34:52.826 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2026-02-23T11:34:52.843 INFO:teuthology.task.internal:Tidying up after the test... 2026-02-23T11:34:52.843 DEBUG:teuthology.orchestra.run.trial030:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2026-02-23T11:34:52.864 INFO:teuthology.orchestra.run.trial030.stdout: 917576 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Feb 23 11:34 /home/ubuntu/cephtest 2026-02-23T11:34:52.865 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2026-02-23T11:34:52.883 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2026-02-23T11:34:52.921 INFO:teuthology.run:Summary data: description: crimson-rados/rbd/{0-crimson_install clusters/crimson-fixed-1 crimson-supported-all-distro/centos_latest crimson_qa_overrides deploy/ceph objectstore/seastore/segmented$/{crimson_seastore_segmented_2q} tasks/rbd_python_api_tests_old_format} duration: 1500.3515667915344 flavor: debug owner: scheduled_shraddhaag@soko04.front.sepia.ceph.com success: true 2026-02-23T11:34:52.922 DEBUG:teuthology.report:Pushing job info to https://paddles-paddles.apps.pok.os.sepia.ceph.com 2026-02-23T11:34:53.052 INFO:teuthology.run:pass