2026-02-23T11:05:34.074 INFO:root:teuthology version: 1.2.4.dev9+gfa17720d0 2026-02-23T11:05:34.088 DEBUG:teuthology.report:Pushing job info to https://paddles-paddles.apps.pok.os.sepia.ceph.com 2026-02-23T11:05:34.165 INFO:teuthology.run:Config: archive_path: /home/teuthworker/mnt/teuthology/shraddhaag-2026-02-23_09:58:48-crimson-rados-main-distro-debug-trial/66185 branch: main description: crimson-rados/singleton/{0-crimson_install all/osd-dump-metrics crimson-supported-all-distro/centos_latest crimson_qa_overrides objectstore/seastore/segmented$/{crimson_seastore_segmented_2q} rados} email: null first_in_suite: false flavor: debug job_id: '66185' kernel: branch: distro kdb: true sha1: distro last_in_suite: false machine_type: trial name: shraddhaag-2026-02-23_09:58:48-crimson-rados-main-distro-debug-trial no_nested_subset: false openstack: - volumes: count: 3 size: 10 os_type: centos os_version: 9.stream overrides: admin_socket: branch: main ceph: conf: global: enable experimental unrecoverable data corrupting features: crimson mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 30 debug ms: 1 debug paxos: 20 mon scrub interval: 300 osd pool default crimson: true osd: bluestore zero block detection: true crimson osd obc lru size: 10 debug ms: 20 debug osd: 20 debug seastore: 20 debug seastore backref: 20 debug seastore cache: 20 debug seastore cleaner: 20 debug seastore device: 20 debug seastore epm: 20 debug seastore fixedkv tree: 20 debug seastore journal: 20 debug seastore lba: 20 debug seastore odata: 20 debug seastore omap: 20 debug seastore onode: 20 debug seastore t: 20 debug seastore tm: 20 osd debug verify cached snaps: true osd debug verify missing on start: true osd mclock override recovery settings: true osd mclock profile: high_recovery_ops osd mclock skip benchmark: true osd objectstore: seastore osd op queue: debug_random osd op queue cut off: debug_random seastore cachepin size pershard: 64M seastore cachepin type: 2Q seastore max concurrent transactions: 8 crimson_compat: true flavor: debug log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - \(POOL_APP_NOT_ENABLED\) sha1: e371433fcd1390a0b82cfb18effc422c1f5d8c1d ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} install: ceph: flavor: debug sha1: e371433fcd1390a0b82cfb18effc422c1f5d8c1d workunit: branch: wip-shraddhaag-enable-cephadm-tests env: CRIMSON_COMPAT: '1' sha1: e371433fcd1390a0b82cfb18effc422c1f5d8c1d owner: scheduled_shraddhaag@soko04.front.sepia.ceph.com priority: 49 repo: https://git.ceph.com/ceph-ci.git roles: - - mon.a - mon.b - mon.c - mgr.x - osd.0 - osd.1 - osd.2 - client.0 seed: 66 sha1: e371433fcd1390a0b82cfb18effc422c1f5d8c1d sleep_before_teardown: 0 suite: crimson-rados suite_branch: wip-shraddhaag-enable-cephadm-tests suite_path: /home/teuthworker/src/github.com_shraddhaag_ceph_e371433fcd1390a0b82cfb18effc422c1f5d8c1d/qa suite_relpath: qa suite_repo: https://github.com/shraddhaag/ceph.git suite_sha1: e371433fcd1390a0b82cfb18effc422c1f5d8c1d targets: trial191.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBHbuM0nLiGLKLeTIrVEMqc4uTbmdtFdgvjMkr94OYob6EeBfFEoLGtnro+X3jGP9hjt0mD2BTwYG4zPY4TwqZsI= tasks: - install: extra_packages: deb: - ceph-osd-crimson - ceph-osd-crimson-dbg rpm: - ceph-osd-crimson - ceph-osd-crimson-debuginfo - pexec: all: - sudo update-alternatives --set ceph-osd /usr/bin/ceph-osd-crimson - install: null - ceph: conf: global: null osd: crimson cpu num: 2 osd min pg log entries: 5 log-ignorelist: - but it is still running - overall HEALTH_ pre-mgr-commands: - sudo ceph config set mgr mgr_pool false --force - workunit: clients: client.0: - cephtool/test_crimson_dump_metrics.sh teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: fa17720d0088c3ac28e473468bfc79eeaff5cd38 timestamp: 2026-02-23_09:58:48 tube: trial user: shraddhaag verbose: false worker_log: /home/teuthworker/mnt/teuthology/worker_logs/dispatcher.trial.3318653 2026-02-23T11:05:34.165 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/github.com_shraddhaag_ceph_e371433fcd1390a0b82cfb18effc422c1f5d8c1d/qa; will attempt to use it 2026-02-23T11:05:34.166 INFO:teuthology.run:Found tasks at /home/teuthworker/src/github.com_shraddhaag_ceph_e371433fcd1390a0b82cfb18effc422c1f5d8c1d/qa/tasks 2026-02-23T11:05:34.166 INFO:teuthology.run_tasks:Running task internal.check_packages... 2026-02-23T11:05:34.171 INFO:teuthology.task.internal:Checking packages... 2026-02-23T11:05:34.171 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'debug' and ceph hash 'e371433fcd1390a0b82cfb18effc422c1f5d8c1d' 2026-02-23T11:05:34.171 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2026-02-23T11:05:34.171 INFO:teuthology.packaging:ref: None 2026-02-23T11:05:34.171 INFO:teuthology.packaging:tag: None 2026-02-23T11:05:34.171 INFO:teuthology.packaging:branch: main 2026-02-23T11:05:34.171 INFO:teuthology.packaging:sha1: e371433fcd1390a0b82cfb18effc422c1f5d8c1d 2026-02-23T11:05:34.171 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=debug&distros=centos%2F9%2Fx86_64&ref=main 2026-02-23T11:05:34.292 INFO:teuthology.task.internal:Found packages for ceph version 20.3.0-5253.g5ccdd6ad 2026-02-23T11:05:34.294 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2026-02-23T11:05:34.305 INFO:teuthology.task.internal:no buildpackages task found 2026-02-23T11:05:34.305 INFO:teuthology.run_tasks:Running task internal.save_config... 2026-02-23T11:05:34.318 INFO:teuthology.task.internal:Saving configuration 2026-02-23T11:05:34.329 INFO:teuthology.run_tasks:Running task internal.check_lock... 2026-02-23T11:05:34.341 INFO:teuthology.task.internal.check_lock:Checking locks... 2026-02-23T11:05:34.418 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'trial191.front.sepia.ceph.com', 'description': '/home/teuthworker/mnt/teuthology/shraddhaag-2026-02-23_09:58:48-crimson-rados-main-distro-debug-trial/66185', 'up': True, 'machine_type': 'trial', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9.stream', 'arch': 'x86_64', 'locked': True, 'locked_since': '2026-02-23 11:02:08.507891', 'locked_by': 'scheduled_shraddhaag@soko04.front.sepia.ceph.com', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBHbuM0nLiGLKLeTIrVEMqc4uTbmdtFdgvjMkr94OYob6EeBfFEoLGtnro+X3jGP9hjt0mD2BTwYG4zPY4TwqZsI='} 2026-02-23T11:05:34.418 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2026-02-23T11:05:34.429 INFO:teuthology.task.internal:roles: ubuntu@trial191.front.sepia.ceph.com - ['mon.a', 'mon.b', 'mon.c', 'mgr.x', 'osd.0', 'osd.1', 'osd.2', 'client.0'] 2026-02-23T11:05:34.429 INFO:teuthology.run_tasks:Running task console_log... 2026-02-23T11:05:34.515 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7dc7d7004ea0>, signals=[15]) 2026-02-23T11:05:34.515 INFO:teuthology.run_tasks:Running task internal.connect... 2026-02-23T11:05:34.526 INFO:teuthology.task.internal:Opening connections... 2026-02-23T11:05:34.526 DEBUG:teuthology.task.internal:connecting to ubuntu@trial191.front.sepia.ceph.com 2026-02-23T11:05:34.527 DEBUG:teuthology.orchestra.connection:{'hostname': 'trial191.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2026-02-23T11:05:34.593 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2026-02-23T11:05:34.607 DEBUG:teuthology.orchestra.run.trial191:> uname -m 2026-02-23T11:05:34.668 INFO:teuthology.orchestra.run.trial191.stdout:x86_64 2026-02-23T11:05:34.668 DEBUG:teuthology.orchestra.run.trial191:> cat /etc/os-release 2026-02-23T11:05:34.723 INFO:teuthology.orchestra.run.trial191.stdout:NAME="CentOS Stream" 2026-02-23T11:05:34.723 INFO:teuthology.orchestra.run.trial191.stdout:VERSION="9" 2026-02-23T11:05:34.724 INFO:teuthology.orchestra.run.trial191.stdout:ID="centos" 2026-02-23T11:05:34.724 INFO:teuthology.orchestra.run.trial191.stdout:ID_LIKE="rhel fedora" 2026-02-23T11:05:34.724 INFO:teuthology.orchestra.run.trial191.stdout:VERSION_ID="9" 2026-02-23T11:05:34.724 INFO:teuthology.orchestra.run.trial191.stdout:PLATFORM_ID="platform:el9" 2026-02-23T11:05:34.724 INFO:teuthology.orchestra.run.trial191.stdout:PRETTY_NAME="CentOS Stream 9" 2026-02-23T11:05:34.724 INFO:teuthology.orchestra.run.trial191.stdout:ANSI_COLOR="0;31" 2026-02-23T11:05:34.724 INFO:teuthology.orchestra.run.trial191.stdout:LOGO="fedora-logo-icon" 2026-02-23T11:05:34.724 INFO:teuthology.orchestra.run.trial191.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2026-02-23T11:05:34.724 INFO:teuthology.orchestra.run.trial191.stdout:HOME_URL="https://centos.org/" 2026-02-23T11:05:34.724 INFO:teuthology.orchestra.run.trial191.stdout:BUG_REPORT_URL="https://issues.redhat.com/" 2026-02-23T11:05:34.724 INFO:teuthology.orchestra.run.trial191.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2026-02-23T11:05:34.724 INFO:teuthology.orchestra.run.trial191.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2026-02-23T11:05:34.725 INFO:teuthology.lock.ops:Updating trial191.front.sepia.ceph.com on lock server 2026-02-23T11:05:34.815 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2026-02-23T11:05:34.827 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2026-02-23T11:05:34.838 INFO:teuthology.task.internal:Checking for old test directory... 2026-02-23T11:05:34.838 DEBUG:teuthology.orchestra.run.trial191:> test '!' -e /home/ubuntu/cephtest 2026-02-23T11:05:34.853 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2026-02-23T11:05:34.864 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2026-02-23T11:05:34.864 DEBUG:teuthology.orchestra.run.trial191:> test -z $(ls -A /var/lib/ceph) 2026-02-23T11:05:34.910 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2026-02-23T11:05:35.016 INFO:teuthology.run_tasks:Running task kernel... 2026-02-23T11:05:35.042 INFO:teuthology.task.kernel:normalize config orig: {'branch': 'distro', 'kdb': True, 'sha1': 'distro'} 2026-02-23T11:05:35.042 INFO:teuthology.task.kernel:config {'mon.a': {'branch': 'distro', 'kdb': True, 'sha1': 'distro'}}, timeout 300 2026-02-23T11:05:35.042 DEBUG:teuthology.orchestra.run.trial191:> test -f /run/.containerenv -o -f /.dockerenv 2026-02-23T11:05:35.056 DEBUG:teuthology.orchestra.run:got remote process result: 1 2026-02-23T11:05:35.057 DEBUG:teuthology.orchestra.run.trial191:> uname -r 2026-02-23T11:05:35.112 INFO:teuthology.orchestra.run.trial191.stdout:5.14.0-681.el9.x86_64 2026-02-23T11:05:35.112 INFO:teuthology.task.kernel:Running kernel on trial191: 5.14.0-681.el9.x86_64 2026-02-23T11:05:35.112 DEBUG:teuthology.orchestra.run.trial191:> sudo yum install -y kernel 2026-02-23T11:05:44.097 INFO:teuthology.orchestra.run.trial191.stdout:CentOS Stream 9 - BaseOS 1.0 MB/s | 8.9 MB 00:08 2026-02-23T11:05:48.003 INFO:teuthology.orchestra.run.trial191.stdout:CentOS Stream 9 - AppStream 7.9 MB/s | 27 MB 00:03 2026-02-23T11:05:52.283 INFO:teuthology.orchestra.run.trial191.stdout:CentOS Stream 9 - CRB 4.0 MB/s | 7.8 MB 00:01 2026-02-23T11:05:53.279 INFO:teuthology.orchestra.run.trial191.stdout:CentOS Stream 9 - Extras packages 74 kB/s | 20 kB 00:00 2026-02-23T11:05:53.834 INFO:teuthology.orchestra.run.trial191.stdout:Extra Packages for Enterprise Linux 41 MB/s | 20 MB 00:00 2026-02-23T11:05:57.021 INFO:teuthology.orchestra.run.trial191.stdout:lab-extras 2.5 MB/s | 50 kB 00:00 2026-02-23T11:05:58.030 INFO:teuthology.orchestra.run.trial191.stdout:Package kernel-5.14.0-645.el9.x86_64 is already installed. 2026-02-23T11:05:58.030 INFO:teuthology.orchestra.run.trial191.stdout:Package kernel-5.14.0-681.el9.x86_64 is already installed. 2026-02-23T11:05:58.049 INFO:teuthology.orchestra.run.trial191.stdout:Dependencies resolved. 2026-02-23T11:05:58.049 INFO:teuthology.orchestra.run.trial191.stdout:Nothing to do. 2026-02-23T11:05:58.050 INFO:teuthology.orchestra.run.trial191.stdout:Complete! 2026-02-23T11:05:58.072 DEBUG:teuthology.orchestra.run.trial191:> echo no | sudo yum reinstall kernel || true 2026-02-23T11:05:58.370 INFO:teuthology.orchestra.run.trial191.stdout:Last metadata expiration check: 0:00:01 ago on Mon 23 Feb 2026 11:05:57 AM UTC. 2026-02-23T11:05:58.452 INFO:teuthology.orchestra.run.trial191.stdout:Dependencies resolved. 2026-02-23T11:05:58.453 INFO:teuthology.orchestra.run.trial191.stdout:================================================================================ 2026-02-23T11:05:58.453 INFO:teuthology.orchestra.run.trial191.stdout: Package Architecture Version Repository Size 2026-02-23T11:05:58.453 INFO:teuthology.orchestra.run.trial191.stdout:================================================================================ 2026-02-23T11:05:58.453 INFO:teuthology.orchestra.run.trial191.stdout:Reinstalling: 2026-02-23T11:05:58.453 INFO:teuthology.orchestra.run.trial191.stdout: kernel x86_64 5.14.0-681.el9 baseos 803 k 2026-02-23T11:05:58.453 INFO:teuthology.orchestra.run.trial191.stdout: 2026-02-23T11:05:58.453 INFO:teuthology.orchestra.run.trial191.stdout:Transaction Summary 2026-02-23T11:05:58.454 INFO:teuthology.orchestra.run.trial191.stdout:================================================================================ 2026-02-23T11:05:58.454 INFO:teuthology.orchestra.run.trial191.stdout: 2026-02-23T11:05:58.454 INFO:teuthology.orchestra.run.trial191.stdout:Total download size: 803 k 2026-02-23T11:05:58.454 INFO:teuthology.orchestra.run.trial191.stdout:Installed size: 0 2026-02-23T11:05:58.454 INFO:teuthology.orchestra.run.trial191.stderr:Operation aborted. 2026-02-23T11:05:58.473 INFO:teuthology.orchestra.run.trial191.stdout:Is this ok [y/N]: 2026-02-23T11:05:58.474 DEBUG:teuthology.orchestra.run.trial191:> sudo yum reinstall -y kernel || true 2026-02-23T11:05:58.772 INFO:teuthology.orchestra.run.trial191.stdout:Last metadata expiration check: 0:00:01 ago on Mon 23 Feb 2026 11:05:57 AM UTC. 2026-02-23T11:05:58.851 INFO:teuthology.orchestra.run.trial191.stdout:Dependencies resolved. 2026-02-23T11:05:58.851 INFO:teuthology.orchestra.run.trial191.stdout:================================================================================ 2026-02-23T11:05:58.851 INFO:teuthology.orchestra.run.trial191.stdout: Package Architecture Version Repository Size 2026-02-23T11:05:58.851 INFO:teuthology.orchestra.run.trial191.stdout:================================================================================ 2026-02-23T11:05:58.852 INFO:teuthology.orchestra.run.trial191.stdout:Reinstalling: 2026-02-23T11:05:58.852 INFO:teuthology.orchestra.run.trial191.stdout: kernel x86_64 5.14.0-681.el9 baseos 803 k 2026-02-23T11:05:58.852 INFO:teuthology.orchestra.run.trial191.stdout: 2026-02-23T11:05:58.852 INFO:teuthology.orchestra.run.trial191.stdout:Transaction Summary 2026-02-23T11:05:58.852 INFO:teuthology.orchestra.run.trial191.stdout:================================================================================ 2026-02-23T11:05:58.852 INFO:teuthology.orchestra.run.trial191.stdout: 2026-02-23T11:05:58.852 INFO:teuthology.orchestra.run.trial191.stdout:Total download size: 803 k 2026-02-23T11:05:58.852 INFO:teuthology.orchestra.run.trial191.stdout:Installed size: 0 2026-02-23T11:05:58.852 INFO:teuthology.orchestra.run.trial191.stdout:Downloading Packages: 2026-02-23T11:05:58.983 INFO:teuthology.orchestra.run.trial191.stdout:kernel-5.14.0-681.el9.x86_64.rpm 11 MB/s | 803 kB 00:00 2026-02-23T11:05:58.984 INFO:teuthology.orchestra.run.trial191.stdout:-------------------------------------------------------------------------------- 2026-02-23T11:05:58.984 INFO:teuthology.orchestra.run.trial191.stdout:Total 5.9 MB/s | 803 kB 00:00 2026-02-23T11:05:58.994 INFO:teuthology.orchestra.run.trial191.stdout:Running transaction check 2026-02-23T11:05:59.027 INFO:teuthology.orchestra.run.trial191.stdout:Transaction check succeeded. 2026-02-23T11:05:59.028 INFO:teuthology.orchestra.run.trial191.stdout:Running transaction test 2026-02-23T11:05:59.032 INFO:teuthology.orchestra.run.trial191.stdout:Transaction test succeeded. 2026-02-23T11:05:59.032 INFO:teuthology.orchestra.run.trial191.stdout:Running transaction 2026-02-23T11:05:59.056 INFO:teuthology.orchestra.run.trial191.stdout: Preparing : 1/1 2026-02-23T11:05:59.060 INFO:teuthology.orchestra.run.trial191.stdout: Reinstalling : kernel-5.14.0-681.el9.x86_64 1/2 2026-02-23T11:05:59.060 INFO:teuthology.orchestra.run.trial191.stdout: Cleanup : kernel-5.14.0-681.el9.x86_64 2/2 2026-02-23T11:05:59.167 INFO:teuthology.orchestra.run.trial191.stdout: Running scriptlet: kernel-5.14.0-681.el9.x86_64 2/2 2026-02-23T11:05:59.167 INFO:teuthology.orchestra.run.trial191.stdout: Verifying : kernel-5.14.0-681.el9.x86_64 1/2 2026-02-23T11:05:59.223 INFO:teuthology.orchestra.run.trial191.stdout: Verifying : kernel-5.14.0-681.el9.x86_64 2/2 2026-02-23T11:05:59.224 INFO:teuthology.orchestra.run.trial191.stdout: 2026-02-23T11:05:59.224 INFO:teuthology.orchestra.run.trial191.stdout:Reinstalled: 2026-02-23T11:05:59.224 INFO:teuthology.orchestra.run.trial191.stdout: kernel-5.14.0-681.el9.x86_64 2026-02-23T11:05:59.224 INFO:teuthology.orchestra.run.trial191.stdout: 2026-02-23T11:05:59.224 INFO:teuthology.orchestra.run.trial191.stdout:Complete! 2026-02-23T11:05:59.252 DEBUG:teuthology.orchestra.run.trial191:> rpm -q kernel | sort -rV | head -n 1 2026-02-23T11:05:59.276 INFO:teuthology.orchestra.run.trial191.stdout:kernel-5.14.0-681.el9.x86_64 2026-02-23T11:05:59.278 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-681.el9.x86_64 2026-02-23T11:05:59.278 INFO:teuthology.task.kernel:Newest distro kernel installed and running 2026-02-23T11:05:59.278 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2026-02-23T11:05:59.279 DEBUG:teuthology.orchestra.run.trial191:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2026-02-23T11:05:59.342 INFO:teuthology.orchestra.run.trial191.stdout:ttyS1 2026-02-23T11:05:59.364 DEBUG:teuthology.parallel:result is None 2026-02-23T11:05:59.365 INFO:teuthology.run_tasks:Running task internal.base... 2026-02-23T11:05:59.378 INFO:teuthology.task.internal:Creating test directory... 2026-02-23T11:05:59.379 DEBUG:teuthology.orchestra.run.trial191:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2026-02-23T11:05:59.421 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2026-02-23T11:05:59.433 INFO:teuthology.run_tasks:Running task internal.archive... 2026-02-23T11:05:59.446 INFO:teuthology.task.internal:Creating archive directory... 2026-02-23T11:05:59.446 DEBUG:teuthology.orchestra.run.trial191:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2026-02-23T11:05:59.483 INFO:teuthology.run_tasks:Running task internal.coredump... 2026-02-23T11:05:59.499 INFO:teuthology.task.internal:Enabling coredump saving... 2026-02-23T11:05:59.499 DEBUG:teuthology.orchestra.run.trial191:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2026-02-23T11:05:59.542 INFO:teuthology.orchestra.run.trial191.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-23T11:05:59.550 INFO:teuthology.orchestra.run.trial191.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-23T11:05:59.552 INFO:teuthology.run_tasks:Running task internal.sudo... 2026-02-23T11:05:59.565 INFO:teuthology.task.internal:Configuring sudo... 2026-02-23T11:05:59.565 DEBUG:teuthology.orchestra.run.trial191:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2026-02-23T11:05:59.616 INFO:teuthology.run_tasks:Running task internal.syslog... 2026-02-23T11:05:59.634 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2026-02-23T11:05:59.635 DEBUG:teuthology.orchestra.run.trial191:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2026-02-23T11:05:59.673 DEBUG:teuthology.orchestra.run.trial191:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2026-02-23T11:05:59.835 DEBUG:teuthology.orchestra.run.trial191:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2026-02-23T11:05:59.859 DEBUG:teuthology.orchestra.run.trial191:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2026-02-23T11:06:00.015 DEBUG:teuthology.orchestra.run.trial191:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2026-02-23T11:06:00.038 DEBUG:teuthology.orchestra.run.trial191:> set -ex 2026-02-23T11:06:00.038 DEBUG:teuthology.orchestra.run.trial191:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2026-02-23T11:06:00.102 DEBUG:teuthology.orchestra.run.trial191:> sudo service rsyslog restart 2026-02-23T11:06:00.169 INFO:teuthology.orchestra.run.trial191.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2026-02-23T11:06:00.678 INFO:teuthology.run_tasks:Running task internal.timer... 2026-02-23T11:06:00.690 INFO:teuthology.task.internal:Starting timer... 2026-02-23T11:06:00.690 INFO:teuthology.run_tasks:Running task pcp... 2026-02-23T11:06:00.715 INFO:teuthology.run_tasks:Running task selinux... 2026-02-23T11:06:00.733 DEBUG:teuthology.orchestra.run.trial191:> sudo service auditd rotate 2026-02-23T11:06:00.762 INFO:teuthology.orchestra.run.trial191.stdout:Rotating logs: 2026-02-23T11:06:00.763 DEBUG:teuthology.task.selinux:Getting current SELinux state 2026-02-23T11:06:00.764 DEBUG:teuthology.orchestra.run.trial191:> /usr/sbin/getenforce 2026-02-23T11:06:00.820 INFO:teuthology.orchestra.run.trial191.stdout:Permissive 2026-02-23T11:06:00.820 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@trial191.front.sepia.ceph.com': 'permissive'} 2026-02-23T11:06:00.821 DEBUG:teuthology.orchestra.run.trial191:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 2026-02-23T11:06:00.885 DEBUG:teuthology.orchestra.run:got remote process result: 1 2026-02-23T11:06:00.886 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2026-02-23T11:06:00.886 DEBUG:teuthology.orchestra.run.trial191:> sudo /usr/sbin/setenforce permissive 2026-02-23T11:06:00.949 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2026-02-23T11:06:00.965 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2026-02-23T11:06:00.977 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2026-02-23T11:06:00.978 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit trial191.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2026-02-23T11:08:46.529 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@trial191.front.sepia.ceph.com')] 2026-02-23T11:08:46.530 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@trial191.front.sepia.ceph.com' 2026-02-23T11:08:46.531 DEBUG:teuthology.orchestra.connection:{'hostname': 'trial191.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2026-02-23T11:08:46.616 DEBUG:teuthology.orchestra.run.trial191:> true 2026-02-23T11:08:46.700 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@trial191.front.sepia.ceph.com' 2026-02-23T11:08:46.700 INFO:teuthology.run_tasks:Running task clock... 2026-02-23T11:08:46.717 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2026-02-23T11:08:46.717 INFO:teuthology.orchestra.run:Running command with timeout 360 2026-02-23T11:08:46.718 DEBUG:teuthology.orchestra.run.trial191:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2026-02-23T11:08:46.769 INFO:teuthology.orchestra.run.trial191.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2026-02-23T11:08:46.781 INFO:teuthology.orchestra.run.trial191.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2026-02-23T11:08:46.802 INFO:teuthology.orchestra.run.trial191.stderr:sudo: ntpd: command not found 2026-02-23T11:08:46.813 INFO:teuthology.orchestra.run.trial191.stdout:506 Cannot talk to daemon 2026-02-23T11:08:46.824 INFO:teuthology.orchestra.run.trial191.stderr:Failed to start ntp.service: Unit ntp.service not found. 2026-02-23T11:08:46.835 INFO:teuthology.orchestra.run.trial191.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2026-02-23T11:08:46.880 INFO:teuthology.orchestra.run.trial191.stderr:bash: line 1: ntpq: command not found 2026-02-23T11:08:46.881 INFO:teuthology.orchestra.run.trial191.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2026-02-23T11:08:46.882 INFO:teuthology.orchestra.run.trial191.stdout:=============================================================================== 2026-02-23T11:08:46.882 INFO:teuthology.run_tasks:Running task install... 2026-02-23T11:08:46.894 DEBUG:teuthology.task.install:project ceph 2026-02-23T11:08:46.894 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'debug', 'sha1': 'e371433fcd1390a0b82cfb18effc422c1f5d8c1d'}} 2026-02-23T11:08:46.894 DEBUG:teuthology.task.install:config {'extra_packages': {'deb': ['ceph-osd-crimson', 'ceph-osd-crimson-dbg'], 'rpm': ['ceph-osd-crimson', 'ceph-osd-crimson-debuginfo']}, 'flavor': 'debug', 'sha1': 'e371433fcd1390a0b82cfb18effc422c1f5d8c1d'} 2026-02-23T11:08:46.894 INFO:teuthology.task.install:Using flavor: debug 2026-02-23T11:08:46.897 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-osd', 'ceph-osd-classic', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-osd', 'ceph-osd-classic', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2026-02-23T11:08:46.897 INFO:teuthology.task.install:extra packages: {'deb': ['ceph-osd-crimson', 'ceph-osd-crimson-dbg'], 'rpm': ['ceph-osd-crimson', 'ceph-osd-crimson-debuginfo']} 2026-02-23T11:08:46.897 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['ceph-osd-crimson', 'ceph-osd-crimson-dbg'], 'rpm': ['ceph-osd-crimson', 'ceph-osd-crimson-debuginfo']}, 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'debug', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': 'e371433fcd1390a0b82cfb18effc422c1f5d8c1d', 'tag': None, 'wait_for_package': False} 2026-02-23T11:08:46.898 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=debug&distros=centos%2F9%2Fx86_64&sha1=e371433fcd1390a0b82cfb18effc422c1f5d8c1d 2026-02-23T11:08:47.042 INFO:teuthology.task.install.rpm:Pulling from https://3.chacra.ceph.com/r/ceph/wip-shraddhaag-enable-cephadm-tests/e371433fcd1390a0b82cfb18effc422c1f5d8c1d/centos/9/flavors/debug/ 2026-02-23T11:08:47.043 INFO:teuthology.task.install.rpm:Package version is 20.3.0-5439.ge371433f 2026-02-23T11:08:47.169 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://3.chacra.ceph.com/r/ceph/wip-shraddhaag-enable-cephadm-tests/e371433fcd1390a0b82cfb18effc422c1f5d8c1d/centos/9/flavors/debug/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://3.chacra.ceph.com/r/ceph/wip-shraddhaag-enable-cephadm-tests/e371433fcd1390a0b82cfb18effc422c1f5d8c1d/centos/9/flavors/debug/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://3.chacra.ceph.com/r/ceph/wip-shraddhaag-enable-cephadm-tests/e371433fcd1390a0b82cfb18effc422c1f5d8c1d/centos/9/flavors/debug/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2026-02-23T11:08:47.169 DEBUG:teuthology.orchestra.run.trial191:> set -ex 2026-02-23T11:08:47.169 DEBUG:teuthology.orchestra.run.trial191:> sudo dd of=/etc/yum.repos.d/ceph.repo 2026-02-23T11:08:47.195 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-osd, ceph-osd-classic, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, ceph-osd-crimson, ceph-osd-crimson-debuginfo on remote rpm x86_64 2026-02-23T11:08:47.195 DEBUG:teuthology.orchestra.run.trial191:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/e371433fcd1390a0b82cfb18effc422c1f5d8c1d/;g' /etc/yum.repos.d/ceph.repo ; fi 2026-02-23T11:08:47.260 DEBUG:teuthology.orchestra.run.trial191:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2026-02-23T11:08:47.333 DEBUG:teuthology.orchestra.run.trial191:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2026-02-23T11:08:47.397 INFO:teuthology.orchestra.run.trial191.stdout:check_obsoletes = 1 2026-02-23T11:08:47.398 DEBUG:teuthology.orchestra.run.trial191:> sudo yum clean all 2026-02-23T11:08:47.557 INFO:teuthology.orchestra.run.trial191.stdout:41 files removed 2026-02-23T11:08:47.570 DEBUG:teuthology.orchestra.run.trial191:> sudo yum -y install ceph-radosgw 2026-02-23T11:08:47.839 INFO:teuthology.orchestra.run.trial191.stdout:ceph packages for x86_64 721 kB/s | 92 kB 00:00 2026-02-23T11:08:47.981 INFO:teuthology.orchestra.run.trial191.stdout:ceph noarch packages 205 kB/s | 24 kB 00:00 2026-02-23T11:08:48.110 INFO:teuthology.orchestra.run.trial191.stdout:ceph source packages 19 kB/s | 2.1 kB 00:00 2026-02-23T11:08:48.809 INFO:teuthology.orchestra.run.trial191.stdout:CentOS Stream 9 - BaseOS 13 MB/s | 8.9 MB 00:00 2026-02-23T11:08:52.271 INFO:teuthology.orchestra.run.trial191.stdout:CentOS Stream 9 - AppStream 9.1 MB/s | 27 MB 00:02 2026-02-23T11:08:59.828 INFO:teuthology.orchestra.run.trial191.stdout:CentOS Stream 9 - CRB 1.5 MB/s | 7.8 MB 00:05 2026-02-23T11:09:00.675 INFO:teuthology.orchestra.run.trial191.stdout:CentOS Stream 9 - Extras packages 163 kB/s | 20 kB 00:00 2026-02-23T11:09:01.459 INFO:teuthology.orchestra.run.trial191.stdout:Extra Packages for Enterprise Linux 28 MB/s | 20 MB 00:00 2026-02-23T11:09:04.646 INFO:teuthology.orchestra.run.trial191.stdout:lab-extras 1.8 MB/s | 50 kB 00:00 2026-02-23T11:09:05.849 INFO:teuthology.orchestra.run.trial191.stdout:Dependencies resolved. 2026-02-23T11:09:05.850 INFO:teuthology.orchestra.run.trial191.stdout:================================================================================ 2026-02-23T11:09:05.850 INFO:teuthology.orchestra.run.trial191.stdout: Package Arch Version Repository Size 2026-02-23T11:09:05.850 INFO:teuthology.orchestra.run.trial191.stdout:================================================================================ 2026-02-23T11:09:05.851 INFO:teuthology.orchestra.run.trial191.stdout:Installing: 2026-02-23T11:09:05.851 INFO:teuthology.orchestra.run.trial191.stdout: ceph-radosgw x86_64 2:20.3.0-5439.ge371433f.el9 ceph 11 M 2026-02-23T11:09:05.851 INFO:teuthology.orchestra.run.trial191.stdout:Upgrading: 2026-02-23T11:09:05.851 INFO:teuthology.orchestra.run.trial191.stdout: librados2 x86_64 2:20.3.0-5439.ge371433f.el9 ceph 3.3 M 2026-02-23T11:09:05.851 INFO:teuthology.orchestra.run.trial191.stdout: librbd1 x86_64 2:20.3.0-5439.ge371433f.el9 ceph 2.7 M 2026-02-23T11:09:05.851 INFO:teuthology.orchestra.run.trial191.stdout:Installing dependencies: 2026-02-23T11:09:05.851 INFO:teuthology.orchestra.run.trial191.stdout: boost-program-options x86_64 1.75.0-13.el9 appstream 104 k 2026-02-23T11:09:05.852 INFO:teuthology.orchestra.run.trial191.stdout: ceph-base x86_64 2:20.3.0-5439.ge371433f.el9 ceph 5.1 M 2026-02-23T11:09:05.852 INFO:teuthology.orchestra.run.trial191.stdout: ceph-common x86_64 2:20.3.0-5439.ge371433f.el9 ceph 21 M 2026-02-23T11:09:05.852 INFO:teuthology.orchestra.run.trial191.stdout: ceph-selinux x86_64 2:20.3.0-5439.ge371433f.el9 ceph 25 k 2026-02-23T11:09:05.852 INFO:teuthology.orchestra.run.trial191.stdout: gperftools-libs x86_64 2.9.1-3.el9 epel 308 k 2026-02-23T11:09:05.852 INFO:teuthology.orchestra.run.trial191.stdout: libarrow x86_64 9.0.0-15.el9 epel 4.4 M 2026-02-23T11:09:05.852 INFO:teuthology.orchestra.run.trial191.stdout: libarrow-doc noarch 9.0.0-15.el9 epel 25 k 2026-02-23T11:09:05.852 INFO:teuthology.orchestra.run.trial191.stdout: libcephfs2 x86_64 2:20.3.0-5439.ge371433f.el9 ceph 860 k 2026-02-23T11:09:05.852 INFO:teuthology.orchestra.run.trial191.stdout: libnbd x86_64 1.20.3-4.el9 appstream 164 k 2026-02-23T11:09:05.852 INFO:teuthology.orchestra.run.trial191.stdout: liboath x86_64 2.6.12-1.el9 epel 49 k 2026-02-23T11:09:05.852 INFO:teuthology.orchestra.run.trial191.stdout: libpmemobj x86_64 1.12.1-1.el9 appstream 160 k 2026-02-23T11:09:05.852 INFO:teuthology.orchestra.run.trial191.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2026-02-23T11:09:05.852 INFO:teuthology.orchestra.run.trial191.stdout: libradosstriper1 x86_64 2:20.3.0-5439.ge371433f.el9 ceph 263 k 2026-02-23T11:09:05.853 INFO:teuthology.orchestra.run.trial191.stdout: librdkafka x86_64 1.6.1-102.el9 appstream 662 k 2026-02-23T11:09:05.853 INFO:teuthology.orchestra.run.trial191.stdout: librgw2 x86_64 2:20.3.0-5439.ge371433f.el9 ceph 5.6 M 2026-02-23T11:09:05.853 INFO:teuthology.orchestra.run.trial191.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2026-02-23T11:09:05.853 INFO:teuthology.orchestra.run.trial191.stdout: lttng-ust x86_64 2.12.0-6.el9 appstream 292 k 2026-02-23T11:09:05.853 INFO:teuthology.orchestra.run.trial191.stdout: mailcap noarch 2.1.49-5.el9 baseos 33 k 2026-02-23T11:09:05.853 INFO:teuthology.orchestra.run.trial191.stdout: parquet-libs x86_64 9.0.0-15.el9 epel 838 k 2026-02-23T11:09:05.853 INFO:teuthology.orchestra.run.trial191.stdout: pciutils x86_64 3.7.0-7.el9 baseos 93 k 2026-02-23T11:09:05.853 INFO:teuthology.orchestra.run.trial191.stdout: python3-ceph-argparse x86_64 2:20.3.0-5439.ge371433f.el9 ceph 45 k 2026-02-23T11:09:05.853 INFO:teuthology.orchestra.run.trial191.stdout: python3-ceph-common x86_64 2:20.3.0-5439.ge371433f.el9 ceph 186 k 2026-02-23T11:09:05.853 INFO:teuthology.orchestra.run.trial191.stdout: python3-cephfs x86_64 2:20.3.0-5439.ge371433f.el9 ceph 204 k 2026-02-23T11:09:05.853 INFO:teuthology.orchestra.run.trial191.stdout: python3-rados x86_64 2:20.3.0-5439.ge371433f.el9 ceph 324 k 2026-02-23T11:09:05.853 INFO:teuthology.orchestra.run.trial191.stdout: python3-rbd x86_64 2:20.3.0-5439.ge371433f.el9 ceph 305 k 2026-02-23T11:09:05.854 INFO:teuthology.orchestra.run.trial191.stdout: python3-rgw x86_64 2:20.3.0-5439.ge371433f.el9 ceph 100 k 2026-02-23T11:09:05.854 INFO:teuthology.orchestra.run.trial191.stdout: qatlib x86_64 25.08.0-2.el9 appstream 240 k 2026-02-23T11:09:05.854 INFO:teuthology.orchestra.run.trial191.stdout: qatzip-libs x86_64 1.3.1-1.el9 appstream 66 k 2026-02-23T11:09:05.854 INFO:teuthology.orchestra.run.trial191.stdout: re2 x86_64 1:20211101-20.el9 epel 191 k 2026-02-23T11:09:05.854 INFO:teuthology.orchestra.run.trial191.stdout: thrift x86_64 0.15.0-4.el9 epel 1.6 M 2026-02-23T11:09:05.854 INFO:teuthology.orchestra.run.trial191.stdout:Installing weak dependencies: 2026-02-23T11:09:05.854 INFO:teuthology.orchestra.run.trial191.stdout: qatlib-service x86_64 25.08.0-2.el9 appstream 37 k 2026-02-23T11:09:05.854 INFO:teuthology.orchestra.run.trial191.stdout: 2026-02-23T11:09:05.855 INFO:teuthology.orchestra.run.trial191.stdout:Transaction Summary 2026-02-23T11:09:05.855 INFO:teuthology.orchestra.run.trial191.stdout:================================================================================ 2026-02-23T11:09:05.855 INFO:teuthology.orchestra.run.trial191.stdout:Install 32 Packages 2026-02-23T11:09:05.855 INFO:teuthology.orchestra.run.trial191.stdout:Upgrade 2 Packages 2026-02-23T11:09:05.855 INFO:teuthology.orchestra.run.trial191.stdout: 2026-02-23T11:09:05.856 INFO:teuthology.orchestra.run.trial191.stdout:Total download size: 60 M 2026-02-23T11:09:05.856 INFO:teuthology.orchestra.run.trial191.stdout:Downloading Packages: 2026-02-23T11:09:06.963 INFO:teuthology.orchestra.run.trial191.stdout:(1/34): ceph-base-20.3.0-5439.ge371433f.el9.x86 13 MB/s | 5.1 MB 00:00 2026-02-23T11:09:07.017 INFO:teuthology.orchestra.run.trial191.stdout:(2/34): ceph-selinux-20.3.0-5439.ge371433f.el9. 466 kB/s | 25 kB 00:00 2026-02-23T11:09:07.089 INFO:teuthology.orchestra.run.trial191.stdout:(3/34): libcephfs2-20.3.0-5439.ge371433f.el9.x8 12 MB/s | 860 kB 00:00 2026-02-23T11:09:07.113 INFO:teuthology.orchestra.run.trial191.stdout:(4/34): ceph-radosgw-20.3.0-5439.ge371433f.el9. 20 MB/s | 11 MB 00:00 2026-02-23T11:09:07.131 INFO:teuthology.orchestra.run.trial191.stdout:(5/34): libradosstriper1-20.3.0-5439.ge371433f. 6.0 MB/s | 263 kB 00:00 2026-02-23T11:09:07.188 INFO:teuthology.orchestra.run.trial191.stdout:(6/34): python3-ceph-argparse-20.3.0-5439.ge371 793 kB/s | 45 kB 00:00 2026-02-23T11:09:07.252 INFO:teuthology.orchestra.run.trial191.stdout:(7/34): python3-ceph-common-20.3.0-5439.ge37143 2.9 MB/s | 186 kB 00:00 2026-02-23T11:09:07.318 INFO:teuthology.orchestra.run.trial191.stdout:(8/34): python3-cephfs-20.3.0-5439.ge371433f.el 3.0 MB/s | 204 kB 00:00 2026-02-23T11:09:07.347 INFO:teuthology.orchestra.run.trial191.stdout:(9/34): librgw2-20.3.0-5439.ge371433f.el9.x86_6 24 MB/s | 5.6 MB 00:00 2026-02-23T11:09:07.367 INFO:teuthology.orchestra.run.trial191.stdout:(10/34): python3-rados-20.3.0-5439.ge371433f.el 6.6 MB/s | 324 kB 00:00 2026-02-23T11:09:07.393 INFO:teuthology.orchestra.run.trial191.stdout:(11/34): python3-rbd-20.3.0-5439.ge371433f.el9. 6.5 MB/s | 305 kB 00:00 2026-02-23T11:09:07.410 INFO:teuthology.orchestra.run.trial191.stdout:(12/34): python3-rgw-20.3.0-5439.ge371433f.el9. 2.3 MB/s | 100 kB 00:00 2026-02-23T11:09:07.442 INFO:teuthology.orchestra.run.trial191.stdout:(13/34): ceph-common-20.3.0-5439.ge371433f.el9. 24 MB/s | 21 MB 00:00 2026-02-23T11:09:07.442 INFO:teuthology.orchestra.run.trial191.stdout:(14/34): mailcap-2.1.49-5.el9.noarch.rpm 675 kB/s | 33 kB 00:00 2026-02-23T11:09:07.478 INFO:teuthology.orchestra.run.trial191.stdout:(15/34): pciutils-3.7.0-7.el9.x86_64.rpm 1.3 MB/s | 93 kB 00:00 2026-02-23T11:09:08.113 INFO:teuthology.orchestra.run.trial191.stdout:(16/34): boost-program-options-1.75.0-13.el9.x8 155 kB/s | 104 kB 00:00 2026-02-23T11:09:08.170 INFO:teuthology.orchestra.run.trial191.stdout:(17/34): libnbd-1.20.3-4.el9.x86_64.rpm 225 kB/s | 164 kB 00:00 2026-02-23T11:09:08.198 INFO:teuthology.orchestra.run.trial191.stdout:(18/34): libpmemobj-1.12.1-1.el9.x86_64.rpm 223 kB/s | 160 kB 00:00 2026-02-23T11:09:08.692 INFO:teuthology.orchestra.run.trial191.stdout:(19/34): librabbitmq-0.11.0-7.el9.x86_64.rpm 78 kB/s | 45 kB 00:00 2026-02-23T11:09:09.028 INFO:teuthology.orchestra.run.trial191.stdout:(20/34): lttng-ust-2.12.0-6.el9.x86_64.rpm 352 kB/s | 292 kB 00:00 2026-02-23T11:09:09.068 INFO:teuthology.orchestra.run.trial191.stdout:(21/34): librdkafka-1.6.1-102.el9.x86_64.rpm 737 kB/s | 662 kB 00:00 2026-02-23T11:09:09.459 INFO:teuthology.orchestra.run.trial191.stdout:(22/34): qatlib-25.08.0-2.el9.x86_64.rpm 312 kB/s | 240 kB 00:00 2026-02-23T11:09:09.494 INFO:teuthology.orchestra.run.trial191.stdout:(23/34): gperftools-libs-2.9.1-3.el9.x86_64.rpm 8.6 MB/s | 308 kB 00:00 2026-02-23T11:09:09.520 INFO:teuthology.orchestra.run.trial191.stdout:(24/34): libarrow-9.0.0-15.el9.x86_64.rpm 171 MB/s | 4.4 MB 00:00 2026-02-23T11:09:09.524 INFO:teuthology.orchestra.run.trial191.stdout:(25/34): libarrow-doc-9.0.0-15.el9.noarch.rpm 6.0 MB/s | 25 kB 00:00 2026-02-23T11:09:09.528 INFO:teuthology.orchestra.run.trial191.stdout:(26/34): liboath-2.6.12-1.el9.x86_64.rpm 12 MB/s | 49 kB 00:00 2026-02-23T11:09:09.532 INFO:teuthology.orchestra.run.trial191.stdout:(27/34): libunwind-1.6.2-1.el9.x86_64.rpm 15 MB/s | 67 kB 00:00 2026-02-23T11:09:09.540 INFO:teuthology.orchestra.run.trial191.stdout:(28/34): parquet-libs-9.0.0-15.el9.x86_64.rpm 109 MB/s | 838 kB 00:00 2026-02-23T11:09:09.545 INFO:teuthology.orchestra.run.trial191.stdout:(29/34): re2-20211101-20.el9.x86_64.rpm 38 MB/s | 191 kB 00:00 2026-02-23T11:09:09.556 INFO:teuthology.orchestra.run.trial191.stdout:(30/34): thrift-0.15.0-4.el9.x86_64.rpm 148 MB/s | 1.6 MB 00:00 2026-02-23T11:09:09.600 INFO:teuthology.orchestra.run.trial191.stdout:(31/34): qatlib-service-25.08.0-2.el9.x86_64.rp 65 kB/s | 37 kB 00:00 2026-02-23T11:09:09.675 INFO:teuthology.orchestra.run.trial191.stdout:(32/34): librados2-20.3.0-5439.ge371433f.el9.x8 28 MB/s | 3.3 MB 00:00 2026-02-23T11:09:09.690 INFO:teuthology.orchestra.run.trial191.stdout:(33/34): qatzip-libs-1.3.1-1.el9.x86_64.rpm 107 kB/s | 66 kB 00:00 2026-02-23T11:09:09.714 INFO:teuthology.orchestra.run.trial191.stdout:(34/34): librbd1-20.3.0-5439.ge371433f.el9.x86_ 24 MB/s | 2.7 MB 00:00 2026-02-23T11:09:09.716 INFO:teuthology.orchestra.run.trial191.stdout:-------------------------------------------------------------------------------- 2026-02-23T11:09:09.716 INFO:teuthology.orchestra.run.trial191.stdout:Total 16 MB/s | 60 MB 00:03 2026-02-23T11:09:09.804 INFO:teuthology.orchestra.run.trial191.stdout:Running transaction check 2026-02-23T11:09:09.822 INFO:teuthology.orchestra.run.trial191.stdout:Transaction check succeeded. 2026-02-23T11:09:09.822 INFO:teuthology.orchestra.run.trial191.stdout:Running transaction test 2026-02-23T11:09:10.067 INFO:teuthology.orchestra.run.trial191.stdout:Transaction test succeeded. 2026-02-23T11:09:10.067 INFO:teuthology.orchestra.run.trial191.stdout:Running transaction 2026-02-23T11:09:10.349 INFO:teuthology.orchestra.run.trial191.stdout: Preparing : 1/1 2026-02-23T11:09:10.495 INFO:teuthology.orchestra.run.trial191.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/36 2026-02-23T11:09:10.499 INFO:teuthology.orchestra.run.trial191.stdout: Upgrading : librados2-2:20.3.0-5439.ge371433f.el9.x86_64 2/36 2026-02-23T11:09:10.531 INFO:teuthology.orchestra.run.trial191.stdout: Running scriptlet: librados2-2:20.3.0-5439.ge371433f.el9.x86_64 2/36 2026-02-23T11:09:10.570 INFO:teuthology.orchestra.run.trial191.stdout: Installing : python3-rados-2:20.3.0-5439.ge371433f.el9.x86_64 3/36 2026-02-23T11:09:10.571 INFO:teuthology.orchestra.run.trial191.stdout: Installing : libcephfs2-2:20.3.0-5439.ge371433f.el9.x86_64 4/36 2026-02-23T11:09:10.582 INFO:teuthology.orchestra.run.trial191.stdout: Running scriptlet: libcephfs2-2:20.3.0-5439.ge371433f.el9.x86_64 4/36 2026-02-23T11:09:10.590 INFO:teuthology.orchestra.run.trial191.stdout: Installing : liboath-2.6.12-1.el9.x86_64 5/36 2026-02-23T11:09:10.592 INFO:teuthology.orchestra.run.trial191.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 6/36 2026-02-23T11:09:10.595 INFO:teuthology.orchestra.run.trial191.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 7/36 2026-02-23T11:09:10.600 INFO:teuthology.orchestra.run.trial191.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 8/36 2026-02-23T11:09:10.612 INFO:teuthology.orchestra.run.trial191.stdout: Installing : python3-ceph-argparse-2:20.3.0-5439.ge371433f.el9. 9/36 2026-02-23T11:09:10.624 INFO:teuthology.orchestra.run.trial191.stdout: Installing : python3-cephfs-2:20.3.0-5439.ge371433f.el9.x86_64 10/36 2026-02-23T11:09:10.625 INFO:teuthology.orchestra.run.trial191.stdout: Installing : libradosstriper1-2:20.3.0-5439.ge371433f.el9.x86_6 11/36 2026-02-23T11:09:10.648 INFO:teuthology.orchestra.run.trial191.stdout: Running scriptlet: libradosstriper1-2:20.3.0-5439.ge371433f.el9.x86_6 11/36 2026-02-23T11:09:10.653 INFO:teuthology.orchestra.run.trial191.stdout: Installing : thrift-0.15.0-4.el9.x86_64 12/36 2026-02-23T11:09:10.686 INFO:teuthology.orchestra.run.trial191.stdout: Installing : re2-1:20211101-20.el9.x86_64 13/36 2026-02-23T11:09:10.689 INFO:teuthology.orchestra.run.trial191.stdout: Installing : libarrow-9.0.0-15.el9.x86_64 14/36 2026-02-23T11:09:10.696 INFO:teuthology.orchestra.run.trial191.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 15/36 2026-02-23T11:09:10.698 INFO:teuthology.orchestra.run.trial191.stdout: Installing : gperftools-libs-2.9.1-3.el9.x86_64 16/36 2026-02-23T11:09:10.703 INFO:teuthology.orchestra.run.trial191.stdout: Installing : libarrow-doc-9.0.0-15.el9.noarch 17/36 2026-02-23T11:09:10.820 INFO:teuthology.orchestra.run.trial191.stdout: Installing : libnbd-1.20.3-4.el9.x86_64 18/36 2026-02-23T11:09:10.821 INFO:teuthology.orchestra.run.trial191.stdout: Upgrading : librbd1-2:20.3.0-5439.ge371433f.el9.x86_64 19/36 2026-02-23T11:09:10.849 INFO:teuthology.orchestra.run.trial191.stdout: Running scriptlet: librbd1-2:20.3.0-5439.ge371433f.el9.x86_64 19/36 2026-02-23T11:09:10.852 INFO:teuthology.orchestra.run.trial191.stdout: Installing : python3-rbd-2:20.3.0-5439.ge371433f.el9.x86_64 20/36 2026-02-23T11:09:10.860 INFO:teuthology.orchestra.run.trial191.stdout: Installing : boost-program-options-1.75.0-13.el9.x86_64 21/36 2026-02-23T11:09:11.112 INFO:teuthology.orchestra.run.trial191.stdout: Installing : parquet-libs-9.0.0-15.el9.x86_64 22/36 2026-02-23T11:09:11.113 INFO:teuthology.orchestra.run.trial191.stdout: Installing : librgw2-2:20.3.0-5439.ge371433f.el9.x86_64 23/36 2026-02-23T11:09:11.130 INFO:teuthology.orchestra.run.trial191.stdout: Running scriptlet: librgw2-2:20.3.0-5439.ge371433f.el9.x86_64 23/36 2026-02-23T11:09:11.135 INFO:teuthology.orchestra.run.trial191.stdout: Installing : python3-rgw-2:20.3.0-5439.ge371433f.el9.x86_64 24/36 2026-02-23T11:09:11.139 INFO:teuthology.orchestra.run.trial191.stdout: Installing : pciutils-3.7.0-7.el9.x86_64 25/36 2026-02-23T11:09:11.142 INFO:teuthology.orchestra.run.trial191.stdout: Installing : qatlib-25.08.0-2.el9.x86_64 26/36 2026-02-23T11:09:11.143 INFO:teuthology.orchestra.run.trial191.stdout: Installing : qatlib-service-25.08.0-2.el9.x86_64 27/36 2026-02-23T11:09:11.162 INFO:teuthology.orchestra.run.trial191.stdout: Running scriptlet: qatlib-service-25.08.0-2.el9.x86_64 27/36 2026-02-23T11:09:11.165 INFO:teuthology.orchestra.run.trial191.stdout: Installing : qatzip-libs-1.3.1-1.el9.x86_64 28/36 2026-02-23T11:09:11.200 INFO:teuthology.orchestra.run.trial191.stdout: Installing : mailcap-2.1.49-5.el9.noarch 29/36 2026-02-23T11:09:11.202 INFO:teuthology.orchestra.run.trial191.stdout: Installing : python3-ceph-common-2:20.3.0-5439.ge371433f.el9.x8 30/36 2026-02-23T11:09:12.155 INFO:teuthology.orchestra.run.trial191.stdout: Running scriptlet: ceph-common-2:20.3.0-5439.ge371433f.el9.x86_64 31/36 2026-02-23T11:09:12.158 INFO:teuthology.orchestra.run.trial191.stdout: Installing : ceph-common-2:20.3.0-5439.ge371433f.el9.x86_64 31/36 2026-02-23T11:09:12.425 INFO:teuthology.orchestra.run.trial191.stdout: Running scriptlet: ceph-common-2:20.3.0-5439.ge371433f.el9.x86_64 31/36 2026-02-23T11:09:12.428 INFO:teuthology.orchestra.run.trial191.stdout: Installing : ceph-base-2:20.3.0-5439.ge371433f.el9.x86_64 32/36 2026-02-23T11:09:12.470 INFO:teuthology.orchestra.run.trial191.stdout: Running scriptlet: ceph-base-2:20.3.0-5439.ge371433f.el9.x86_64 32/36 2026-02-23T11:09:12.470 INFO:teuthology.orchestra.run.trial191.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2026-02-23T11:09:12.470 INFO:teuthology.orchestra.run.trial191.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2026-02-23T11:09:12.471 INFO:teuthology.orchestra.run.trial191.stdout: 2026-02-23T11:09:12.473 INFO:teuthology.orchestra.run.trial191.stdout: Installing : ceph-selinux-2:20.3.0-5439.ge371433f.el9.x86_64 33/36 2026-02-23T11:09:17.003 INFO:teuthology.orchestra.run.trial191.stdout: Running scriptlet: ceph-selinux-2:20.3.0-5439.ge371433f.el9.x86_64 33/36 2026-02-23T11:09:17.003 INFO:teuthology.orchestra.run.trial191.stdout:skipping the directory /sys 2026-02-23T11:09:17.003 INFO:teuthology.orchestra.run.trial191.stdout:skipping the directory /proc 2026-02-23T11:09:17.003 INFO:teuthology.orchestra.run.trial191.stdout:skipping the directory /mnt 2026-02-23T11:09:17.003 INFO:teuthology.orchestra.run.trial191.stdout:skipping the directory /var/tmp 2026-02-23T11:09:17.003 INFO:teuthology.orchestra.run.trial191.stdout:skipping the directory /home 2026-02-23T11:09:17.003 INFO:teuthology.orchestra.run.trial191.stdout:skipping the directory /root 2026-02-23T11:09:17.003 INFO:teuthology.orchestra.run.trial191.stdout:skipping the directory /tmp 2026-02-23T11:09:17.003 INFO:teuthology.orchestra.run.trial191.stdout: 2026-02-23T11:09:17.504 INFO:teuthology.orchestra.run.trial191.stdout: Installing : ceph-radosgw-2:20.3.0-5439.ge371433f.el9.x86_64 34/36 2026-02-23T11:09:17.527 INFO:teuthology.orchestra.run.trial191.stdout: Running scriptlet: ceph-radosgw-2:20.3.0-5439.ge371433f.el9.x86_64 34/36 2026-02-23T11:09:17.527 INFO:teuthology.orchestra.run.trial191.stdout:Glob pattern passed to enable, but globs are not supported for this. 2026-02-23T11:09:17.528 INFO:teuthology.orchestra.run.trial191.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2026-02-23T11:09:17.528 INFO:teuthology.orchestra.run.trial191.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2026-02-23T11:09:17.528 INFO:teuthology.orchestra.run.trial191.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2026-02-23T11:09:17.528 INFO:teuthology.orchestra.run.trial191.stdout: 2026-02-23T11:09:17.529 INFO:teuthology.orchestra.run.trial191.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 35/36 2026-02-23T11:09:17.541 INFO:teuthology.orchestra.run.trial191.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 35/36 2026-02-23T11:09:17.541 INFO:teuthology.orchestra.run.trial191.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 36/36 2026-02-23T11:09:17.596 INFO:teuthology.orchestra.run.trial191.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 36/36 2026-02-23T11:09:17.596 INFO:teuthology.orchestra.run.trial191.stdout:Creating group 'qat' with GID 992. 2026-02-23T11:09:17.596 INFO:teuthology.orchestra.run.trial191.stdout: 2026-02-23T11:09:17.904 INFO:teuthology.orchestra.run.trial191.stdout: Verifying : ceph-base-2:20.3.0-5439.ge371433f.el9.x86_64 1/36 2026-02-23T11:09:17.904 INFO:teuthology.orchestra.run.trial191.stdout: Verifying : ceph-common-2:20.3.0-5439.ge371433f.el9.x86_64 2/36 2026-02-23T11:09:17.904 INFO:teuthology.orchestra.run.trial191.stdout: Verifying : ceph-radosgw-2:20.3.0-5439.ge371433f.el9.x86_64 3/36 2026-02-23T11:09:17.904 INFO:teuthology.orchestra.run.trial191.stdout: Verifying : ceph-selinux-2:20.3.0-5439.ge371433f.el9.x86_64 4/36 2026-02-23T11:09:17.905 INFO:teuthology.orchestra.run.trial191.stdout: Verifying : libcephfs2-2:20.3.0-5439.ge371433f.el9.x86_64 5/36 2026-02-23T11:09:17.905 INFO:teuthology.orchestra.run.trial191.stdout: Verifying : libradosstriper1-2:20.3.0-5439.ge371433f.el9.x86_6 6/36 2026-02-23T11:09:17.905 INFO:teuthology.orchestra.run.trial191.stdout: Verifying : librgw2-2:20.3.0-5439.ge371433f.el9.x86_64 7/36 2026-02-23T11:09:17.905 INFO:teuthology.orchestra.run.trial191.stdout: Verifying : python3-ceph-argparse-2:20.3.0-5439.ge371433f.el9. 8/36 2026-02-23T11:09:17.905 INFO:teuthology.orchestra.run.trial191.stdout: Verifying : python3-ceph-common-2:20.3.0-5439.ge371433f.el9.x8 9/36 2026-02-23T11:09:17.905 INFO:teuthology.orchestra.run.trial191.stdout: Verifying : python3-cephfs-2:20.3.0-5439.ge371433f.el9.x86_64 10/36 2026-02-23T11:09:17.905 INFO:teuthology.orchestra.run.trial191.stdout: Verifying : python3-rados-2:20.3.0-5439.ge371433f.el9.x86_64 11/36 2026-02-23T11:09:17.905 INFO:teuthology.orchestra.run.trial191.stdout: Verifying : python3-rbd-2:20.3.0-5439.ge371433f.el9.x86_64 12/36 2026-02-23T11:09:17.905 INFO:teuthology.orchestra.run.trial191.stdout: Verifying : python3-rgw-2:20.3.0-5439.ge371433f.el9.x86_64 13/36 2026-02-23T11:09:17.905 INFO:teuthology.orchestra.run.trial191.stdout: Verifying : mailcap-2.1.49-5.el9.noarch 14/36 2026-02-23T11:09:17.905 INFO:teuthology.orchestra.run.trial191.stdout: Verifying : pciutils-3.7.0-7.el9.x86_64 15/36 2026-02-23T11:09:17.905 INFO:teuthology.orchestra.run.trial191.stdout: Verifying : boost-program-options-1.75.0-13.el9.x86_64 16/36 2026-02-23T11:09:17.905 INFO:teuthology.orchestra.run.trial191.stdout: Verifying : libnbd-1.20.3-4.el9.x86_64 17/36 2026-02-23T11:09:17.905 INFO:teuthology.orchestra.run.trial191.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 18/36 2026-02-23T11:09:17.905 INFO:teuthology.orchestra.run.trial191.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 19/36 2026-02-23T11:09:17.905 INFO:teuthology.orchestra.run.trial191.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 20/36 2026-02-23T11:09:17.906 INFO:teuthology.orchestra.run.trial191.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 21/36 2026-02-23T11:09:17.906 INFO:teuthology.orchestra.run.trial191.stdout: Verifying : qatlib-25.08.0-2.el9.x86_64 22/36 2026-02-23T11:09:17.906 INFO:teuthology.orchestra.run.trial191.stdout: Verifying : qatlib-service-25.08.0-2.el9.x86_64 23/36 2026-02-23T11:09:17.906 INFO:teuthology.orchestra.run.trial191.stdout: Verifying : qatzip-libs-1.3.1-1.el9.x86_64 24/36 2026-02-23T11:09:17.906 INFO:teuthology.orchestra.run.trial191.stdout: Verifying : gperftools-libs-2.9.1-3.el9.x86_64 25/36 2026-02-23T11:09:17.906 INFO:teuthology.orchestra.run.trial191.stdout: Verifying : libarrow-9.0.0-15.el9.x86_64 26/36 2026-02-23T11:09:17.906 INFO:teuthology.orchestra.run.trial191.stdout: Verifying : libarrow-doc-9.0.0-15.el9.noarch 27/36 2026-02-23T11:09:17.906 INFO:teuthology.orchestra.run.trial191.stdout: Verifying : liboath-2.6.12-1.el9.x86_64 28/36 2026-02-23T11:09:17.906 INFO:teuthology.orchestra.run.trial191.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 29/36 2026-02-23T11:09:17.906 INFO:teuthology.orchestra.run.trial191.stdout: Verifying : parquet-libs-9.0.0-15.el9.x86_64 30/36 2026-02-23T11:09:17.906 INFO:teuthology.orchestra.run.trial191.stdout: Verifying : re2-1:20211101-20.el9.x86_64 31/36 2026-02-23T11:09:17.906 INFO:teuthology.orchestra.run.trial191.stdout: Verifying : thrift-0.15.0-4.el9.x86_64 32/36 2026-02-23T11:09:17.906 INFO:teuthology.orchestra.run.trial191.stdout: Verifying : librados2-2:20.3.0-5439.ge371433f.el9.x86_64 33/36 2026-02-23T11:09:17.906 INFO:teuthology.orchestra.run.trial191.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 34/36 2026-02-23T11:09:17.906 INFO:teuthology.orchestra.run.trial191.stdout: Verifying : librbd1-2:20.3.0-5439.ge371433f.el9.x86_64 35/36 2026-02-23T11:09:17.965 INFO:teuthology.orchestra.run.trial191.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 36/36 2026-02-23T11:09:17.966 INFO:teuthology.orchestra.run.trial191.stdout: 2026-02-23T11:09:17.966 INFO:teuthology.orchestra.run.trial191.stdout:Upgraded: 2026-02-23T11:09:17.966 INFO:teuthology.orchestra.run.trial191.stdout: librados2-2:20.3.0-5439.ge371433f.el9.x86_64 2026-02-23T11:09:17.966 INFO:teuthology.orchestra.run.trial191.stdout: librbd1-2:20.3.0-5439.ge371433f.el9.x86_64 2026-02-23T11:09:17.966 INFO:teuthology.orchestra.run.trial191.stdout:Installed: 2026-02-23T11:09:17.966 INFO:teuthology.orchestra.run.trial191.stdout: boost-program-options-1.75.0-13.el9.x86_64 2026-02-23T11:09:17.968 INFO:teuthology.orchestra.run.trial191.stdout: ceph-base-2:20.3.0-5439.ge371433f.el9.x86_64 2026-02-23T11:09:17.968 INFO:teuthology.orchestra.run.trial191.stdout: ceph-common-2:20.3.0-5439.ge371433f.el9.x86_64 2026-02-23T11:09:17.968 INFO:teuthology.orchestra.run.trial191.stdout: ceph-radosgw-2:20.3.0-5439.ge371433f.el9.x86_64 2026-02-23T11:09:17.968 INFO:teuthology.orchestra.run.trial191.stdout: ceph-selinux-2:20.3.0-5439.ge371433f.el9.x86_64 2026-02-23T11:09:17.968 INFO:teuthology.orchestra.run.trial191.stdout: gperftools-libs-2.9.1-3.el9.x86_64 2026-02-23T11:09:17.968 INFO:teuthology.orchestra.run.trial191.stdout: libarrow-9.0.0-15.el9.x86_64 2026-02-23T11:09:17.968 INFO:teuthology.orchestra.run.trial191.stdout: libarrow-doc-9.0.0-15.el9.noarch 2026-02-23T11:09:17.968 INFO:teuthology.orchestra.run.trial191.stdout: libcephfs2-2:20.3.0-5439.ge371433f.el9.x86_64 2026-02-23T11:09:17.968 INFO:teuthology.orchestra.run.trial191.stdout: libnbd-1.20.3-4.el9.x86_64 2026-02-23T11:09:17.968 INFO:teuthology.orchestra.run.trial191.stdout: liboath-2.6.12-1.el9.x86_64 2026-02-23T11:09:17.968 INFO:teuthology.orchestra.run.trial191.stdout: libpmemobj-1.12.1-1.el9.x86_64 2026-02-23T11:09:17.968 INFO:teuthology.orchestra.run.trial191.stdout: librabbitmq-0.11.0-7.el9.x86_64 2026-02-23T11:09:17.968 INFO:teuthology.orchestra.run.trial191.stdout: libradosstriper1-2:20.3.0-5439.ge371433f.el9.x86_64 2026-02-23T11:09:17.968 INFO:teuthology.orchestra.run.trial191.stdout: librdkafka-1.6.1-102.el9.x86_64 2026-02-23T11:09:17.968 INFO:teuthology.orchestra.run.trial191.stdout: librgw2-2:20.3.0-5439.ge371433f.el9.x86_64 2026-02-23T11:09:17.968 INFO:teuthology.orchestra.run.trial191.stdout: libunwind-1.6.2-1.el9.x86_64 2026-02-23T11:09:17.968 INFO:teuthology.orchestra.run.trial191.stdout: lttng-ust-2.12.0-6.el9.x86_64 2026-02-23T11:09:17.968 INFO:teuthology.orchestra.run.trial191.stdout: mailcap-2.1.49-5.el9.noarch 2026-02-23T11:09:17.968 INFO:teuthology.orchestra.run.trial191.stdout: parquet-libs-9.0.0-15.el9.x86_64 2026-02-23T11:09:17.968 INFO:teuthology.orchestra.run.trial191.stdout: pciutils-3.7.0-7.el9.x86_64 2026-02-23T11:09:17.968 INFO:teuthology.orchestra.run.trial191.stdout: python3-ceph-argparse-2:20.3.0-5439.ge371433f.el9.x86_64 2026-02-23T11:09:17.968 INFO:teuthology.orchestra.run.trial191.stdout: python3-ceph-common-2:20.3.0-5439.ge371433f.el9.x86_64 2026-02-23T11:09:17.968 INFO:teuthology.orchestra.run.trial191.stdout: python3-cephfs-2:20.3.0-5439.ge371433f.el9.x86_64 2026-02-23T11:09:17.968 INFO:teuthology.orchestra.run.trial191.stdout: python3-rados-2:20.3.0-5439.ge371433f.el9.x86_64 2026-02-23T11:09:17.968 INFO:teuthology.orchestra.run.trial191.stdout: python3-rbd-2:20.3.0-5439.ge371433f.el9.x86_64 2026-02-23T11:09:17.968 INFO:teuthology.orchestra.run.trial191.stdout: python3-rgw-2:20.3.0-5439.ge371433f.el9.x86_64 2026-02-23T11:09:17.968 INFO:teuthology.orchestra.run.trial191.stdout: qatlib-25.08.0-2.el9.x86_64 2026-02-23T11:09:17.968 INFO:teuthology.orchestra.run.trial191.stdout: qatlib-service-25.08.0-2.el9.x86_64 2026-02-23T11:09:17.968 INFO:teuthology.orchestra.run.trial191.stdout: qatzip-libs-1.3.1-1.el9.x86_64 2026-02-23T11:09:17.968 INFO:teuthology.orchestra.run.trial191.stdout: re2-1:20211101-20.el9.x86_64 2026-02-23T11:09:17.969 INFO:teuthology.orchestra.run.trial191.stdout: thrift-0.15.0-4.el9.x86_64 2026-02-23T11:09:17.969 INFO:teuthology.orchestra.run.trial191.stdout: 2026-02-23T11:09:17.969 INFO:teuthology.orchestra.run.trial191.stdout:Complete! 2026-02-23T11:09:18.006 DEBUG:teuthology.orchestra.run.trial191:> sudo yum -y install ceph-test 2026-02-23T11:09:18.314 INFO:teuthology.orchestra.run.trial191.stdout:Last metadata expiration check: 0:00:14 ago on Mon 23 Feb 2026 11:09:04 AM UTC. 2026-02-23T11:09:18.396 INFO:teuthology.orchestra.run.trial191.stdout:Dependencies resolved. 2026-02-23T11:09:18.397 INFO:teuthology.orchestra.run.trial191.stdout:================================================================================ 2026-02-23T11:09:18.397 INFO:teuthology.orchestra.run.trial191.stdout: Package Arch Version Repository Size 2026-02-23T11:09:18.397 INFO:teuthology.orchestra.run.trial191.stdout:================================================================================ 2026-02-23T11:09:18.397 INFO:teuthology.orchestra.run.trial191.stdout:Installing: 2026-02-23T11:09:18.398 INFO:teuthology.orchestra.run.trial191.stdout: ceph-test x86_64 2:20.3.0-5439.ge371433f.el9 ceph 126 M 2026-02-23T11:09:18.398 INFO:teuthology.orchestra.run.trial191.stdout:Installing dependencies: 2026-02-23T11:09:18.398 INFO:teuthology.orchestra.run.trial191.stdout: hwloc-libs x86_64 2.4.1-6.el9 baseos 2.1 M 2026-02-23T11:09:18.398 INFO:teuthology.orchestra.run.trial191.stdout: libasan8 x86_64 15.2.1-7.el9 appstream 525 k 2026-02-23T11:09:18.398 INFO:teuthology.orchestra.run.trial191.stdout: libatomic x86_64 11.5.0-14.el9 baseos 23 k 2026-02-23T11:09:18.398 INFO:teuthology.orchestra.run.trial191.stdout: libcephsqlite x86_64 2:20.3.0-5439.ge371433f.el9 ceph 153 k 2026-02-23T11:09:18.398 INFO:teuthology.orchestra.run.trial191.stdout: libubsan x86_64 11.5.0-14.el9 appstream 176 k 2026-02-23T11:09:18.398 INFO:teuthology.orchestra.run.trial191.stdout: libxslt x86_64 1.1.34-12.el9 appstream 233 k 2026-02-23T11:09:18.398 INFO:teuthology.orchestra.run.trial191.stdout: lksctp-tools x86_64 1.0.19-2.el9 baseos 94 k 2026-02-23T11:09:18.398 INFO:teuthology.orchestra.run.trial191.stdout: protobuf x86_64 3.14.0-17.el9 appstream 1.0 M 2026-02-23T11:09:18.398 INFO:teuthology.orchestra.run.trial191.stdout: socat x86_64 1.7.4.1-8.el9 appstream 303 k 2026-02-23T11:09:18.398 INFO:teuthology.orchestra.run.trial191.stdout: xmlstarlet x86_64 1.6.1-20.el9 appstream 64 k 2026-02-23T11:09:18.398 INFO:teuthology.orchestra.run.trial191.stdout: yaml-cpp x86_64 0.6.3-5.el9 epel 122 k 2026-02-23T11:09:18.398 INFO:teuthology.orchestra.run.trial191.stdout: 2026-02-23T11:09:18.398 INFO:teuthology.orchestra.run.trial191.stdout:Transaction Summary 2026-02-23T11:09:18.398 INFO:teuthology.orchestra.run.trial191.stdout:================================================================================ 2026-02-23T11:09:18.398 INFO:teuthology.orchestra.run.trial191.stdout:Install 12 Packages 2026-02-23T11:09:18.398 INFO:teuthology.orchestra.run.trial191.stdout: 2026-02-23T11:09:18.398 INFO:teuthology.orchestra.run.trial191.stdout:Total download size: 131 M 2026-02-23T11:09:18.398 INFO:teuthology.orchestra.run.trial191.stdout:Installed size: 977 M 2026-02-23T11:09:18.398 INFO:teuthology.orchestra.run.trial191.stdout:Downloading Packages: 2026-02-23T11:09:19.154 INFO:teuthology.orchestra.run.trial191.stdout:(1/12): libcephsqlite-20.3.0-5439.ge371433f.el9 1.3 MB/s | 153 kB 00:00 2026-02-23T11:09:19.698 INFO:teuthology.orchestra.run.trial191.stdout:(2/12): libatomic-11.5.0-14.el9.x86_64.rpm 42 kB/s | 23 kB 00:00 2026-02-23T11:09:20.060 INFO:teuthology.orchestra.run.trial191.stdout:(3/12): hwloc-libs-2.4.1-6.el9.x86_64.rpm 2.1 MB/s | 2.1 MB 00:01 2026-02-23T11:09:20.125 INFO:teuthology.orchestra.run.trial191.stdout:(4/12): libasan8-15.2.1-7.el9.x86_64.rpm 7.9 MB/s | 525 kB 00:00 2026-02-23T11:09:20.147 INFO:teuthology.orchestra.run.trial191.stdout:(5/12): libubsan-11.5.0-14.el9.x86_64.rpm 7.8 MB/s | 176 kB 00:00 2026-02-23T11:09:20.169 INFO:teuthology.orchestra.run.trial191.stdout:(6/12): libxslt-1.1.34-12.el9.x86_64.rpm 10 MB/s | 233 kB 00:00 2026-02-23T11:09:20.200 INFO:teuthology.orchestra.run.trial191.stdout:(7/12): protobuf-3.14.0-17.el9.x86_64.rpm 32 MB/s | 1.0 MB 00:00 2026-02-23T11:09:20.228 INFO:teuthology.orchestra.run.trial191.stdout:(8/12): socat-1.7.4.1-8.el9.x86_64.rpm 11 MB/s | 303 kB 00:00 2026-02-23T11:09:20.258 INFO:teuthology.orchestra.run.trial191.stdout:(9/12): xmlstarlet-1.6.1-20.el9.x86_64.rpm 2.1 MB/s | 64 kB 00:00 2026-02-23T11:09:20.354 INFO:teuthology.orchestra.run.trial191.stdout:(10/12): lksctp-tools-1.0.19-2.el9.x86_64.rpm 143 kB/s | 94 kB 00:00 2026-02-23T11:09:20.535 INFO:teuthology.orchestra.run.trial191.stdout:(11/12): yaml-cpp-0.6.3-5.el9.x86_64.rpm 439 kB/s | 122 kB 00:00 2026-02-23T11:09:22.087 INFO:teuthology.orchestra.run.trial191.stdout:(12/12): ceph-test-20.3.0-5439.ge371433f.el9.x8 41 MB/s | 126 MB 00:03 2026-02-23T11:09:22.089 INFO:teuthology.orchestra.run.trial191.stdout:-------------------------------------------------------------------------------- 2026-02-23T11:09:22.089 INFO:teuthology.orchestra.run.trial191.stdout:Total 35 MB/s | 131 MB 00:03 2026-02-23T11:09:22.171 INFO:teuthology.orchestra.run.trial191.stdout:Running transaction check 2026-02-23T11:09:22.182 INFO:teuthology.orchestra.run.trial191.stdout:Transaction check succeeded. 2026-02-23T11:09:22.182 INFO:teuthology.orchestra.run.trial191.stdout:Running transaction test 2026-02-23T11:09:22.612 INFO:teuthology.orchestra.run.trial191.stdout:Transaction test succeeded. 2026-02-23T11:09:22.612 INFO:teuthology.orchestra.run.trial191.stdout:Running transaction 2026-02-23T11:09:23.066 INFO:teuthology.orchestra.run.trial191.stdout: Preparing : 1/1 2026-02-23T11:09:23.074 INFO:teuthology.orchestra.run.trial191.stdout: Installing : yaml-cpp-0.6.3-5.el9.x86_64 1/12 2026-02-23T11:09:23.083 INFO:teuthology.orchestra.run.trial191.stdout: Installing : socat-1.7.4.1-8.el9.x86_64 2/12 2026-02-23T11:09:23.088 INFO:teuthology.orchestra.run.trial191.stdout: Installing : protobuf-3.14.0-17.el9.x86_64 3/12 2026-02-23T11:09:23.092 INFO:teuthology.orchestra.run.trial191.stdout: Installing : libxslt-1.1.34-12.el9.x86_64 4/12 2026-02-23T11:09:23.094 INFO:teuthology.orchestra.run.trial191.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 5/12 2026-02-23T11:09:23.099 INFO:teuthology.orchestra.run.trial191.stdout: Installing : libubsan-11.5.0-14.el9.x86_64 6/12 2026-02-23T11:09:23.100 INFO:teuthology.orchestra.run.trial191.stdout: Installing : libasan8-15.2.1-7.el9.x86_64 7/12 2026-02-23T11:09:23.116 INFO:teuthology.orchestra.run.trial191.stdout: Running scriptlet: libasan8-15.2.1-7.el9.x86_64 7/12 2026-02-23T11:09:23.118 INFO:teuthology.orchestra.run.trial191.stdout: Installing : lksctp-tools-1.0.19-2.el9.x86_64 8/12 2026-02-23T11:09:23.128 INFO:teuthology.orchestra.run.trial191.stdout: Installing : libatomic-11.5.0-14.el9.x86_64 9/12 2026-02-23T11:09:23.136 INFO:teuthology.orchestra.run.trial191.stdout: Installing : hwloc-libs-2.4.1-6.el9.x86_64 10/12 2026-02-23T11:09:23.137 INFO:teuthology.orchestra.run.trial191.stdout: Installing : libcephsqlite-2:20.3.0-5439.ge371433f.el9.x86_64 11/12 2026-02-23T11:09:30.117 INFO:teuthology.orchestra.run.trial191.stdout: Running scriptlet: libcephsqlite-2:20.3.0-5439.ge371433f.el9.x86_64 11/12 2026-02-23T11:09:30.134 INFO:teuthology.orchestra.run.trial191.stdout: Installing : ceph-test-2:20.3.0-5439.ge371433f.el9.x86_64 12/12 2026-02-23T11:09:30.933 INFO:teuthology.orchestra.run.trial191.stdout: Running scriptlet: ceph-test-2:20.3.0-5439.ge371433f.el9.x86_64 12/12 2026-02-23T11:09:30.933 INFO:teuthology.orchestra.run.trial191.stdout: Verifying : ceph-test-2:20.3.0-5439.ge371433f.el9.x86_64 1/12 2026-02-23T11:09:30.933 INFO:teuthology.orchestra.run.trial191.stdout: Verifying : libcephsqlite-2:20.3.0-5439.ge371433f.el9.x86_64 2/12 2026-02-23T11:09:30.933 INFO:teuthology.orchestra.run.trial191.stdout: Verifying : hwloc-libs-2.4.1-6.el9.x86_64 3/12 2026-02-23T11:09:30.934 INFO:teuthology.orchestra.run.trial191.stdout: Verifying : libatomic-11.5.0-14.el9.x86_64 4/12 2026-02-23T11:09:30.934 INFO:teuthology.orchestra.run.trial191.stdout: Verifying : lksctp-tools-1.0.19-2.el9.x86_64 5/12 2026-02-23T11:09:30.934 INFO:teuthology.orchestra.run.trial191.stdout: Verifying : libasan8-15.2.1-7.el9.x86_64 6/12 2026-02-23T11:09:30.934 INFO:teuthology.orchestra.run.trial191.stdout: Verifying : libubsan-11.5.0-14.el9.x86_64 7/12 2026-02-23T11:09:30.934 INFO:teuthology.orchestra.run.trial191.stdout: Verifying : libxslt-1.1.34-12.el9.x86_64 8/12 2026-02-23T11:09:30.934 INFO:teuthology.orchestra.run.trial191.stdout: Verifying : protobuf-3.14.0-17.el9.x86_64 9/12 2026-02-23T11:09:30.934 INFO:teuthology.orchestra.run.trial191.stdout: Verifying : socat-1.7.4.1-8.el9.x86_64 10/12 2026-02-23T11:09:30.934 INFO:teuthology.orchestra.run.trial191.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 11/12 2026-02-23T11:09:30.989 INFO:teuthology.orchestra.run.trial191.stdout: Verifying : yaml-cpp-0.6.3-5.el9.x86_64 12/12 2026-02-23T11:09:30.989 INFO:teuthology.orchestra.run.trial191.stdout: 2026-02-23T11:09:30.990 INFO:teuthology.orchestra.run.trial191.stdout:Installed: 2026-02-23T11:09:30.990 INFO:teuthology.orchestra.run.trial191.stdout: ceph-test-2:20.3.0-5439.ge371433f.el9.x86_64 2026-02-23T11:09:30.990 INFO:teuthology.orchestra.run.trial191.stdout: hwloc-libs-2.4.1-6.el9.x86_64 2026-02-23T11:09:30.990 INFO:teuthology.orchestra.run.trial191.stdout: libasan8-15.2.1-7.el9.x86_64 2026-02-23T11:09:30.990 INFO:teuthology.orchestra.run.trial191.stdout: libatomic-11.5.0-14.el9.x86_64 2026-02-23T11:09:30.990 INFO:teuthology.orchestra.run.trial191.stdout: libcephsqlite-2:20.3.0-5439.ge371433f.el9.x86_64 2026-02-23T11:09:30.990 INFO:teuthology.orchestra.run.trial191.stdout: libubsan-11.5.0-14.el9.x86_64 2026-02-23T11:09:30.990 INFO:teuthology.orchestra.run.trial191.stdout: libxslt-1.1.34-12.el9.x86_64 2026-02-23T11:09:30.990 INFO:teuthology.orchestra.run.trial191.stdout: lksctp-tools-1.0.19-2.el9.x86_64 2026-02-23T11:09:30.990 INFO:teuthology.orchestra.run.trial191.stdout: protobuf-3.14.0-17.el9.x86_64 2026-02-23T11:09:30.990 INFO:teuthology.orchestra.run.trial191.stdout: socat-1.7.4.1-8.el9.x86_64 2026-02-23T11:09:30.990 INFO:teuthology.orchestra.run.trial191.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2026-02-23T11:09:30.990 INFO:teuthology.orchestra.run.trial191.stdout: yaml-cpp-0.6.3-5.el9.x86_64 2026-02-23T11:09:30.990 INFO:teuthology.orchestra.run.trial191.stdout: 2026-02-23T11:09:30.990 INFO:teuthology.orchestra.run.trial191.stdout:Complete! 2026-02-23T11:09:31.033 DEBUG:teuthology.orchestra.run.trial191:> sudo yum -y install ceph 2026-02-23T11:09:31.338 INFO:teuthology.orchestra.run.trial191.stdout:Last metadata expiration check: 0:00:27 ago on Mon 23 Feb 2026 11:09:04 AM UTC. 2026-02-23T11:09:31.421 INFO:teuthology.orchestra.run.trial191.stdout:Dependencies resolved. 2026-02-23T11:09:31.423 INFO:teuthology.orchestra.run.trial191.stdout:================================================================================ 2026-02-23T11:09:31.423 INFO:teuthology.orchestra.run.trial191.stdout: Package Arch Version Repository Size 2026-02-23T11:09:31.423 INFO:teuthology.orchestra.run.trial191.stdout:================================================================================ 2026-02-23T11:09:31.423 INFO:teuthology.orchestra.run.trial191.stdout:Installing: 2026-02-23T11:09:31.423 INFO:teuthology.orchestra.run.trial191.stdout: ceph x86_64 2:20.3.0-5439.ge371433f.el9 ceph 6.5 k 2026-02-23T11:09:31.423 INFO:teuthology.orchestra.run.trial191.stdout:Installing dependencies: 2026-02-23T11:09:31.423 INFO:teuthology.orchestra.run.trial191.stdout: ceph-mds x86_64 2:20.3.0-5439.ge371433f.el9 ceph 2.0 M 2026-02-23T11:09:31.423 INFO:teuthology.orchestra.run.trial191.stdout: ceph-mgr x86_64 2:20.3.0-5439.ge371433f.el9 ceph 937 k 2026-02-23T11:09:31.423 INFO:teuthology.orchestra.run.trial191.stdout: ceph-mgr-modules-core noarch 2:20.3.0-5439.ge371433f.el9 ceph-noarch 298 k 2026-02-23T11:09:31.423 INFO:teuthology.orchestra.run.trial191.stdout: ceph-mon x86_64 2:20.3.0-5439.ge371433f.el9 ceph 4.9 M 2026-02-23T11:09:31.423 INFO:teuthology.orchestra.run.trial191.stdout: ceph-osd x86_64 2:20.3.0-5439.ge371433f.el9 ceph 164 k 2026-02-23T11:09:31.423 INFO:teuthology.orchestra.run.trial191.stdout: ceph-osd-classic x86_64 2:20.3.0-5439.ge371433f.el9 ceph 15 M 2026-02-23T11:09:31.423 INFO:teuthology.orchestra.run.trial191.stdout: ceph-osd-crimson x86_64 2:20.3.0-5439.ge371433f.el9 ceph 87 M 2026-02-23T11:09:31.423 INFO:teuthology.orchestra.run.trial191.stdout: ledmon-libs x86_64 1.1.0-3.el9 baseos 40 k 2026-02-23T11:09:31.423 INFO:teuthology.orchestra.run.trial191.stdout: libconfig x86_64 1.7.2-9.el9 baseos 72 k 2026-02-23T11:09:31.423 INFO:teuthology.orchestra.run.trial191.stdout: libstoragemgmt x86_64 1.10.1-1.el9 appstream 246 k 2026-02-23T11:09:31.423 INFO:teuthology.orchestra.run.trial191.stdout: lua x86_64 5.4.4-4.el9 appstream 188 k 2026-02-23T11:09:31.423 INFO:teuthology.orchestra.run.trial191.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2026-02-23T11:09:31.423 INFO:teuthology.orchestra.run.trial191.stdout: luarocks noarch 3.9.2-5.el9 epel 151 k 2026-02-23T11:09:31.423 INFO:teuthology.orchestra.run.trial191.stdout: python3-autocommand noarch 2.2.2-8.el9 epel 29 k 2026-02-23T11:09:31.423 INFO:teuthology.orchestra.run.trial191.stdout: python3-backports-tarfile noarch 1.2.0-1.el9 epel 60 k 2026-02-23T11:09:31.423 INFO:teuthology.orchestra.run.trial191.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2026-02-23T11:09:31.423 INFO:teuthology.orchestra.run.trial191.stdout: python3-cffi x86_64 1.14.5-5.el9 baseos 253 k 2026-02-23T11:09:31.423 INFO:teuthology.orchestra.run.trial191.stdout: python3-cheroot noarch 10.0.1-4.el9 epel 173 k 2026-02-23T11:09:31.423 INFO:teuthology.orchestra.run.trial191.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2026-02-23T11:09:31.423 INFO:teuthology.orchestra.run.trial191.stdout: python3-cryptography x86_64 36.0.1-5.el9 baseos 1.2 M 2026-02-23T11:09:31.424 INFO:teuthology.orchestra.run.trial191.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2026-02-23T11:09:31.424 INFO:teuthology.orchestra.run.trial191.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2026-02-23T11:09:31.424 INFO:teuthology.orchestra.run.trial191.stdout: python3-jaraco-collections 2026-02-23T11:09:31.424 INFO:teuthology.orchestra.run.trial191.stdout: noarch 3.0.0-8.el9 epel 23 k 2026-02-23T11:09:31.424 INFO:teuthology.orchestra.run.trial191.stdout: python3-jaraco-context noarch 6.0.1-3.el9 epel 20 k 2026-02-23T11:09:31.424 INFO:teuthology.orchestra.run.trial191.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2026-02-23T11:09:31.424 INFO:teuthology.orchestra.run.trial191.stdout: python3-jaraco-text noarch 4.0.0-2.el9 epel 26 k 2026-02-23T11:09:31.424 INFO:teuthology.orchestra.run.trial191.stdout: python3-libstoragemgmt x86_64 1.10.1-1.el9 appstream 177 k 2026-02-23T11:09:31.424 INFO:teuthology.orchestra.run.trial191.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2026-02-23T11:09:31.424 INFO:teuthology.orchestra.run.trial191.stdout: python3-packaging noarch 20.9-5.el9 appstream 77 k 2026-02-23T11:09:31.424 INFO:teuthology.orchestra.run.trial191.stdout: python3-ply noarch 3.11-14.el9 baseos 106 k 2026-02-23T11:09:31.424 INFO:teuthology.orchestra.run.trial191.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2026-02-23T11:09:31.424 INFO:teuthology.orchestra.run.trial191.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2026-02-23T11:09:31.424 INFO:teuthology.orchestra.run.trial191.stdout: python3-pycparser noarch 2.20-6.el9 baseos 135 k 2026-02-23T11:09:31.424 INFO:teuthology.orchestra.run.trial191.stdout: python3-pyparsing noarch 2.4.7-9.el9 baseos 150 k 2026-02-23T11:09:31.424 INFO:teuthology.orchestra.run.trial191.stdout: python3-requests noarch 2.25.1-10.el9 baseos 126 k 2026-02-23T11:09:31.424 INFO:teuthology.orchestra.run.trial191.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2026-02-23T11:09:31.424 INFO:teuthology.orchestra.run.trial191.stdout: python3-toml noarch 0.10.2-6.el9 appstream 42 k 2026-02-23T11:09:31.424 INFO:teuthology.orchestra.run.trial191.stdout: python3-urllib3 noarch 1.26.5-7.el9 baseos 218 k 2026-02-23T11:09:31.424 INFO:teuthology.orchestra.run.trial191.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2026-02-23T11:09:31.424 INFO:teuthology.orchestra.run.trial191.stdout: unzip x86_64 6.0-59.el9 baseos 182 k 2026-02-23T11:09:31.424 INFO:teuthology.orchestra.run.trial191.stdout: zip x86_64 3.0-35.el9 baseos 266 k 2026-02-23T11:09:31.424 INFO:teuthology.orchestra.run.trial191.stdout: 2026-02-23T11:09:31.424 INFO:teuthology.orchestra.run.trial191.stdout:Transaction Summary 2026-02-23T11:09:31.424 INFO:teuthology.orchestra.run.trial191.stdout:================================================================================ 2026-02-23T11:09:31.424 INFO:teuthology.orchestra.run.trial191.stdout:Install 42 Packages 2026-02-23T11:09:31.424 INFO:teuthology.orchestra.run.trial191.stdout: 2026-02-23T11:09:31.424 INFO:teuthology.orchestra.run.trial191.stdout:Total download size: 114 M 2026-02-23T11:09:31.425 INFO:teuthology.orchestra.run.trial191.stdout:Installed size: 961 M 2026-02-23T11:09:31.425 INFO:teuthology.orchestra.run.trial191.stdout:Downloading Packages: 2026-02-23T11:09:31.993 INFO:teuthology.orchestra.run.trial191.stdout:(1/42): ceph-20.3.0-5439.ge371433f.el9.x86_64.r 26 kB/s | 6.5 kB 00:00 2026-02-23T11:09:32.497 INFO:teuthology.orchestra.run.trial191.stdout:(2/42): ceph-mgr-20.3.0-5439.ge371433f.el9.x86_ 1.2 MB/s | 937 kB 00:00 2026-02-23T11:09:32.612 INFO:teuthology.orchestra.run.trial191.stdout:(3/42): ceph-osd-20.3.0-5439.ge371433f.el9.x86_ 1.4 MB/s | 164 kB 00:00 2026-02-23T11:09:32.667 INFO:teuthology.orchestra.run.trial191.stdout:(4/42): ceph-mds-20.3.0-5439.ge371433f.el9.x86_ 2.2 MB/s | 2.0 MB 00:00 2026-02-23T11:09:34.077 INFO:teuthology.orchestra.run.trial191.stdout:(5/42): ceph-mon-20.3.0-5439.ge371433f.el9.x86_ 2.3 MB/s | 4.9 MB 00:02 2026-02-23T11:09:34.278 INFO:teuthology.orchestra.run.trial191.stdout:(6/42): ceph-mgr-modules-core-20.3.0-5439.ge371 1.4 MB/s | 298 kB 00:00 2026-02-23T11:09:34.834 INFO:teuthology.orchestra.run.trial191.stdout:(7/42): ledmon-libs-1.1.0-3.el9.x86_64.rpm 73 kB/s | 40 kB 00:00 2026-02-23T11:09:35.069 INFO:teuthology.orchestra.run.trial191.stdout:(8/42): libconfig-1.7.2-9.el9.x86_64.rpm 306 kB/s | 72 kB 00:00 2026-02-23T11:09:35.384 INFO:teuthology.orchestra.run.trial191.stdout:(9/42): python3-cffi-1.14.5-5.el9.x86_64.rpm 805 kB/s | 253 kB 00:00 2026-02-23T11:09:35.700 INFO:teuthology.orchestra.run.trial191.stdout:(10/42): python3-cryptography-36.0.1-5.el9.x86_ 3.9 MB/s | 1.2 MB 00:00 2026-02-23T11:09:35.860 INFO:teuthology.orchestra.run.trial191.stdout:(11/42): python3-ply-3.11-14.el9.noarch.rpm 667 kB/s | 106 kB 00:00 2026-02-23T11:09:36.020 INFO:teuthology.orchestra.run.trial191.stdout:(12/42): python3-pycparser-2.20-6.el9.noarch.rp 845 kB/s | 135 kB 00:00 2026-02-23T11:09:36.179 INFO:teuthology.orchestra.run.trial191.stdout:(13/42): python3-pyparsing-2.4.7-9.el9.noarch.r 947 kB/s | 150 kB 00:00 2026-02-23T11:09:36.339 INFO:teuthology.orchestra.run.trial191.stdout:(14/42): python3-requests-2.25.1-10.el9.noarch. 795 kB/s | 126 kB 00:00 2026-02-23T11:09:36.498 INFO:teuthology.orchestra.run.trial191.stdout:(15/42): python3-urllib3-1.26.5-7.el9.noarch.rp 1.3 MB/s | 218 kB 00:00 2026-02-23T11:09:36.657 INFO:teuthology.orchestra.run.trial191.stdout:(16/42): unzip-6.0-59.el9.x86_64.rpm 1.1 MB/s | 182 kB 00:00 2026-02-23T11:09:36.815 INFO:teuthology.orchestra.run.trial191.stdout:(17/42): zip-3.0-35.el9.x86_64.rpm 1.6 MB/s | 266 kB 00:00 2026-02-23T11:09:36.873 INFO:teuthology.orchestra.run.trial191.stdout:(18/42): libstoragemgmt-1.10.1-1.el9.x86_64.rpm 4.2 MB/s | 246 kB 00:00 2026-02-23T11:09:36.888 INFO:teuthology.orchestra.run.trial191.stdout:(19/42): lua-5.4.4-4.el9.x86_64.rpm 13 MB/s | 188 kB 00:00 2026-02-23T11:09:36.901 INFO:teuthology.orchestra.run.trial191.stdout:(20/42): python3-libstoragemgmt-1.10.1-1.el9.x8 13 MB/s | 177 kB 00:00 2026-02-23T11:09:36.912 INFO:teuthology.orchestra.run.trial191.stdout:(21/42): python3-packaging-20.9-5.el9.noarch.rp 6.8 MB/s | 77 kB 00:00 2026-02-23T11:09:36.923 INFO:teuthology.orchestra.run.trial191.stdout:(22/42): python3-toml-0.10.2-6.el9.noarch.rpm 3.9 MB/s | 42 kB 00:00 2026-02-23T11:09:36.933 INFO:teuthology.orchestra.run.trial191.stdout:(23/42): lua-devel-5.4.4-4.el9.x86_64.rpm 2.1 MB/s | 22 kB 00:00 2026-02-23T11:09:36.962 INFO:teuthology.orchestra.run.trial191.stdout:(24/42): luarocks-3.9.2-5.el9.noarch.rpm 5.2 MB/s | 151 kB 00:00 2026-02-23T11:09:36.966 INFO:teuthology.orchestra.run.trial191.stdout:(25/42): python3-autocommand-2.2.2-8.el9.noarch 8.0 MB/s | 29 kB 00:00 2026-02-23T11:09:36.970 INFO:teuthology.orchestra.run.trial191.stdout:(26/42): python3-backports-tarfile-1.2.0-1.el9. 16 MB/s | 60 kB 00:00 2026-02-23T11:09:36.973 INFO:teuthology.orchestra.run.trial191.stdout:(27/42): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 12 MB/s | 43 kB 00:00 2026-02-23T11:09:36.980 INFO:teuthology.orchestra.run.trial191.stdout:(28/42): python3-cheroot-10.0.1-4.el9.noarch.rp 26 MB/s | 173 kB 00:00 2026-02-23T11:09:36.987 INFO:teuthology.orchestra.run.trial191.stdout:(29/42): python3-cherrypy-18.6.1-2.el9.noarch.r 51 MB/s | 358 kB 00:00 2026-02-23T11:09:36.991 INFO:teuthology.orchestra.run.trial191.stdout:(30/42): python3-jaraco-8.2.1-3.el9.noarch.rpm 3.0 MB/s | 11 kB 00:00 2026-02-23T11:09:36.994 INFO:teuthology.orchestra.run.trial191.stdout:(31/42): python3-jaraco-classes-3.2.1-5.el9.noa 5.2 MB/s | 18 kB 00:00 2026-02-23T11:09:36.998 INFO:teuthology.orchestra.run.trial191.stdout:(32/42): python3-jaraco-collections-3.0.0-8.el9 6.4 MB/s | 23 kB 00:00 2026-02-23T11:09:37.001 INFO:teuthology.orchestra.run.trial191.stdout:(33/42): python3-jaraco-context-6.0.1-3.el9.noa 5.8 MB/s | 20 kB 00:00 2026-02-23T11:09:37.005 INFO:teuthology.orchestra.run.trial191.stdout:(34/42): python3-jaraco-functools-3.5.0-2.el9.n 5.7 MB/s | 19 kB 00:00 2026-02-23T11:09:37.008 INFO:teuthology.orchestra.run.trial191.stdout:(35/42): python3-jaraco-text-4.0.0-2.el9.noarch 7.7 MB/s | 26 kB 00:00 2026-02-23T11:09:37.012 INFO:teuthology.orchestra.run.trial191.stdout:(36/42): python3-more-itertools-8.12.0-2.el9.no 21 MB/s | 79 kB 00:00 2026-02-23T11:09:37.015 INFO:teuthology.orchestra.run.trial191.stdout:(37/42): python3-portend-3.1.0-2.el9.noarch.rpm 4.6 MB/s | 16 kB 00:00 2026-02-23T11:09:37.020 INFO:teuthology.orchestra.run.trial191.stdout:(38/42): python3-pyOpenSSL-21.0.0-1.el9.noarch. 24 MB/s | 90 kB 00:00 2026-02-23T11:09:37.023 INFO:teuthology.orchestra.run.trial191.stdout:(39/42): python3-tempora-5.0.0-2.el9.noarch.rpm 10 MB/s | 36 kB 00:00 2026-02-23T11:09:37.027 INFO:teuthology.orchestra.run.trial191.stdout:(40/42): python3-zc-lockfile-2.0-10.el9.noarch. 5.6 MB/s | 20 kB 00:00 2026-02-23T11:09:37.165 INFO:teuthology.orchestra.run.trial191.stdout:(41/42): ceph-osd-classic-20.3.0-5439.ge371433f 3.2 MB/s | 15 MB 00:04 2026-02-23T11:09:47.225 INFO:teuthology.orchestra.run.trial191.stdout:(42/42): ceph-osd-crimson-20.3.0-5439.ge371433f 6.0 MB/s | 87 MB 00:14 2026-02-23T11:09:47.228 INFO:teuthology.orchestra.run.trial191.stdout:-------------------------------------------------------------------------------- 2026-02-23T11:09:47.228 INFO:teuthology.orchestra.run.trial191.stdout:Total 7.2 MB/s | 114 MB 00:15 2026-02-23T11:09:47.373 INFO:teuthology.orchestra.run.trial191.stdout:Running transaction check 2026-02-23T11:09:47.393 INFO:teuthology.orchestra.run.trial191.stdout:Transaction check succeeded. 2026-02-23T11:09:47.393 INFO:teuthology.orchestra.run.trial191.stdout:Running transaction test 2026-02-23T11:09:47.786 INFO:teuthology.orchestra.run.trial191.stdout:Transaction test succeeded. 2026-02-23T11:09:47.787 INFO:teuthology.orchestra.run.trial191.stdout:Running transaction 2026-02-23T11:09:48.222 INFO:teuthology.orchestra.run.trial191.stdout: Preparing : 1/1 2026-02-23T11:09:48.224 INFO:teuthology.orchestra.run.trial191.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/42 2026-02-23T11:09:48.229 INFO:teuthology.orchestra.run.trial191.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/42 2026-02-23T11:09:48.234 INFO:teuthology.orchestra.run.trial191.stdout: Installing : lua-5.4.4-4.el9.x86_64 3/42 2026-02-23T11:09:48.236 INFO:teuthology.orchestra.run.trial191.stdout: Installing : unzip-6.0-59.el9.x86_64 4/42 2026-02-23T11:09:48.242 INFO:teuthology.orchestra.run.trial191.stdout: Installing : ledmon-libs-1.1.0-3.el9.x86_64 5/42 2026-02-23T11:09:48.266 INFO:teuthology.orchestra.run.trial191.stdout: Installing : zip-3.0-35.el9.x86_64 6/42 2026-02-23T11:09:48.270 INFO:teuthology.orchestra.run.trial191.stdout: Installing : luarocks-3.9.2-5.el9.noarch 7/42 2026-02-23T11:09:48.274 INFO:teuthology.orchestra.run.trial191.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 8/42 2026-02-23T11:09:48.279 INFO:teuthology.orchestra.run.trial191.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 9/42 2026-02-23T11:09:48.285 INFO:teuthology.orchestra.run.trial191.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 10/42 2026-02-23T11:09:48.289 INFO:teuthology.orchestra.run.trial191.stdout: Installing : python3-backports-tarfile-1.2.0-1.el9.noarch 11/42 2026-02-23T11:09:48.295 INFO:teuthology.orchestra.run.trial191.stdout: Installing : python3-jaraco-context-6.0.1-3.el9.noarch 12/42 2026-02-23T11:09:48.300 INFO:teuthology.orchestra.run.trial191.stdout: Installing : python3-autocommand-2.2.2-8.el9.noarch 13/42 2026-02-23T11:09:48.304 INFO:teuthology.orchestra.run.trial191.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 14/42 2026-02-23T11:09:48.310 INFO:teuthology.orchestra.run.trial191.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 15/42 2026-02-23T11:09:48.313 INFO:teuthology.orchestra.run.trial191.stdout: Installing : python3-jaraco-text-4.0.0-2.el9.noarch 16/42 2026-02-23T11:09:48.320 INFO:teuthology.orchestra.run.trial191.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 17/42 2026-02-23T11:09:48.323 INFO:teuthology.orchestra.run.trial191.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 18/42 2026-02-23T11:09:48.346 INFO:teuthology.orchestra.run.trial191.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 19/42 2026-02-23T11:09:48.359 INFO:teuthology.orchestra.run.trial191.stdout: Installing : python3-urllib3-1.26.5-7.el9.noarch 20/42 2026-02-23T11:09:48.364 INFO:teuthology.orchestra.run.trial191.stdout: Installing : python3-requests-2.25.1-10.el9.noarch 21/42 2026-02-23T11:09:48.373 INFO:teuthology.orchestra.run.trial191.stdout: Installing : python3-pyparsing-2.4.7-9.el9.noarch 22/42 2026-02-23T11:09:48.380 INFO:teuthology.orchestra.run.trial191.stdout: Installing : python3-packaging-20.9-5.el9.noarch 23/42 2026-02-23T11:09:48.393 INFO:teuthology.orchestra.run.trial191.stdout: Installing : python3-ply-3.11-14.el9.noarch 24/42 2026-02-23T11:09:48.409 INFO:teuthology.orchestra.run.trial191.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 25/42 2026-02-23T11:09:48.484 INFO:teuthology.orchestra.run.trial191.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 26/42 2026-02-23T11:09:48.492 INFO:teuthology.orchestra.run.trial191.stdout: Installing : python3-cryptography-36.0.1-5.el9.x86_64 27/42 2026-02-23T11:09:48.515 INFO:teuthology.orchestra.run.trial191.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 28/42 2026-02-23T11:09:48.545 INFO:teuthology.orchestra.run.trial191.stdout: Installing : python3-cheroot-10.0.1-4.el9.noarch 29/42 2026-02-23T11:09:48.550 INFO:teuthology.orchestra.run.trial191.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 30/42 2026-02-23T11:09:48.602 INFO:teuthology.orchestra.run.trial191.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 31/42 2026-02-23T11:09:48.642 INFO:teuthology.orchestra.run.trial191.stdout: Installing : ceph-mgr-modules-core-2:20.3.0-5439.ge371433f.el9. 32/42 2026-02-23T11:09:48.643 INFO:teuthology.orchestra.run.trial191.stdout: Installing : ceph-mgr-2:20.3.0-5439.ge371433f.el9.x86_64 33/42 2026-02-23T11:09:48.668 INFO:teuthology.orchestra.run.trial191.stdout: Running scriptlet: ceph-mgr-2:20.3.0-5439.ge371433f.el9.x86_64 33/42 2026-02-23T11:09:48.668 INFO:teuthology.orchestra.run.trial191.stdout:Glob pattern passed to enable, but globs are not supported for this. 2026-02-23T11:09:48.668 INFO:teuthology.orchestra.run.trial191.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2026-02-23T11:09:48.668 INFO:teuthology.orchestra.run.trial191.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2026-02-23T11:09:48.669 INFO:teuthology.orchestra.run.trial191.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2026-02-23T11:09:48.669 INFO:teuthology.orchestra.run.trial191.stdout: 2026-02-23T11:09:48.674 INFO:teuthology.orchestra.run.trial191.stdout: Installing : libconfig-1.7.2-9.el9.x86_64 34/42 2026-02-23T11:09:48.692 INFO:teuthology.orchestra.run.trial191.stdout: Running scriptlet: libstoragemgmt-1.10.1-1.el9.x86_64 35/42 2026-02-23T11:09:48.692 INFO:teuthology.orchestra.run.trial191.stdout:Creating group 'libstoragemgmt' with GID 991. 2026-02-23T11:09:48.692 INFO:teuthology.orchestra.run.trial191.stdout:Creating user 'libstoragemgmt' (daemon account for libstoragemgmt) with UID 991 and GID 991. 2026-02-23T11:09:48.692 INFO:teuthology.orchestra.run.trial191.stdout: 2026-02-23T11:09:48.700 INFO:teuthology.orchestra.run.trial191.stdout: Installing : libstoragemgmt-1.10.1-1.el9.x86_64 35/42 2026-02-23T11:09:48.725 INFO:teuthology.orchestra.run.trial191.stdout: Running scriptlet: libstoragemgmt-1.10.1-1.el9.x86_64 35/42 2026-02-23T11:09:48.725 INFO:teuthology.orchestra.run.trial191.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/libstoragemgmt.service → /usr/lib/systemd/system/libstoragemgmt.service. 2026-02-23T11:09:48.725 INFO:teuthology.orchestra.run.trial191.stdout: 2026-02-23T11:09:49.327 INFO:teuthology.orchestra.run.trial191.stdout: Installing : python3-libstoragemgmt-1.10.1-1.el9.x86_64 36/42 2026-02-23T11:09:54.659 INFO:teuthology.orchestra.run.trial191.stdout: Installing : ceph-osd-classic-2:20.3.0-5439.ge371433f.el9.x86_6 37/42 2026-02-23T11:09:54.669 INFO:teuthology.orchestra.run.trial191.stdout: Installing : ceph-osd-crimson-2:20.3.0-5439.ge371433f.el9.x86_6 38/42 2026-02-23T11:09:54.670 INFO:teuthology.orchestra.run.trial191.stdout: Installing : ceph-osd-2:20.3.0-5439.ge371433f.el9.x86_64 39/42 2026-02-23T11:09:54.696 INFO:teuthology.orchestra.run.trial191.stdout: Running scriptlet: ceph-osd-2:20.3.0-5439.ge371433f.el9.x86_64 39/42 2026-02-23T11:09:54.696 INFO:teuthology.orchestra.run.trial191.stdout:Glob pattern passed to enable, but globs are not supported for this. 2026-02-23T11:09:54.696 INFO:teuthology.orchestra.run.trial191.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2026-02-23T11:09:54.696 INFO:teuthology.orchestra.run.trial191.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2026-02-23T11:09:54.696 INFO:teuthology.orchestra.run.trial191.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2026-02-23T11:09:54.696 INFO:teuthology.orchestra.run.trial191.stdout: 2026-02-23T11:09:54.900 INFO:teuthology.orchestra.run.trial191.stdout: Installing : ceph-mon-2:20.3.0-5439.ge371433f.el9.x86_64 40/42 2026-02-23T11:09:54.921 INFO:teuthology.orchestra.run.trial191.stdout: Running scriptlet: ceph-mon-2:20.3.0-5439.ge371433f.el9.x86_64 40/42 2026-02-23T11:09:54.921 INFO:teuthology.orchestra.run.trial191.stdout:Glob pattern passed to enable, but globs are not supported for this. 2026-02-23T11:09:54.921 INFO:teuthology.orchestra.run.trial191.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2026-02-23T11:09:54.921 INFO:teuthology.orchestra.run.trial191.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2026-02-23T11:09:54.921 INFO:teuthology.orchestra.run.trial191.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2026-02-23T11:09:54.922 INFO:teuthology.orchestra.run.trial191.stdout: 2026-02-23T11:09:55.004 INFO:teuthology.orchestra.run.trial191.stdout: Installing : ceph-mds-2:20.3.0-5439.ge371433f.el9.x86_64 41/42 2026-02-23T11:09:55.024 INFO:teuthology.orchestra.run.trial191.stdout: Running scriptlet: ceph-mds-2:20.3.0-5439.ge371433f.el9.x86_64 41/42 2026-02-23T11:09:55.024 INFO:teuthology.orchestra.run.trial191.stdout:Glob pattern passed to enable, but globs are not supported for this. 2026-02-23T11:09:55.024 INFO:teuthology.orchestra.run.trial191.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2026-02-23T11:09:55.025 INFO:teuthology.orchestra.run.trial191.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2026-02-23T11:09:55.025 INFO:teuthology.orchestra.run.trial191.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2026-02-23T11:09:55.025 INFO:teuthology.orchestra.run.trial191.stdout: 2026-02-23T11:09:55.025 INFO:teuthology.orchestra.run.trial191.stdout: Installing : ceph-2:20.3.0-5439.ge371433f.el9.x86_64 42/42 2026-02-23T11:09:55.031 INFO:teuthology.orchestra.run.trial191.stdout: Running scriptlet: ceph-osd-classic-2:20.3.0-5439.ge371433f.el9.x86_6 42/42 2026-02-23T11:09:55.054 INFO:teuthology.orchestra.run.trial191.stdout: Running scriptlet: ceph-osd-crimson-2:20.3.0-5439.ge371433f.el9.x86_6 42/42 2026-02-23T11:09:55.964 INFO:teuthology.orchestra.run.trial191.stdout: Running scriptlet: ceph-2:20.3.0-5439.ge371433f.el9.x86_64 42/42 2026-02-23T11:09:55.965 INFO:teuthology.orchestra.run.trial191.stdout: Verifying : ceph-2:20.3.0-5439.ge371433f.el9.x86_64 1/42 2026-02-23T11:09:55.965 INFO:teuthology.orchestra.run.trial191.stdout: Verifying : ceph-mds-2:20.3.0-5439.ge371433f.el9.x86_64 2/42 2026-02-23T11:09:55.965 INFO:teuthology.orchestra.run.trial191.stdout: Verifying : ceph-mgr-2:20.3.0-5439.ge371433f.el9.x86_64 3/42 2026-02-23T11:09:55.965 INFO:teuthology.orchestra.run.trial191.stdout: Verifying : ceph-mon-2:20.3.0-5439.ge371433f.el9.x86_64 4/42 2026-02-23T11:09:55.965 INFO:teuthology.orchestra.run.trial191.stdout: Verifying : ceph-osd-2:20.3.0-5439.ge371433f.el9.x86_64 5/42 2026-02-23T11:09:55.965 INFO:teuthology.orchestra.run.trial191.stdout: Verifying : ceph-osd-classic-2:20.3.0-5439.ge371433f.el9.x86_6 6/42 2026-02-23T11:09:55.965 INFO:teuthology.orchestra.run.trial191.stdout: Verifying : ceph-osd-crimson-2:20.3.0-5439.ge371433f.el9.x86_6 7/42 2026-02-23T11:09:55.965 INFO:teuthology.orchestra.run.trial191.stdout: Verifying : ceph-mgr-modules-core-2:20.3.0-5439.ge371433f.el9. 8/42 2026-02-23T11:09:55.965 INFO:teuthology.orchestra.run.trial191.stdout: Verifying : ledmon-libs-1.1.0-3.el9.x86_64 9/42 2026-02-23T11:09:55.965 INFO:teuthology.orchestra.run.trial191.stdout: Verifying : libconfig-1.7.2-9.el9.x86_64 10/42 2026-02-23T11:09:55.965 INFO:teuthology.orchestra.run.trial191.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 11/42 2026-02-23T11:09:55.966 INFO:teuthology.orchestra.run.trial191.stdout: Verifying : python3-cryptography-36.0.1-5.el9.x86_64 12/42 2026-02-23T11:09:55.966 INFO:teuthology.orchestra.run.trial191.stdout: Verifying : python3-ply-3.11-14.el9.noarch 13/42 2026-02-23T11:09:55.966 INFO:teuthology.orchestra.run.trial191.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 14/42 2026-02-23T11:09:55.966 INFO:teuthology.orchestra.run.trial191.stdout: Verifying : python3-pyparsing-2.4.7-9.el9.noarch 15/42 2026-02-23T11:09:55.966 INFO:teuthology.orchestra.run.trial191.stdout: Verifying : python3-requests-2.25.1-10.el9.noarch 16/42 2026-02-23T11:09:55.966 INFO:teuthology.orchestra.run.trial191.stdout: Verifying : python3-urllib3-1.26.5-7.el9.noarch 17/42 2026-02-23T11:09:55.966 INFO:teuthology.orchestra.run.trial191.stdout: Verifying : unzip-6.0-59.el9.x86_64 18/42 2026-02-23T11:09:55.966 INFO:teuthology.orchestra.run.trial191.stdout: Verifying : zip-3.0-35.el9.x86_64 19/42 2026-02-23T11:09:55.966 INFO:teuthology.orchestra.run.trial191.stdout: Verifying : libstoragemgmt-1.10.1-1.el9.x86_64 20/42 2026-02-23T11:09:55.966 INFO:teuthology.orchestra.run.trial191.stdout: Verifying : lua-5.4.4-4.el9.x86_64 21/42 2026-02-23T11:09:55.966 INFO:teuthology.orchestra.run.trial191.stdout: Verifying : python3-libstoragemgmt-1.10.1-1.el9.x86_64 22/42 2026-02-23T11:09:55.966 INFO:teuthology.orchestra.run.trial191.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 23/42 2026-02-23T11:09:55.967 INFO:teuthology.orchestra.run.trial191.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 24/42 2026-02-23T11:09:55.967 INFO:teuthology.orchestra.run.trial191.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 25/42 2026-02-23T11:09:55.967 INFO:teuthology.orchestra.run.trial191.stdout: Verifying : luarocks-3.9.2-5.el9.noarch 26/42 2026-02-23T11:09:55.967 INFO:teuthology.orchestra.run.trial191.stdout: Verifying : python3-autocommand-2.2.2-8.el9.noarch 27/42 2026-02-23T11:09:55.967 INFO:teuthology.orchestra.run.trial191.stdout: Verifying : python3-backports-tarfile-1.2.0-1.el9.noarch 28/42 2026-02-23T11:09:55.967 INFO:teuthology.orchestra.run.trial191.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 29/42 2026-02-23T11:09:55.967 INFO:teuthology.orchestra.run.trial191.stdout: Verifying : python3-cheroot-10.0.1-4.el9.noarch 30/42 2026-02-23T11:09:55.967 INFO:teuthology.orchestra.run.trial191.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 31/42 2026-02-23T11:09:55.967 INFO:teuthology.orchestra.run.trial191.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 32/42 2026-02-23T11:09:55.967 INFO:teuthology.orchestra.run.trial191.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 33/42 2026-02-23T11:09:55.967 INFO:teuthology.orchestra.run.trial191.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 34/42 2026-02-23T11:09:55.967 INFO:teuthology.orchestra.run.trial191.stdout: Verifying : python3-jaraco-context-6.0.1-3.el9.noarch 35/42 2026-02-23T11:09:55.967 INFO:teuthology.orchestra.run.trial191.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 36/42 2026-02-23T11:09:55.968 INFO:teuthology.orchestra.run.trial191.stdout: Verifying : python3-jaraco-text-4.0.0-2.el9.noarch 37/42 2026-02-23T11:09:55.968 INFO:teuthology.orchestra.run.trial191.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 38/42 2026-02-23T11:09:55.968 INFO:teuthology.orchestra.run.trial191.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 39/42 2026-02-23T11:09:55.968 INFO:teuthology.orchestra.run.trial191.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 40/42 2026-02-23T11:09:55.968 INFO:teuthology.orchestra.run.trial191.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 41/42 2026-02-23T11:09:56.020 INFO:teuthology.orchestra.run.trial191.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 42/42 2026-02-23T11:09:56.020 INFO:teuthology.orchestra.run.trial191.stdout: 2026-02-23T11:09:56.020 INFO:teuthology.orchestra.run.trial191.stdout:Installed: 2026-02-23T11:09:56.020 INFO:teuthology.orchestra.run.trial191.stdout: ceph-2:20.3.0-5439.ge371433f.el9.x86_64 2026-02-23T11:09:56.020 INFO:teuthology.orchestra.run.trial191.stdout: ceph-mds-2:20.3.0-5439.ge371433f.el9.x86_64 2026-02-23T11:09:56.020 INFO:teuthology.orchestra.run.trial191.stdout: ceph-mgr-2:20.3.0-5439.ge371433f.el9.x86_64 2026-02-23T11:09:56.020 INFO:teuthology.orchestra.run.trial191.stdout: ceph-mgr-modules-core-2:20.3.0-5439.ge371433f.el9.noarch 2026-02-23T11:09:56.020 INFO:teuthology.orchestra.run.trial191.stdout: ceph-mon-2:20.3.0-5439.ge371433f.el9.x86_64 2026-02-23T11:09:56.020 INFO:teuthology.orchestra.run.trial191.stdout: ceph-osd-2:20.3.0-5439.ge371433f.el9.x86_64 2026-02-23T11:09:56.020 INFO:teuthology.orchestra.run.trial191.stdout: ceph-osd-classic-2:20.3.0-5439.ge371433f.el9.x86_64 2026-02-23T11:09:56.020 INFO:teuthology.orchestra.run.trial191.stdout: ceph-osd-crimson-2:20.3.0-5439.ge371433f.el9.x86_64 2026-02-23T11:09:56.020 INFO:teuthology.orchestra.run.trial191.stdout: ledmon-libs-1.1.0-3.el9.x86_64 2026-02-23T11:09:56.020 INFO:teuthology.orchestra.run.trial191.stdout: libconfig-1.7.2-9.el9.x86_64 2026-02-23T11:09:56.020 INFO:teuthology.orchestra.run.trial191.stdout: libstoragemgmt-1.10.1-1.el9.x86_64 2026-02-23T11:09:56.020 INFO:teuthology.orchestra.run.trial191.stdout: lua-5.4.4-4.el9.x86_64 2026-02-23T11:09:56.021 INFO:teuthology.orchestra.run.trial191.stdout: lua-devel-5.4.4-4.el9.x86_64 2026-02-23T11:09:56.021 INFO:teuthology.orchestra.run.trial191.stdout: luarocks-3.9.2-5.el9.noarch 2026-02-23T11:09:56.021 INFO:teuthology.orchestra.run.trial191.stdout: python3-autocommand-2.2.2-8.el9.noarch 2026-02-23T11:09:56.021 INFO:teuthology.orchestra.run.trial191.stdout: python3-backports-tarfile-1.2.0-1.el9.noarch 2026-02-23T11:09:56.021 INFO:teuthology.orchestra.run.trial191.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2026-02-23T11:09:56.021 INFO:teuthology.orchestra.run.trial191.stdout: python3-cffi-1.14.5-5.el9.x86_64 2026-02-23T11:09:56.021 INFO:teuthology.orchestra.run.trial191.stdout: python3-cheroot-10.0.1-4.el9.noarch 2026-02-23T11:09:56.021 INFO:teuthology.orchestra.run.trial191.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2026-02-23T11:09:56.021 INFO:teuthology.orchestra.run.trial191.stdout: python3-cryptography-36.0.1-5.el9.x86_64 2026-02-23T11:09:56.021 INFO:teuthology.orchestra.run.trial191.stdout: python3-jaraco-8.2.1-3.el9.noarch 2026-02-23T11:09:56.021 INFO:teuthology.orchestra.run.trial191.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2026-02-23T11:09:56.021 INFO:teuthology.orchestra.run.trial191.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2026-02-23T11:09:56.021 INFO:teuthology.orchestra.run.trial191.stdout: python3-jaraco-context-6.0.1-3.el9.noarch 2026-02-23T11:09:56.021 INFO:teuthology.orchestra.run.trial191.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2026-02-23T11:09:56.021 INFO:teuthology.orchestra.run.trial191.stdout: python3-jaraco-text-4.0.0-2.el9.noarch 2026-02-23T11:09:56.021 INFO:teuthology.orchestra.run.trial191.stdout: python3-libstoragemgmt-1.10.1-1.el9.x86_64 2026-02-23T11:09:56.021 INFO:teuthology.orchestra.run.trial191.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2026-02-23T11:09:56.021 INFO:teuthology.orchestra.run.trial191.stdout: python3-packaging-20.9-5.el9.noarch 2026-02-23T11:09:56.021 INFO:teuthology.orchestra.run.trial191.stdout: python3-ply-3.11-14.el9.noarch 2026-02-23T11:09:56.021 INFO:teuthology.orchestra.run.trial191.stdout: python3-portend-3.1.0-2.el9.noarch 2026-02-23T11:09:56.021 INFO:teuthology.orchestra.run.trial191.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2026-02-23T11:09:56.021 INFO:teuthology.orchestra.run.trial191.stdout: python3-pycparser-2.20-6.el9.noarch 2026-02-23T11:09:56.021 INFO:teuthology.orchestra.run.trial191.stdout: python3-pyparsing-2.4.7-9.el9.noarch 2026-02-23T11:09:56.021 INFO:teuthology.orchestra.run.trial191.stdout: python3-requests-2.25.1-10.el9.noarch 2026-02-23T11:09:56.021 INFO:teuthology.orchestra.run.trial191.stdout: python3-tempora-5.0.0-2.el9.noarch 2026-02-23T11:09:56.021 INFO:teuthology.orchestra.run.trial191.stdout: python3-toml-0.10.2-6.el9.noarch 2026-02-23T11:09:56.021 INFO:teuthology.orchestra.run.trial191.stdout: python3-urllib3-1.26.5-7.el9.noarch 2026-02-23T11:09:56.021 INFO:teuthology.orchestra.run.trial191.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2026-02-23T11:09:56.021 INFO:teuthology.orchestra.run.trial191.stdout: unzip-6.0-59.el9.x86_64 2026-02-23T11:09:56.021 INFO:teuthology.orchestra.run.trial191.stdout: zip-3.0-35.el9.x86_64 2026-02-23T11:09:56.021 INFO:teuthology.orchestra.run.trial191.stdout: 2026-02-23T11:09:56.021 INFO:teuthology.orchestra.run.trial191.stdout:Complete! 2026-02-23T11:09:56.066 DEBUG:teuthology.orchestra.run.trial191:> sudo yum -y install ceph-base 2026-02-23T11:09:56.372 INFO:teuthology.orchestra.run.trial191.stdout:Last metadata expiration check: 0:00:52 ago on Mon 23 Feb 2026 11:09:04 AM UTC. 2026-02-23T11:09:56.436 INFO:teuthology.orchestra.run.trial191.stdout:Package ceph-base-2:20.3.0-5439.ge371433f.el9.x86_64 is already installed. 2026-02-23T11:09:56.454 INFO:teuthology.orchestra.run.trial191.stdout:Dependencies resolved. 2026-02-23T11:09:56.454 INFO:teuthology.orchestra.run.trial191.stdout:Nothing to do. 2026-02-23T11:09:56.455 INFO:teuthology.orchestra.run.trial191.stdout:Complete! 2026-02-23T11:09:56.476 DEBUG:teuthology.orchestra.run.trial191:> sudo yum -y install cephadm 2026-02-23T11:09:56.782 INFO:teuthology.orchestra.run.trial191.stdout:Last metadata expiration check: 0:00:52 ago on Mon 23 Feb 2026 11:09:04 AM UTC. 2026-02-23T11:09:56.863 INFO:teuthology.orchestra.run.trial191.stdout:Dependencies resolved. 2026-02-23T11:09:56.864 INFO:teuthology.orchestra.run.trial191.stdout:================================================================================ 2026-02-23T11:09:56.864 INFO:teuthology.orchestra.run.trial191.stdout: Package Arch Version Repository Size 2026-02-23T11:09:56.864 INFO:teuthology.orchestra.run.trial191.stdout:================================================================================ 2026-02-23T11:09:56.864 INFO:teuthology.orchestra.run.trial191.stdout:Installing: 2026-02-23T11:09:56.865 INFO:teuthology.orchestra.run.trial191.stdout: cephadm noarch 2:20.3.0-5439.ge371433f.el9 ceph-noarch 1.0 M 2026-02-23T11:09:56.865 INFO:teuthology.orchestra.run.trial191.stdout: 2026-02-23T11:09:56.865 INFO:teuthology.orchestra.run.trial191.stdout:Transaction Summary 2026-02-23T11:09:56.865 INFO:teuthology.orchestra.run.trial191.stdout:================================================================================ 2026-02-23T11:09:56.865 INFO:teuthology.orchestra.run.trial191.stdout:Install 1 Package 2026-02-23T11:09:56.865 INFO:teuthology.orchestra.run.trial191.stdout: 2026-02-23T11:09:56.866 INFO:teuthology.orchestra.run.trial191.stdout:Total download size: 1.0 M 2026-02-23T11:09:56.866 INFO:teuthology.orchestra.run.trial191.stdout:Installed size: 1.0 M 2026-02-23T11:09:56.866 INFO:teuthology.orchestra.run.trial191.stdout:Downloading Packages: 2026-02-23T11:09:57.001 INFO:teuthology.orchestra.run.trial191.stdout:cephadm-20.3.0-5439.ge371433f.el9.noarch.rpm 7.5 MB/s | 1.0 MB 00:00 2026-02-23T11:09:57.001 INFO:teuthology.orchestra.run.trial191.stdout:-------------------------------------------------------------------------------- 2026-02-23T11:09:57.001 INFO:teuthology.orchestra.run.trial191.stdout:Total 7.4 MB/s | 1.0 MB 00:00 2026-02-23T11:09:57.001 INFO:teuthology.orchestra.run.trial191.stdout:Running transaction check 2026-02-23T11:09:57.003 INFO:teuthology.orchestra.run.trial191.stdout:Transaction check succeeded. 2026-02-23T11:09:57.003 INFO:teuthology.orchestra.run.trial191.stdout:Running transaction test 2026-02-23T11:09:57.007 INFO:teuthology.orchestra.run.trial191.stdout:Transaction test succeeded. 2026-02-23T11:09:57.007 INFO:teuthology.orchestra.run.trial191.stdout:Running transaction 2026-02-23T11:09:57.025 INFO:teuthology.orchestra.run.trial191.stdout: Preparing : 1/1 2026-02-23T11:09:57.085 INFO:teuthology.orchestra.run.trial191.stdout: Running scriptlet: cephadm-2:20.3.0-5439.ge371433f.el9.noarch 1/1 2026-02-23T11:09:57.099 INFO:teuthology.orchestra.run.trial191.stdout: Installing : cephadm-2:20.3.0-5439.ge371433f.el9.noarch 1/1 2026-02-23T11:09:57.169 INFO:teuthology.orchestra.run.trial191.stdout: Running scriptlet: cephadm-2:20.3.0-5439.ge371433f.el9.noarch 1/1 2026-02-23T11:09:57.224 INFO:teuthology.orchestra.run.trial191.stdout: Verifying : cephadm-2:20.3.0-5439.ge371433f.el9.noarch 1/1 2026-02-23T11:09:57.224 INFO:teuthology.orchestra.run.trial191.stdout: 2026-02-23T11:09:57.224 INFO:teuthology.orchestra.run.trial191.stdout:Installed: 2026-02-23T11:09:57.224 INFO:teuthology.orchestra.run.trial191.stdout: cephadm-2:20.3.0-5439.ge371433f.el9.noarch 2026-02-23T11:09:57.224 INFO:teuthology.orchestra.run.trial191.stdout: 2026-02-23T11:09:57.225 INFO:teuthology.orchestra.run.trial191.stdout:Complete! 2026-02-23T11:09:57.252 DEBUG:teuthology.orchestra.run.trial191:> sudo yum -y install ceph-immutable-object-cache 2026-02-23T11:09:57.560 INFO:teuthology.orchestra.run.trial191.stdout:Last metadata expiration check: 0:00:53 ago on Mon 23 Feb 2026 11:09:04 AM UTC. 2026-02-23T11:09:57.642 INFO:teuthology.orchestra.run.trial191.stdout:Dependencies resolved. 2026-02-23T11:09:57.643 INFO:teuthology.orchestra.run.trial191.stdout:================================================================================ 2026-02-23T11:09:57.643 INFO:teuthology.orchestra.run.trial191.stdout: Package Arch Version Repo Size 2026-02-23T11:09:57.643 INFO:teuthology.orchestra.run.trial191.stdout:================================================================================ 2026-02-23T11:09:57.643 INFO:teuthology.orchestra.run.trial191.stdout:Installing: 2026-02-23T11:09:57.643 INFO:teuthology.orchestra.run.trial191.stdout: ceph-immutable-object-cache x86_64 2:20.3.0-5439.ge371433f.el9 ceph 149 k 2026-02-23T11:09:57.643 INFO:teuthology.orchestra.run.trial191.stdout: 2026-02-23T11:09:57.643 INFO:teuthology.orchestra.run.trial191.stdout:Transaction Summary 2026-02-23T11:09:57.644 INFO:teuthology.orchestra.run.trial191.stdout:================================================================================ 2026-02-23T11:09:57.644 INFO:teuthology.orchestra.run.trial191.stdout:Install 1 Package 2026-02-23T11:09:57.644 INFO:teuthology.orchestra.run.trial191.stdout: 2026-02-23T11:09:57.644 INFO:teuthology.orchestra.run.trial191.stdout:Total download size: 149 k 2026-02-23T11:09:57.644 INFO:teuthology.orchestra.run.trial191.stdout:Installed size: 455 k 2026-02-23T11:09:57.644 INFO:teuthology.orchestra.run.trial191.stdout:Downloading Packages: 2026-02-23T11:09:58.043 INFO:teuthology.orchestra.run.trial191.stdout:ceph-immutable-object-cache-20.3.0-5439.ge37143 373 kB/s | 149 kB 00:00 2026-02-23T11:09:58.043 INFO:teuthology.orchestra.run.trial191.stdout:-------------------------------------------------------------------------------- 2026-02-23T11:09:58.043 INFO:teuthology.orchestra.run.trial191.stdout:Total 372 kB/s | 149 kB 00:00 2026-02-23T11:09:58.044 INFO:teuthology.orchestra.run.trial191.stdout:Running transaction check 2026-02-23T11:09:58.048 INFO:teuthology.orchestra.run.trial191.stdout:Transaction check succeeded. 2026-02-23T11:09:58.048 INFO:teuthology.orchestra.run.trial191.stdout:Running transaction test 2026-02-23T11:09:58.074 INFO:teuthology.orchestra.run.trial191.stdout:Transaction test succeeded. 2026-02-23T11:09:58.074 INFO:teuthology.orchestra.run.trial191.stdout:Running transaction 2026-02-23T11:09:58.123 INFO:teuthology.orchestra.run.trial191.stdout: Preparing : 1/1 2026-02-23T11:09:58.125 INFO:teuthology.orchestra.run.trial191.stdout: Installing : ceph-immutable-object-cache-2:20.3.0-5439.ge371433f. 1/1 2026-02-23T11:09:58.147 INFO:teuthology.orchestra.run.trial191.stdout: Running scriptlet: ceph-immutable-object-cache-2:20.3.0-5439.ge371433f. 1/1 2026-02-23T11:09:58.147 INFO:teuthology.orchestra.run.trial191.stdout:Glob pattern passed to enable, but globs are not supported for this. 2026-02-23T11:09:58.147 INFO:teuthology.orchestra.run.trial191.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2026-02-23T11:09:58.147 INFO:teuthology.orchestra.run.trial191.stdout: 2026-02-23T11:09:58.414 INFO:teuthology.orchestra.run.trial191.stdout: Verifying : ceph-immutable-object-cache-2:20.3.0-5439.ge371433f. 1/1 2026-02-23T11:09:58.414 INFO:teuthology.orchestra.run.trial191.stdout: 2026-02-23T11:09:58.415 INFO:teuthology.orchestra.run.trial191.stdout:Installed: 2026-02-23T11:09:58.415 INFO:teuthology.orchestra.run.trial191.stdout: ceph-immutable-object-cache-2:20.3.0-5439.ge371433f.el9.x86_64 2026-02-23T11:09:58.415 INFO:teuthology.orchestra.run.trial191.stdout: 2026-02-23T11:09:58.415 INFO:teuthology.orchestra.run.trial191.stdout:Complete! 2026-02-23T11:09:58.444 DEBUG:teuthology.orchestra.run.trial191:> sudo yum -y install ceph-mgr 2026-02-23T11:09:58.755 INFO:teuthology.orchestra.run.trial191.stdout:Last metadata expiration check: 0:00:54 ago on Mon 23 Feb 2026 11:09:04 AM UTC. 2026-02-23T11:09:58.820 INFO:teuthology.orchestra.run.trial191.stdout:Package ceph-mgr-2:20.3.0-5439.ge371433f.el9.x86_64 is already installed. 2026-02-23T11:09:58.837 INFO:teuthology.orchestra.run.trial191.stdout:Dependencies resolved. 2026-02-23T11:09:58.838 INFO:teuthology.orchestra.run.trial191.stdout:Nothing to do. 2026-02-23T11:09:58.838 INFO:teuthology.orchestra.run.trial191.stdout:Complete! 2026-02-23T11:09:58.860 DEBUG:teuthology.orchestra.run.trial191:> sudo yum -y install ceph-mgr-dashboard 2026-02-23T11:09:59.165 INFO:teuthology.orchestra.run.trial191.stdout:Last metadata expiration check: 0:00:55 ago on Mon 23 Feb 2026 11:09:04 AM UTC. 2026-02-23T11:09:59.247 INFO:teuthology.orchestra.run.trial191.stdout:Dependencies resolved. 2026-02-23T11:09:59.248 INFO:teuthology.orchestra.run.trial191.stdout:================================================================================ 2026-02-23T11:09:59.248 INFO:teuthology.orchestra.run.trial191.stdout: Package Arch Version Repository Size 2026-02-23T11:09:59.249 INFO:teuthology.orchestra.run.trial191.stdout:================================================================================ 2026-02-23T11:09:59.249 INFO:teuthology.orchestra.run.trial191.stdout:Installing: 2026-02-23T11:09:59.249 INFO:teuthology.orchestra.run.trial191.stdout: ceph-mgr-dashboard noarch 2:20.3.0-5439.ge371433f.el9 ceph-noarch 51 M 2026-02-23T11:09:59.249 INFO:teuthology.orchestra.run.trial191.stdout:Installing dependencies: 2026-02-23T11:09:59.249 INFO:teuthology.orchestra.run.trial191.stdout: abseil-cpp x86_64 20211102.0-4.el9 epel 551 k 2026-02-23T11:09:59.249 INFO:teuthology.orchestra.run.trial191.stdout: ceph-grafana-dashboards noarch 2:20.3.0-5439.ge371433f.el9 ceph-noarch 57 k 2026-02-23T11:09:59.249 INFO:teuthology.orchestra.run.trial191.stdout: ceph-prometheus-alerts noarch 2:20.3.0-5439.ge371433f.el9 ceph-noarch 17 k 2026-02-23T11:09:59.249 INFO:teuthology.orchestra.run.trial191.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2026-02-23T11:09:59.249 INFO:teuthology.orchestra.run.trial191.stdout: protobuf-compiler x86_64 3.14.0-17.el9 crb 862 k 2026-02-23T11:09:59.249 INFO:teuthology.orchestra.run.trial191.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2026-02-23T11:09:59.249 INFO:teuthology.orchestra.run.trial191.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2026-02-23T11:09:59.250 INFO:teuthology.orchestra.run.trial191.stdout: python3-jmespath noarch 1.0.1-1.el9 appstream 48 k 2026-02-23T11:09:59.250 INFO:teuthology.orchestra.run.trial191.stdout: python3-protobuf noarch 3.14.0-17.el9 appstream 267 k 2026-02-23T11:09:59.250 INFO:teuthology.orchestra.run.trial191.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2026-02-23T11:09:59.250 INFO:teuthology.orchestra.run.trial191.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2026-02-23T11:09:59.250 INFO:teuthology.orchestra.run.trial191.stdout: python3-xmltodict noarch 0.12.0-15.el9 epel 22 k 2026-02-23T11:09:59.250 INFO:teuthology.orchestra.run.trial191.stdout: 2026-02-23T11:09:59.250 INFO:teuthology.orchestra.run.trial191.stdout:Transaction Summary 2026-02-23T11:09:59.250 INFO:teuthology.orchestra.run.trial191.stdout:================================================================================ 2026-02-23T11:09:59.250 INFO:teuthology.orchestra.run.trial191.stdout:Install 13 Packages 2026-02-23T11:09:59.250 INFO:teuthology.orchestra.run.trial191.stdout: 2026-02-23T11:09:59.250 INFO:teuthology.orchestra.run.trial191.stdout:Total download size: 55 M 2026-02-23T11:09:59.250 INFO:teuthology.orchestra.run.trial191.stdout:Installed size: 262 M 2026-02-23T11:09:59.251 INFO:teuthology.orchestra.run.trial191.stdout:Downloading Packages: 2026-02-23T11:09:59.979 INFO:teuthology.orchestra.run.trial191.stdout:(1/13): ceph-prometheus-alerts-20.3.0-5439.ge37 71 kB/s | 17 kB 00:00 2026-02-23T11:10:00.039 INFO:teuthology.orchestra.run.trial191.stdout:(2/13): ceph-grafana-dashboards-20.3.0-5439.ge3 189 kB/s | 57 kB 00:00 2026-02-23T11:10:00.113 INFO:teuthology.orchestra.run.trial191.stdout:(3/13): python3-jmespath-1.0.1-1.el9.noarch.rpm 356 kB/s | 48 kB 00:00 2026-02-23T11:10:00.200 INFO:teuthology.orchestra.run.trial191.stdout:(4/13): python3-protobuf-3.14.0-17.el9.noarch.r 1.6 MB/s | 267 kB 00:00 2026-02-23T11:10:00.529 INFO:teuthology.orchestra.run.trial191.stdout:(5/13): abseil-cpp-20211102.0-4.el9.x86_64.rpm 1.6 MB/s | 551 kB 00:00 2026-02-23T11:10:00.567 INFO:teuthology.orchestra.run.trial191.stdout:(6/13): grpc-data-1.46.7-10.el9.noarch.rpm 510 kB/s | 19 kB 00:00 2026-02-23T11:10:00.694 INFO:teuthology.orchestra.run.trial191.stdout:(7/13): python3-grpcio-1.46.7-10.el9.x86_64.rpm 16 MB/s | 2.0 MB 00:00 2026-02-23T11:10:00.733 INFO:teuthology.orchestra.run.trial191.stdout:(8/13): python3-grpcio-tools-1.46.7-10.el9.x86_ 3.6 MB/s | 144 kB 00:00 2026-02-23T11:10:00.771 INFO:teuthology.orchestra.run.trial191.stdout:(9/13): python3-repoze-lru-0.7-16.el9.noarch.rp 813 kB/s | 31 kB 00:00 2026-02-23T11:10:00.811 INFO:teuthology.orchestra.run.trial191.stdout:(10/13): python3-routes-2.5.1-5.el9.noarch.rpm 4.7 MB/s | 188 kB 00:00 2026-02-23T11:10:00.849 INFO:teuthology.orchestra.run.trial191.stdout:(11/13): python3-xmltodict-0.12.0-15.el9.noarch 587 kB/s | 22 kB 00:00 2026-02-23T11:10:01.022 INFO:teuthology.orchestra.run.trial191.stdout:(12/13): protobuf-compiler-3.14.0-17.el9.x86_64 948 kB/s | 862 kB 00:00 2026-02-23T11:10:01.700 INFO:teuthology.orchestra.run.trial191.stdout:(13/13): ceph-mgr-dashboard-20.3.0-5439.ge37143 26 MB/s | 51 MB 00:01 2026-02-23T11:10:01.703 INFO:teuthology.orchestra.run.trial191.stdout:-------------------------------------------------------------------------------- 2026-02-23T11:10:01.703 INFO:teuthology.orchestra.run.trial191.stdout:Total 22 MB/s | 55 MB 00:02 2026-02-23T11:10:01.730 INFO:teuthology.orchestra.run.trial191.stdout:Running transaction check 2026-02-23T11:10:01.740 INFO:teuthology.orchestra.run.trial191.stdout:Transaction check succeeded. 2026-02-23T11:10:01.740 INFO:teuthology.orchestra.run.trial191.stdout:Running transaction test 2026-02-23T11:10:01.954 INFO:teuthology.orchestra.run.trial191.stdout:Transaction test succeeded. 2026-02-23T11:10:01.954 INFO:teuthology.orchestra.run.trial191.stdout:Running transaction 2026-02-23T11:10:02.182 INFO:teuthology.orchestra.run.trial191.stdout: Preparing : 1/1 2026-02-23T11:10:02.188 INFO:teuthology.orchestra.run.trial191.stdout: Installing : python3-xmltodict-0.12.0-15.el9.noarch 1/13 2026-02-23T11:10:02.199 INFO:teuthology.orchestra.run.trial191.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 2/13 2026-02-23T11:10:02.201 INFO:teuthology.orchestra.run.trial191.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 3/13 2026-02-23T11:10:02.228 INFO:teuthology.orchestra.run.trial191.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 4/13 2026-02-23T11:10:02.271 INFO:teuthology.orchestra.run.trial191.stdout: Installing : abseil-cpp-20211102.0-4.el9.x86_64 5/13 2026-02-23T11:10:02.279 INFO:teuthology.orchestra.run.trial191.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 6/13 2026-02-23T11:10:02.307 INFO:teuthology.orchestra.run.trial191.stdout: Installing : protobuf-compiler-3.14.0-17.el9.x86_64 7/13 2026-02-23T11:10:02.318 INFO:teuthology.orchestra.run.trial191.stdout: Installing : python3-protobuf-3.14.0-17.el9.noarch 8/13 2026-02-23T11:10:02.325 INFO:teuthology.orchestra.run.trial191.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 9/13 2026-02-23T11:10:02.328 INFO:teuthology.orchestra.run.trial191.stdout: Installing : python3-jmespath-1.0.1-1.el9.noarch 10/13 2026-02-23T11:10:02.337 INFO:teuthology.orchestra.run.trial191.stdout: Installing : ceph-prometheus-alerts-2:20.3.0-5439.ge371433f.el9 11/13 2026-02-23T11:10:05.346 INFO:teuthology.orchestra.run.trial191.stdout: Installing : ceph-grafana-dashboards-2:20.3.0-5439.ge371433f.el 12/13 2026-02-23T11:10:05.353 INFO:teuthology.orchestra.run.trial191.stdout: Installing : ceph-mgr-dashboard-2:20.3.0-5439.ge371433f.el9.noa 13/13 2026-02-23T11:10:05.633 INFO:teuthology.orchestra.run.trial191.stdout: Running scriptlet: ceph-mgr-dashboard-2:20.3.0-5439.ge371433f.el9.noa 13/13 2026-02-23T11:10:05.634 INFO:teuthology.orchestra.run.trial191.stdout: Verifying : ceph-grafana-dashboards-2:20.3.0-5439.ge371433f.el 1/13 2026-02-23T11:10:05.634 INFO:teuthology.orchestra.run.trial191.stdout: Verifying : ceph-mgr-dashboard-2:20.3.0-5439.ge371433f.el9.noa 2/13 2026-02-23T11:10:05.634 INFO:teuthology.orchestra.run.trial191.stdout: Verifying : ceph-prometheus-alerts-2:20.3.0-5439.ge371433f.el9 3/13 2026-02-23T11:10:05.634 INFO:teuthology.orchestra.run.trial191.stdout: Verifying : python3-jmespath-1.0.1-1.el9.noarch 4/13 2026-02-23T11:10:05.634 INFO:teuthology.orchestra.run.trial191.stdout: Verifying : python3-protobuf-3.14.0-17.el9.noarch 5/13 2026-02-23T11:10:05.634 INFO:teuthology.orchestra.run.trial191.stdout: Verifying : protobuf-compiler-3.14.0-17.el9.x86_64 6/13 2026-02-23T11:10:05.634 INFO:teuthology.orchestra.run.trial191.stdout: Verifying : abseil-cpp-20211102.0-4.el9.x86_64 7/13 2026-02-23T11:10:05.634 INFO:teuthology.orchestra.run.trial191.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 8/13 2026-02-23T11:10:05.634 INFO:teuthology.orchestra.run.trial191.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 9/13 2026-02-23T11:10:05.634 INFO:teuthology.orchestra.run.trial191.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 10/13 2026-02-23T11:10:05.634 INFO:teuthology.orchestra.run.trial191.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 11/13 2026-02-23T11:10:05.634 INFO:teuthology.orchestra.run.trial191.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 12/13 2026-02-23T11:10:05.690 INFO:teuthology.orchestra.run.trial191.stdout: Verifying : python3-xmltodict-0.12.0-15.el9.noarch 13/13 2026-02-23T11:10:05.690 INFO:teuthology.orchestra.run.trial191.stdout: 2026-02-23T11:10:05.690 INFO:teuthology.orchestra.run.trial191.stdout:Installed: 2026-02-23T11:10:05.691 INFO:teuthology.orchestra.run.trial191.stdout: abseil-cpp-20211102.0-4.el9.x86_64 2026-02-23T11:10:05.691 INFO:teuthology.orchestra.run.trial191.stdout: ceph-grafana-dashboards-2:20.3.0-5439.ge371433f.el9.noarch 2026-02-23T11:10:05.691 INFO:teuthology.orchestra.run.trial191.stdout: ceph-mgr-dashboard-2:20.3.0-5439.ge371433f.el9.noarch 2026-02-23T11:10:05.691 INFO:teuthology.orchestra.run.trial191.stdout: ceph-prometheus-alerts-2:20.3.0-5439.ge371433f.el9.noarch 2026-02-23T11:10:05.691 INFO:teuthology.orchestra.run.trial191.stdout: grpc-data-1.46.7-10.el9.noarch 2026-02-23T11:10:05.691 INFO:teuthology.orchestra.run.trial191.stdout: protobuf-compiler-3.14.0-17.el9.x86_64 2026-02-23T11:10:05.691 INFO:teuthology.orchestra.run.trial191.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2026-02-23T11:10:05.691 INFO:teuthology.orchestra.run.trial191.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2026-02-23T11:10:05.691 INFO:teuthology.orchestra.run.trial191.stdout: python3-jmespath-1.0.1-1.el9.noarch 2026-02-23T11:10:05.691 INFO:teuthology.orchestra.run.trial191.stdout: python3-protobuf-3.14.0-17.el9.noarch 2026-02-23T11:10:05.691 INFO:teuthology.orchestra.run.trial191.stdout: python3-repoze-lru-0.7-16.el9.noarch 2026-02-23T11:10:05.691 INFO:teuthology.orchestra.run.trial191.stdout: python3-routes-2.5.1-5.el9.noarch 2026-02-23T11:10:05.691 INFO:teuthology.orchestra.run.trial191.stdout: python3-xmltodict-0.12.0-15.el9.noarch 2026-02-23T11:10:05.691 INFO:teuthology.orchestra.run.trial191.stdout: 2026-02-23T11:10:05.691 INFO:teuthology.orchestra.run.trial191.stdout:Complete! 2026-02-23T11:10:05.730 DEBUG:teuthology.orchestra.run.trial191:> sudo yum -y install ceph-mgr-diskprediction-local 2026-02-23T11:10:06.045 INFO:teuthology.orchestra.run.trial191.stdout:Last metadata expiration check: 0:01:02 ago on Mon 23 Feb 2026 11:09:04 AM UTC. 2026-02-23T11:10:06.128 INFO:teuthology.orchestra.run.trial191.stdout:Dependencies resolved. 2026-02-23T11:10:06.129 INFO:teuthology.orchestra.run.trial191.stdout:======================================================================================= 2026-02-23T11:10:06.129 INFO:teuthology.orchestra.run.trial191.stdout: Package Arch Version Repository Size 2026-02-23T11:10:06.129 INFO:teuthology.orchestra.run.trial191.stdout:======================================================================================= 2026-02-23T11:10:06.129 INFO:teuthology.orchestra.run.trial191.stdout:Installing: 2026-02-23T11:10:06.129 INFO:teuthology.orchestra.run.trial191.stdout: ceph-mgr-diskprediction-local noarch 2:20.3.0-5439.ge371433f.el9 ceph-noarch 7.4 M 2026-02-23T11:10:06.130 INFO:teuthology.orchestra.run.trial191.stdout:Installing dependencies: 2026-02-23T11:10:06.130 INFO:teuthology.orchestra.run.trial191.stdout: flexiblas x86_64 3.0.4-9.el9 appstream 30 k 2026-02-23T11:10:06.130 INFO:teuthology.orchestra.run.trial191.stdout: flexiblas-netlib x86_64 3.0.4-9.el9 appstream 3.0 M 2026-02-23T11:10:06.130 INFO:teuthology.orchestra.run.trial191.stdout: flexiblas-openblas-openmp x86_64 3.0.4-9.el9 appstream 15 k 2026-02-23T11:10:06.130 INFO:teuthology.orchestra.run.trial191.stdout: libgfortran x86_64 11.5.0-14.el9 baseos 794 k 2026-02-23T11:10:06.130 INFO:teuthology.orchestra.run.trial191.stdout: libquadmath x86_64 11.5.0-14.el9 baseos 184 k 2026-02-23T11:10:06.130 INFO:teuthology.orchestra.run.trial191.stdout: openblas x86_64 0.3.29-1.el9 appstream 42 k 2026-02-23T11:10:06.130 INFO:teuthology.orchestra.run.trial191.stdout: openblas-openmp x86_64 0.3.29-1.el9 appstream 5.3 M 2026-02-23T11:10:06.130 INFO:teuthology.orchestra.run.trial191.stdout: python3-devel x86_64 3.9.25-3.el9 appstream 244 k 2026-02-23T11:10:06.130 INFO:teuthology.orchestra.run.trial191.stdout: python3-numpy x86_64 1:1.23.5-2.el9 appstream 6.1 M 2026-02-23T11:10:06.130 INFO:teuthology.orchestra.run.trial191.stdout: python3-numpy-f2py x86_64 1:1.23.5-2.el9 appstream 442 k 2026-02-23T11:10:06.131 INFO:teuthology.orchestra.run.trial191.stdout: python3-scipy x86_64 1.9.3-2.el9 appstream 19 M 2026-02-23T11:10:06.131 INFO:teuthology.orchestra.run.trial191.stdout: 2026-02-23T11:10:06.131 INFO:teuthology.orchestra.run.trial191.stdout:Transaction Summary 2026-02-23T11:10:06.131 INFO:teuthology.orchestra.run.trial191.stdout:======================================================================================= 2026-02-23T11:10:06.131 INFO:teuthology.orchestra.run.trial191.stdout:Install 12 Packages 2026-02-23T11:10:06.131 INFO:teuthology.orchestra.run.trial191.stdout: 2026-02-23T11:10:06.131 INFO:teuthology.orchestra.run.trial191.stdout:Total download size: 43 M 2026-02-23T11:10:06.131 INFO:teuthology.orchestra.run.trial191.stdout:Installed size: 234 M 2026-02-23T11:10:06.131 INFO:teuthology.orchestra.run.trial191.stdout:Downloading Packages: 2026-02-23T11:10:06.614 INFO:teuthology.orchestra.run.trial191.stdout:(1/12): libquadmath-11.5.0-14.el9.x86_64.rpm 1.5 MB/s | 184 kB 00:00 2026-02-23T11:10:06.625 INFO:teuthology.orchestra.run.trial191.stdout:(2/12): flexiblas-3.0.4-9.el9.x86_64.rpm 2.7 MB/s | 30 kB 00:00 2026-02-23T11:10:06.639 INFO:teuthology.orchestra.run.trial191.stdout:(3/12): libgfortran-11.5.0-14.el9.x86_64.rpm 5.2 MB/s | 794 kB 00:00 2026-02-23T11:10:06.653 INFO:teuthology.orchestra.run.trial191.stdout:(4/12): flexiblas-netlib-3.0.4-9.el9.x86_64.rpm 108 MB/s | 3.0 MB 00:00 2026-02-23T11:10:06.653 INFO:teuthology.orchestra.run.trial191.stdout:(5/12): flexiblas-openblas-openmp-3.0.4-9.el9.x 1.0 MB/s | 15 kB 00:00 2026-02-23T11:10:06.658 INFO:teuthology.orchestra.run.trial191.stdout:(6/12): openblas-0.3.29-1.el9.x86_64.rpm 9.0 MB/s | 42 kB 00:00 2026-02-23T11:10:06.664 INFO:teuthology.orchestra.run.trial191.stdout:(7/12): python3-devel-3.9.25-3.el9.x86_64.rpm 38 MB/s | 244 kB 00:00 2026-02-23T11:10:06.690 INFO:teuthology.orchestra.run.trial191.stdout:(8/12): python3-numpy-1.23.5-2.el9.x86_64.rpm 236 MB/s | 6.1 MB 00:00 2026-02-23T11:10:06.703 INFO:teuthology.orchestra.run.trial191.stdout:(9/12): openblas-openmp-0.3.29-1.el9.x86_64.rpm 108 MB/s | 5.3 MB 00:00 2026-02-23T11:10:06.704 INFO:teuthology.orchestra.run.trial191.stdout:(10/12): python3-numpy-f2py-1.23.5-2.el9.x86_64 32 MB/s | 442 kB 00:00 2026-02-23T11:10:06.779 INFO:teuthology.orchestra.run.trial191.stdout:(11/12): python3-scipy-1.9.3-2.el9.x86_64.rpm 253 MB/s | 19 MB 00:00 2026-02-23T11:10:09.536 INFO:teuthology.orchestra.run.trial191.stdout:(12/12): ceph-mgr-diskprediction-local-20.3.0-5 2.4 MB/s | 7.4 MB 00:03 2026-02-23T11:10:09.538 INFO:teuthology.orchestra.run.trial191.stdout:-------------------------------------------------------------------------------- 2026-02-23T11:10:09.538 INFO:teuthology.orchestra.run.trial191.stdout:Total 13 MB/s | 43 MB 00:03 2026-02-23T11:10:09.699 INFO:teuthology.orchestra.run.trial191.stdout:Running transaction check 2026-02-23T11:10:09.713 INFO:teuthology.orchestra.run.trial191.stdout:Transaction check succeeded. 2026-02-23T11:10:09.713 INFO:teuthology.orchestra.run.trial191.stdout:Running transaction test 2026-02-23T11:10:09.914 INFO:teuthology.orchestra.run.trial191.stdout:Transaction test succeeded. 2026-02-23T11:10:09.914 INFO:teuthology.orchestra.run.trial191.stdout:Running transaction 2026-02-23T11:10:10.129 INFO:teuthology.orchestra.run.trial191.stdout: Preparing : 1/1 2026-02-23T11:10:10.131 INFO:teuthology.orchestra.run.trial191.stdout: Installing : flexiblas-3.0.4-9.el9.x86_64 1/12 2026-02-23T11:10:10.138 INFO:teuthology.orchestra.run.trial191.stdout: Installing : libquadmath-11.5.0-14.el9.x86_64 2/12 2026-02-23T11:10:10.171 INFO:teuthology.orchestra.run.trial191.stdout: Installing : libgfortran-11.5.0-14.el9.x86_64 3/12 2026-02-23T11:10:10.176 INFO:teuthology.orchestra.run.trial191.stdout: Installing : python3-devel-3.9.25-3.el9.x86_64 4/12 2026-02-23T11:10:10.235 INFO:teuthology.orchestra.run.trial191.stdout: Installing : openblas-0.3.29-1.el9.x86_64 5/12 2026-02-23T11:10:10.238 INFO:teuthology.orchestra.run.trial191.stdout: Installing : openblas-openmp-0.3.29-1.el9.x86_64 6/12 2026-02-23T11:10:10.259 INFO:teuthology.orchestra.run.trial191.stdout: Installing : flexiblas-openblas-openmp-3.0.4-9.el9.x86_64 7/12 2026-02-23T11:10:10.585 INFO:teuthology.orchestra.run.trial191.stdout: Installing : flexiblas-netlib-3.0.4-9.el9.x86_64 8/12 2026-02-23T11:10:10.659 INFO:teuthology.orchestra.run.trial191.stdout: Installing : python3-numpy-1:1.23.5-2.el9.x86_64 9/12 2026-02-23T11:10:11.336 INFO:teuthology.orchestra.run.trial191.stdout: Installing : python3-numpy-f2py-1:1.23.5-2.el9.x86_64 10/12 2026-02-23T11:10:11.793 INFO:teuthology.orchestra.run.trial191.stdout: Installing : python3-scipy-1.9.3-2.el9.x86_64 11/12 2026-02-23T11:10:11.794 INFO:teuthology.orchestra.run.trial191.stdout: Installing : ceph-mgr-diskprediction-local-2:20.3.0-5439.ge3714 12/12 2026-02-23T11:10:12.092 INFO:teuthology.orchestra.run.trial191.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:20.3.0-5439.ge3714 12/12 2026-02-23T11:10:12.092 INFO:teuthology.orchestra.run.trial191.stdout: Verifying : ceph-mgr-diskprediction-local-2:20.3.0-5439.ge3714 1/12 2026-02-23T11:10:12.093 INFO:teuthology.orchestra.run.trial191.stdout: Verifying : libgfortran-11.5.0-14.el9.x86_64 2/12 2026-02-23T11:10:12.093 INFO:teuthology.orchestra.run.trial191.stdout: Verifying : libquadmath-11.5.0-14.el9.x86_64 3/12 2026-02-23T11:10:12.093 INFO:teuthology.orchestra.run.trial191.stdout: Verifying : flexiblas-3.0.4-9.el9.x86_64 4/12 2026-02-23T11:10:12.093 INFO:teuthology.orchestra.run.trial191.stdout: Verifying : flexiblas-netlib-3.0.4-9.el9.x86_64 5/12 2026-02-23T11:10:12.093 INFO:teuthology.orchestra.run.trial191.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-9.el9.x86_64 6/12 2026-02-23T11:10:12.093 INFO:teuthology.orchestra.run.trial191.stdout: Verifying : openblas-0.3.29-1.el9.x86_64 7/12 2026-02-23T11:10:12.093 INFO:teuthology.orchestra.run.trial191.stdout: Verifying : openblas-openmp-0.3.29-1.el9.x86_64 8/12 2026-02-23T11:10:12.093 INFO:teuthology.orchestra.run.trial191.stdout: Verifying : python3-devel-3.9.25-3.el9.x86_64 9/12 2026-02-23T11:10:12.093 INFO:teuthology.orchestra.run.trial191.stdout: Verifying : python3-numpy-1:1.23.5-2.el9.x86_64 10/12 2026-02-23T11:10:12.093 INFO:teuthology.orchestra.run.trial191.stdout: Verifying : python3-numpy-f2py-1:1.23.5-2.el9.x86_64 11/12 2026-02-23T11:10:12.150 INFO:teuthology.orchestra.run.trial191.stdout: Verifying : python3-scipy-1.9.3-2.el9.x86_64 12/12 2026-02-23T11:10:12.151 INFO:teuthology.orchestra.run.trial191.stdout: 2026-02-23T11:10:12.151 INFO:teuthology.orchestra.run.trial191.stdout:Installed: 2026-02-23T11:10:12.151 INFO:teuthology.orchestra.run.trial191.stdout: ceph-mgr-diskprediction-local-2:20.3.0-5439.ge371433f.el9.noarch 2026-02-23T11:10:12.151 INFO:teuthology.orchestra.run.trial191.stdout: flexiblas-3.0.4-9.el9.x86_64 2026-02-23T11:10:12.151 INFO:teuthology.orchestra.run.trial191.stdout: flexiblas-netlib-3.0.4-9.el9.x86_64 2026-02-23T11:10:12.151 INFO:teuthology.orchestra.run.trial191.stdout: flexiblas-openblas-openmp-3.0.4-9.el9.x86_64 2026-02-23T11:10:12.151 INFO:teuthology.orchestra.run.trial191.stdout: libgfortran-11.5.0-14.el9.x86_64 2026-02-23T11:10:12.151 INFO:teuthology.orchestra.run.trial191.stdout: libquadmath-11.5.0-14.el9.x86_64 2026-02-23T11:10:12.151 INFO:teuthology.orchestra.run.trial191.stdout: openblas-0.3.29-1.el9.x86_64 2026-02-23T11:10:12.151 INFO:teuthology.orchestra.run.trial191.stdout: openblas-openmp-0.3.29-1.el9.x86_64 2026-02-23T11:10:12.151 INFO:teuthology.orchestra.run.trial191.stdout: python3-devel-3.9.25-3.el9.x86_64 2026-02-23T11:10:12.151 INFO:teuthology.orchestra.run.trial191.stdout: python3-numpy-1:1.23.5-2.el9.x86_64 2026-02-23T11:10:12.152 INFO:teuthology.orchestra.run.trial191.stdout: python3-numpy-f2py-1:1.23.5-2.el9.x86_64 2026-02-23T11:10:12.152 INFO:teuthology.orchestra.run.trial191.stdout: python3-scipy-1.9.3-2.el9.x86_64 2026-02-23T11:10:12.152 INFO:teuthology.orchestra.run.trial191.stdout: 2026-02-23T11:10:12.152 INFO:teuthology.orchestra.run.trial191.stdout:Complete! 2026-02-23T11:10:12.190 DEBUG:teuthology.orchestra.run.trial191:> sudo yum -y install ceph-mgr-rook 2026-02-23T11:10:12.500 INFO:teuthology.orchestra.run.trial191.stdout:Last metadata expiration check: 0:01:08 ago on Mon 23 Feb 2026 11:09:04 AM UTC. 2026-02-23T11:10:12.586 INFO:teuthology.orchestra.run.trial191.stdout:Dependencies resolved. 2026-02-23T11:10:12.586 INFO:teuthology.orchestra.run.trial191.stdout:================================================================================ 2026-02-23T11:10:12.586 INFO:teuthology.orchestra.run.trial191.stdout: Package Arch Version Repository Size 2026-02-23T11:10:12.586 INFO:teuthology.orchestra.run.trial191.stdout:================================================================================ 2026-02-23T11:10:12.586 INFO:teuthology.orchestra.run.trial191.stdout:Installing: 2026-02-23T11:10:12.586 INFO:teuthology.orchestra.run.trial191.stdout: ceph-mgr-rook noarch 2:20.3.0-5439.ge371433f.el9 ceph-noarch 50 k 2026-02-23T11:10:12.586 INFO:teuthology.orchestra.run.trial191.stdout:Installing dependencies: 2026-02-23T11:10:12.587 INFO:teuthology.orchestra.run.trial191.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2026-02-23T11:10:12.587 INFO:teuthology.orchestra.run.trial191.stdout: python3-certifi noarch 2023.05.07-4.el9 epel 14 k 2026-02-23T11:10:12.587 INFO:teuthology.orchestra.run.trial191.stdout: python3-google-auth noarch 1:2.45.0-1.el9 epel 254 k 2026-02-23T11:10:12.587 INFO:teuthology.orchestra.run.trial191.stdout: python3-kubernetes noarch 1:26.1.0-3.el9 epel 1.0 M 2026-02-23T11:10:12.587 INFO:teuthology.orchestra.run.trial191.stdout: python3-pyasn1 noarch 0.4.8-6.el9 appstream 159 k 2026-02-23T11:10:12.587 INFO:teuthology.orchestra.run.trial191.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 appstream 279 k 2026-02-23T11:10:12.587 INFO:teuthology.orchestra.run.trial191.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 appstream 54 k 2026-02-23T11:10:12.587 INFO:teuthology.orchestra.run.trial191.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2026-02-23T11:10:12.587 INFO:teuthology.orchestra.run.trial191.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2026-02-23T11:10:12.587 INFO:teuthology.orchestra.run.trial191.stdout: 2026-02-23T11:10:12.587 INFO:teuthology.orchestra.run.trial191.stdout:Transaction Summary 2026-02-23T11:10:12.587 INFO:teuthology.orchestra.run.trial191.stdout:================================================================================ 2026-02-23T11:10:12.587 INFO:teuthology.orchestra.run.trial191.stdout:Install 10 Packages 2026-02-23T11:10:12.587 INFO:teuthology.orchestra.run.trial191.stdout: 2026-02-23T11:10:12.587 INFO:teuthology.orchestra.run.trial191.stdout:Total download size: 2.0 M 2026-02-23T11:10:12.587 INFO:teuthology.orchestra.run.trial191.stdout:Installed size: 25 M 2026-02-23T11:10:12.587 INFO:teuthology.orchestra.run.trial191.stdout:Downloading Packages: 2026-02-23T11:10:12.861 INFO:teuthology.orchestra.run.trial191.stdout:(1/10): python3-pyasn1-0.4.8-6.el9.noarch.rpm 2.7 MB/s | 159 kB 00:00 2026-02-23T11:10:12.863 INFO:teuthology.orchestra.run.trial191.stdout:(2/10): python3-pyasn1-modules-0.4.8-6.el9.noar 4.6 MB/s | 279 kB 00:00 2026-02-23T11:10:12.872 INFO:teuthology.orchestra.run.trial191.stdout:(3/10): python3-requests-oauthlib-1.3.0-12.el9. 4.6 MB/s | 54 kB 00:00 2026-02-23T11:10:12.996 INFO:teuthology.orchestra.run.trial191.stdout:(4/10): python3-certifi-2023.05.07-4.el9.noarch 114 kB/s | 14 kB 00:00 2026-02-23T11:10:13.004 INFO:teuthology.orchestra.run.trial191.stdout:(5/10): python3-cachetools-4.2.4-1.el9.noarch.r 227 kB/s | 32 kB 00:00 2026-02-23T11:10:13.119 INFO:teuthology.orchestra.run.trial191.stdout:(6/10): ceph-mgr-rook-20.3.0-5439.ge371433f.el9 159 kB/s | 50 kB 00:00 2026-02-23T11:10:13.127 INFO:teuthology.orchestra.run.trial191.stdout:(7/10): python3-google-auth-2.45.0-1.el9.noarch 1.9 MB/s | 254 kB 00:00 2026-02-23T11:10:13.164 INFO:teuthology.orchestra.run.trial191.stdout:(8/10): python3-websocket-client-1.2.3-2.el9.no 2.4 MB/s | 90 kB 00:00 2026-02-23T11:10:13.184 INFO:teuthology.orchestra.run.trial191.stdout:(9/10): python3-kubernetes-26.1.0-3.el9.noarch. 5.7 MB/s | 1.0 MB 00:00 2026-02-23T11:10:13.278 INFO:teuthology.orchestra.run.trial191.stdout:(10/10): python3-rsa-4.9-2.el9.noarch.rpm 371 kB/s | 59 kB 00:00 2026-02-23T11:10:13.280 INFO:teuthology.orchestra.run.trial191.stdout:-------------------------------------------------------------------------------- 2026-02-23T11:10:13.280 INFO:teuthology.orchestra.run.trial191.stdout:Total 2.9 MB/s | 2.0 MB 00:00 2026-02-23T11:10:13.308 INFO:teuthology.orchestra.run.trial191.stdout:Running transaction check 2026-02-23T11:10:13.316 INFO:teuthology.orchestra.run.trial191.stdout:Transaction check succeeded. 2026-02-23T11:10:13.317 INFO:teuthology.orchestra.run.trial191.stdout:Running transaction test 2026-02-23T11:10:13.355 INFO:teuthology.orchestra.run.trial191.stdout:Transaction test succeeded. 2026-02-23T11:10:13.356 INFO:teuthology.orchestra.run.trial191.stdout:Running transaction 2026-02-23T11:10:13.430 INFO:teuthology.orchestra.run.trial191.stdout: Preparing : 1/1 2026-02-23T11:10:13.481 INFO:teuthology.orchestra.run.trial191.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/10 2026-02-23T11:10:13.494 INFO:teuthology.orchestra.run.trial191.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/10 2026-02-23T11:10:13.509 INFO:teuthology.orchestra.run.trial191.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/10 2026-02-23T11:10:13.513 INFO:teuthology.orchestra.run.trial191.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/10 2026-02-23T11:10:13.521 INFO:teuthology.orchestra.run.trial191.stdout: Installing : python3-certifi-2023.05.07-4.el9.noarch 5/10 2026-02-23T11:10:13.561 INFO:teuthology.orchestra.run.trial191.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/10 2026-02-23T11:10:13.573 INFO:teuthology.orchestra.run.trial191.stdout: Installing : python3-google-auth-1:2.45.0-1.el9.noarch 7/10 2026-02-23T11:10:13.892 INFO:teuthology.orchestra.run.trial191.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 8/10 2026-02-23T11:10:13.906 INFO:teuthology.orchestra.run.trial191.stdout: Installing : python3-kubernetes-1:26.1.0-3.el9.noarch 9/10 2026-02-23T11:10:13.907 INFO:teuthology.orchestra.run.trial191.stdout: Installing : ceph-mgr-rook-2:20.3.0-5439.ge371433f.el9.noarch 10/10 2026-02-23T11:10:14.016 INFO:teuthology.orchestra.run.trial191.stdout: Running scriptlet: ceph-mgr-rook-2:20.3.0-5439.ge371433f.el9.noarch 10/10 2026-02-23T11:10:14.016 INFO:teuthology.orchestra.run.trial191.stdout: Verifying : ceph-mgr-rook-2:20.3.0-5439.ge371433f.el9.noarch 1/10 2026-02-23T11:10:14.016 INFO:teuthology.orchestra.run.trial191.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 2/10 2026-02-23T11:10:14.016 INFO:teuthology.orchestra.run.trial191.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 3/10 2026-02-23T11:10:14.016 INFO:teuthology.orchestra.run.trial191.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 4/10 2026-02-23T11:10:14.016 INFO:teuthology.orchestra.run.trial191.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 5/10 2026-02-23T11:10:14.016 INFO:teuthology.orchestra.run.trial191.stdout: Verifying : python3-certifi-2023.05.07-4.el9.noarch 6/10 2026-02-23T11:10:14.016 INFO:teuthology.orchestra.run.trial191.stdout: Verifying : python3-google-auth-1:2.45.0-1.el9.noarch 7/10 2026-02-23T11:10:14.016 INFO:teuthology.orchestra.run.trial191.stdout: Verifying : python3-kubernetes-1:26.1.0-3.el9.noarch 8/10 2026-02-23T11:10:14.016 INFO:teuthology.orchestra.run.trial191.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 9/10 2026-02-23T11:10:14.075 INFO:teuthology.orchestra.run.trial191.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 10/10 2026-02-23T11:10:14.075 INFO:teuthology.orchestra.run.trial191.stdout: 2026-02-23T11:10:14.075 INFO:teuthology.orchestra.run.trial191.stdout:Installed: 2026-02-23T11:10:14.075 INFO:teuthology.orchestra.run.trial191.stdout: ceph-mgr-rook-2:20.3.0-5439.ge371433f.el9.noarch 2026-02-23T11:10:14.075 INFO:teuthology.orchestra.run.trial191.stdout: python3-cachetools-4.2.4-1.el9.noarch 2026-02-23T11:10:14.075 INFO:teuthology.orchestra.run.trial191.stdout: python3-certifi-2023.05.07-4.el9.noarch 2026-02-23T11:10:14.075 INFO:teuthology.orchestra.run.trial191.stdout: python3-google-auth-1:2.45.0-1.el9.noarch 2026-02-23T11:10:14.076 INFO:teuthology.orchestra.run.trial191.stdout: python3-kubernetes-1:26.1.0-3.el9.noarch 2026-02-23T11:10:14.076 INFO:teuthology.orchestra.run.trial191.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2026-02-23T11:10:14.076 INFO:teuthology.orchestra.run.trial191.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2026-02-23T11:10:14.076 INFO:teuthology.orchestra.run.trial191.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2026-02-23T11:10:14.076 INFO:teuthology.orchestra.run.trial191.stdout: python3-rsa-4.9-2.el9.noarch 2026-02-23T11:10:14.076 INFO:teuthology.orchestra.run.trial191.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2026-02-23T11:10:14.076 INFO:teuthology.orchestra.run.trial191.stdout: 2026-02-23T11:10:14.076 INFO:teuthology.orchestra.run.trial191.stdout:Complete! 2026-02-23T11:10:14.105 DEBUG:teuthology.orchestra.run.trial191:> sudo yum -y install ceph-mgr-cephadm 2026-02-23T11:10:14.417 INFO:teuthology.orchestra.run.trial191.stdout:Last metadata expiration check: 0:01:10 ago on Mon 23 Feb 2026 11:09:04 AM UTC. 2026-02-23T11:10:14.499 INFO:teuthology.orchestra.run.trial191.stdout:Dependencies resolved. 2026-02-23T11:10:14.499 INFO:teuthology.orchestra.run.trial191.stdout:================================================================================ 2026-02-23T11:10:14.500 INFO:teuthology.orchestra.run.trial191.stdout: Package Arch Version Repository Size 2026-02-23T11:10:14.500 INFO:teuthology.orchestra.run.trial191.stdout:================================================================================ 2026-02-23T11:10:14.500 INFO:teuthology.orchestra.run.trial191.stdout:Installing: 2026-02-23T11:10:14.500 INFO:teuthology.orchestra.run.trial191.stdout: ceph-mgr-cephadm noarch 2:20.3.0-5439.ge371433f.el9 ceph-noarch 184 k 2026-02-23T11:10:14.500 INFO:teuthology.orchestra.run.trial191.stdout:Installing dependencies: 2026-02-23T11:10:14.500 INFO:teuthology.orchestra.run.trial191.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2026-02-23T11:10:14.500 INFO:teuthology.orchestra.run.trial191.stdout: python3-babel noarch 2.9.1-2.el9 appstream 6.0 M 2026-02-23T11:10:14.500 INFO:teuthology.orchestra.run.trial191.stdout: python3-jinja2 noarch 2.11.3-8.el9 appstream 249 k 2026-02-23T11:10:14.500 INFO:teuthology.orchestra.run.trial191.stdout: python3-markupsafe x86_64 1.1.1-12.el9 appstream 35 k 2026-02-23T11:10:14.500 INFO:teuthology.orchestra.run.trial191.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2026-02-23T11:10:14.500 INFO:teuthology.orchestra.run.trial191.stdout: python3-typing-extensions noarch 4.15.0-1.el9 epel 86 k 2026-02-23T11:10:14.500 INFO:teuthology.orchestra.run.trial191.stdout: 2026-02-23T11:10:14.500 INFO:teuthology.orchestra.run.trial191.stdout:Transaction Summary 2026-02-23T11:10:14.500 INFO:teuthology.orchestra.run.trial191.stdout:================================================================================ 2026-02-23T11:10:14.500 INFO:teuthology.orchestra.run.trial191.stdout:Install 7 Packages 2026-02-23T11:10:14.500 INFO:teuthology.orchestra.run.trial191.stdout: 2026-02-23T11:10:14.500 INFO:teuthology.orchestra.run.trial191.stdout:Total download size: 7.1 M 2026-02-23T11:10:14.500 INFO:teuthology.orchestra.run.trial191.stdout:Installed size: 34 M 2026-02-23T11:10:14.500 INFO:teuthology.orchestra.run.trial191.stdout:Downloading Packages: 2026-02-23T11:10:15.346 INFO:teuthology.orchestra.run.trial191.stdout:(1/7): ceph-mgr-cephadm-20.3.0-5439.ge371433f.e 287 kB/s | 184 kB 00:00 2026-02-23T11:10:15.389 INFO:teuthology.orchestra.run.trial191.stdout:(2/7): python3-jinja2-2.11.3-8.el9.noarch.rpm 363 kB/s | 249 kB 00:00 2026-02-23T11:10:15.611 INFO:teuthology.orchestra.run.trial191.stdout:(3/7): python3-asyncssh-2.13.2-5.el9.noarch.rpm 2.4 MB/s | 548 kB 00:00 2026-02-23T11:10:15.640 INFO:teuthology.orchestra.run.trial191.stdout:(4/7): python3-natsort-7.1.1-5.el9.noarch.rpm 2.0 MB/s | 58 kB 00:00 2026-02-23T11:10:15.668 INFO:teuthology.orchestra.run.trial191.stdout:(5/7): python3-typing-extensions-4.15.0-1.el9.n 3.1 MB/s | 86 kB 00:00 2026-02-23T11:10:15.770 INFO:teuthology.orchestra.run.trial191.stdout:(6/7): python3-markupsafe-1.1.1-12.el9.x86_64.r 82 kB/s | 35 kB 00:00 2026-02-23T11:10:18.190 INFO:teuthology.orchestra.run.trial191.stdout:(7/7): python3-babel-2.9.1-2.el9.noarch.rpm 1.7 MB/s | 6.0 MB 00:03 2026-02-23T11:10:18.192 INFO:teuthology.orchestra.run.trial191.stdout:-------------------------------------------------------------------------------- 2026-02-23T11:10:18.192 INFO:teuthology.orchestra.run.trial191.stdout:Total 1.9 MB/s | 7.1 MB 00:03 2026-02-23T11:10:18.229 INFO:teuthology.orchestra.run.trial191.stdout:Running transaction check 2026-02-23T11:10:18.235 INFO:teuthology.orchestra.run.trial191.stdout:Transaction check succeeded. 2026-02-23T11:10:18.235 INFO:teuthology.orchestra.run.trial191.stdout:Running transaction test 2026-02-23T11:10:18.306 INFO:teuthology.orchestra.run.trial191.stdout:Transaction test succeeded. 2026-02-23T11:10:18.306 INFO:teuthology.orchestra.run.trial191.stdout:Running transaction 2026-02-23T11:10:18.395 INFO:teuthology.orchestra.run.trial191.stdout: Preparing : 1/1 2026-02-23T11:10:18.444 INFO:teuthology.orchestra.run.trial191.stdout: Installing : python3-typing-extensions-4.15.0-1.el9.noarch 1/7 2026-02-23T11:10:18.453 INFO:teuthology.orchestra.run.trial191.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 2/7 2026-02-23T11:10:18.459 INFO:teuthology.orchestra.run.trial191.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/7 2026-02-23T11:10:18.677 INFO:teuthology.orchestra.run.trial191.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 4/7 2026-02-23T11:10:18.701 INFO:teuthology.orchestra.run.trial191.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 5/7 2026-02-23T11:10:18.725 INFO:teuthology.orchestra.run.trial191.stdout: Installing : python3-jinja2-2.11.3-8.el9.noarch 6/7 2026-02-23T11:10:18.726 INFO:teuthology.orchestra.run.trial191.stdout: Installing : ceph-mgr-cephadm-2:20.3.0-5439.ge371433f.el9.noarch 7/7 2026-02-23T11:10:18.832 INFO:teuthology.orchestra.run.trial191.stdout: Running scriptlet: ceph-mgr-cephadm-2:20.3.0-5439.ge371433f.el9.noarch 7/7 2026-02-23T11:10:18.832 INFO:teuthology.orchestra.run.trial191.stdout: Verifying : ceph-mgr-cephadm-2:20.3.0-5439.ge371433f.el9.noarch 1/7 2026-02-23T11:10:18.832 INFO:teuthology.orchestra.run.trial191.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/7 2026-02-23T11:10:18.832 INFO:teuthology.orchestra.run.trial191.stdout: Verifying : python3-jinja2-2.11.3-8.el9.noarch 3/7 2026-02-23T11:10:18.833 INFO:teuthology.orchestra.run.trial191.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 4/7 2026-02-23T11:10:18.833 INFO:teuthology.orchestra.run.trial191.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 5/7 2026-02-23T11:10:18.833 INFO:teuthology.orchestra.run.trial191.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 6/7 2026-02-23T11:10:18.888 INFO:teuthology.orchestra.run.trial191.stdout: Verifying : python3-typing-extensions-4.15.0-1.el9.noarch 7/7 2026-02-23T11:10:18.888 INFO:teuthology.orchestra.run.trial191.stdout: 2026-02-23T11:10:18.888 INFO:teuthology.orchestra.run.trial191.stdout:Installed: 2026-02-23T11:10:18.888 INFO:teuthology.orchestra.run.trial191.stdout: ceph-mgr-cephadm-2:20.3.0-5439.ge371433f.el9.noarch 2026-02-23T11:10:18.888 INFO:teuthology.orchestra.run.trial191.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2026-02-23T11:10:18.888 INFO:teuthology.orchestra.run.trial191.stdout: python3-babel-2.9.1-2.el9.noarch 2026-02-23T11:10:18.889 INFO:teuthology.orchestra.run.trial191.stdout: python3-jinja2-2.11.3-8.el9.noarch 2026-02-23T11:10:18.889 INFO:teuthology.orchestra.run.trial191.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2026-02-23T11:10:18.889 INFO:teuthology.orchestra.run.trial191.stdout: python3-natsort-7.1.1-5.el9.noarch 2026-02-23T11:10:18.889 INFO:teuthology.orchestra.run.trial191.stdout: python3-typing-extensions-4.15.0-1.el9.noarch 2026-02-23T11:10:18.889 INFO:teuthology.orchestra.run.trial191.stdout: 2026-02-23T11:10:18.889 INFO:teuthology.orchestra.run.trial191.stdout:Complete! 2026-02-23T11:10:18.917 DEBUG:teuthology.orchestra.run.trial191:> sudo yum -y install ceph-osd 2026-02-23T11:10:19.226 INFO:teuthology.orchestra.run.trial191.stdout:Last metadata expiration check: 0:01:15 ago on Mon 23 Feb 2026 11:09:04 AM UTC. 2026-02-23T11:10:19.290 INFO:teuthology.orchestra.run.trial191.stdout:Package ceph-osd-2:20.3.0-5439.ge371433f.el9.x86_64 is already installed. 2026-02-23T11:10:19.308 INFO:teuthology.orchestra.run.trial191.stdout:Dependencies resolved. 2026-02-23T11:10:19.308 INFO:teuthology.orchestra.run.trial191.stdout:Nothing to do. 2026-02-23T11:10:19.309 INFO:teuthology.orchestra.run.trial191.stdout:Complete! 2026-02-23T11:10:19.330 DEBUG:teuthology.orchestra.run.trial191:> sudo yum -y install ceph-osd-classic 2026-02-23T11:10:19.628 INFO:teuthology.orchestra.run.trial191.stdout:Last metadata expiration check: 0:01:15 ago on Mon 23 Feb 2026 11:09:04 AM UTC. 2026-02-23T11:10:19.693 INFO:teuthology.orchestra.run.trial191.stdout:Package ceph-osd-classic-2:20.3.0-5439.ge371433f.el9.x86_64 is already installed. 2026-02-23T11:10:19.711 INFO:teuthology.orchestra.run.trial191.stdout:Dependencies resolved. 2026-02-23T11:10:19.711 INFO:teuthology.orchestra.run.trial191.stdout:Nothing to do. 2026-02-23T11:10:19.711 INFO:teuthology.orchestra.run.trial191.stdout:Complete! 2026-02-23T11:10:19.731 DEBUG:teuthology.orchestra.run.trial191:> sudo yum -y install ceph-fuse 2026-02-23T11:10:20.040 INFO:teuthology.orchestra.run.trial191.stdout:Last metadata expiration check: 0:01:16 ago on Mon 23 Feb 2026 11:09:04 AM UTC. 2026-02-23T11:10:20.123 INFO:teuthology.orchestra.run.trial191.stdout:Dependencies resolved. 2026-02-23T11:10:20.123 INFO:teuthology.orchestra.run.trial191.stdout:================================================================================ 2026-02-23T11:10:20.123 INFO:teuthology.orchestra.run.trial191.stdout: Package Arch Version Repo Size 2026-02-23T11:10:20.123 INFO:teuthology.orchestra.run.trial191.stdout:================================================================================ 2026-02-23T11:10:20.123 INFO:teuthology.orchestra.run.trial191.stdout:Installing: 2026-02-23T11:10:20.124 INFO:teuthology.orchestra.run.trial191.stdout: ceph-fuse x86_64 2:20.3.0-5439.ge371433f.el9 ceph 914 k 2026-02-23T11:10:20.124 INFO:teuthology.orchestra.run.trial191.stdout: 2026-02-23T11:10:20.124 INFO:teuthology.orchestra.run.trial191.stdout:Transaction Summary 2026-02-23T11:10:20.124 INFO:teuthology.orchestra.run.trial191.stdout:================================================================================ 2026-02-23T11:10:20.124 INFO:teuthology.orchestra.run.trial191.stdout:Install 1 Package 2026-02-23T11:10:20.124 INFO:teuthology.orchestra.run.trial191.stdout: 2026-02-23T11:10:20.124 INFO:teuthology.orchestra.run.trial191.stdout:Total download size: 914 k 2026-02-23T11:10:20.124 INFO:teuthology.orchestra.run.trial191.stdout:Installed size: 2.9 M 2026-02-23T11:10:20.124 INFO:teuthology.orchestra.run.trial191.stdout:Downloading Packages: 2026-02-23T11:10:20.791 INFO:teuthology.orchestra.run.trial191.stdout:ceph-fuse-20.3.0-5439.ge371433f.el9.x86_64.rpm 1.3 MB/s | 914 kB 00:00 2026-02-23T11:10:20.791 INFO:teuthology.orchestra.run.trial191.stdout:-------------------------------------------------------------------------------- 2026-02-23T11:10:20.792 INFO:teuthology.orchestra.run.trial191.stdout:Total 1.3 MB/s | 914 kB 00:00 2026-02-23T11:10:20.792 INFO:teuthology.orchestra.run.trial191.stdout:Running transaction check 2026-02-23T11:10:20.796 INFO:teuthology.orchestra.run.trial191.stdout:Transaction check succeeded. 2026-02-23T11:10:20.796 INFO:teuthology.orchestra.run.trial191.stdout:Running transaction test 2026-02-23T11:10:20.829 INFO:teuthology.orchestra.run.trial191.stdout:Transaction test succeeded. 2026-02-23T11:10:20.829 INFO:teuthology.orchestra.run.trial191.stdout:Running transaction 2026-02-23T11:10:20.918 INFO:teuthology.orchestra.run.trial191.stdout: Preparing : 1/1 2026-02-23T11:10:20.935 INFO:teuthology.orchestra.run.trial191.stdout: Installing : ceph-fuse-2:20.3.0-5439.ge371433f.el9.x86_64 1/1 2026-02-23T11:10:21.176 INFO:teuthology.orchestra.run.trial191.stdout: Running scriptlet: ceph-fuse-2:20.3.0-5439.ge371433f.el9.x86_64 1/1 2026-02-23T11:10:21.234 INFO:teuthology.orchestra.run.trial191.stdout: Verifying : ceph-fuse-2:20.3.0-5439.ge371433f.el9.x86_64 1/1 2026-02-23T11:10:21.234 INFO:teuthology.orchestra.run.trial191.stdout: 2026-02-23T11:10:21.234 INFO:teuthology.orchestra.run.trial191.stdout:Installed: 2026-02-23T11:10:21.234 INFO:teuthology.orchestra.run.trial191.stdout: ceph-fuse-2:20.3.0-5439.ge371433f.el9.x86_64 2026-02-23T11:10:21.234 INFO:teuthology.orchestra.run.trial191.stdout: 2026-02-23T11:10:21.234 INFO:teuthology.orchestra.run.trial191.stdout:Complete! 2026-02-23T11:10:21.263 DEBUG:teuthology.orchestra.run.trial191:> sudo yum -y install ceph-volume 2026-02-23T11:10:21.573 INFO:teuthology.orchestra.run.trial191.stdout:Last metadata expiration check: 0:01:17 ago on Mon 23 Feb 2026 11:09:04 AM UTC. 2026-02-23T11:10:21.655 INFO:teuthology.orchestra.run.trial191.stdout:Dependencies resolved. 2026-02-23T11:10:21.656 INFO:teuthology.orchestra.run.trial191.stdout:================================================================================ 2026-02-23T11:10:21.656 INFO:teuthology.orchestra.run.trial191.stdout: Package Arch Version Repository Size 2026-02-23T11:10:21.656 INFO:teuthology.orchestra.run.trial191.stdout:================================================================================ 2026-02-23T11:10:21.656 INFO:teuthology.orchestra.run.trial191.stdout:Installing: 2026-02-23T11:10:21.656 INFO:teuthology.orchestra.run.trial191.stdout: ceph-volume noarch 2:20.3.0-5439.ge371433f.el9 ceph-noarch 301 k 2026-02-23T11:10:21.656 INFO:teuthology.orchestra.run.trial191.stdout:Installing dependencies: 2026-02-23T11:10:21.656 INFO:teuthology.orchestra.run.trial191.stdout: cryptsetup x86_64 2.8.1-3.el9 baseos 351 k 2026-02-23T11:10:21.656 INFO:teuthology.orchestra.run.trial191.stdout: 2026-02-23T11:10:21.656 INFO:teuthology.orchestra.run.trial191.stdout:Transaction Summary 2026-02-23T11:10:21.656 INFO:teuthology.orchestra.run.trial191.stdout:================================================================================ 2026-02-23T11:10:21.656 INFO:teuthology.orchestra.run.trial191.stdout:Install 2 Packages 2026-02-23T11:10:21.656 INFO:teuthology.orchestra.run.trial191.stdout: 2026-02-23T11:10:21.656 INFO:teuthology.orchestra.run.trial191.stdout:Total download size: 652 k 2026-02-23T11:10:21.656 INFO:teuthology.orchestra.run.trial191.stdout:Installed size: 2.2 M 2026-02-23T11:10:21.656 INFO:teuthology.orchestra.run.trial191.stdout:Downloading Packages: 2026-02-23T11:10:22.031 INFO:teuthology.orchestra.run.trial191.stdout:(1/2): cryptsetup-2.8.1-3.el9.x86_64.rpm 3.1 MB/s | 351 kB 00:00 2026-02-23T11:10:22.447 INFO:teuthology.orchestra.run.trial191.stdout:(2/2): ceph-volume-20.3.0-5439.ge371433f.el9.no 571 kB/s | 301 kB 00:00 2026-02-23T11:10:22.447 INFO:teuthology.orchestra.run.trial191.stdout:-------------------------------------------------------------------------------- 2026-02-23T11:10:22.447 INFO:teuthology.orchestra.run.trial191.stdout:Total 825 kB/s | 652 kB 00:00 2026-02-23T11:10:22.455 INFO:teuthology.orchestra.run.trial191.stdout:Running transaction check 2026-02-23T11:10:22.461 INFO:teuthology.orchestra.run.trial191.stdout:Transaction check succeeded. 2026-02-23T11:10:22.461 INFO:teuthology.orchestra.run.trial191.stdout:Running transaction test 2026-02-23T11:10:22.505 INFO:teuthology.orchestra.run.trial191.stdout:Transaction test succeeded. 2026-02-23T11:10:22.505 INFO:teuthology.orchestra.run.trial191.stdout:Running transaction 2026-02-23T11:10:22.586 INFO:teuthology.orchestra.run.trial191.stdout: Preparing : 1/1 2026-02-23T11:10:22.648 INFO:teuthology.orchestra.run.trial191.stdout: Installing : cryptsetup-2.8.1-3.el9.x86_64 1/2 2026-02-23T11:10:22.650 INFO:teuthology.orchestra.run.trial191.stdout: Installing : ceph-volume-2:20.3.0-5439.ge371433f.el9.noarch 2/2 2026-02-23T11:10:22.665 INFO:teuthology.orchestra.run.trial191.stdout: Running scriptlet: ceph-volume-2:20.3.0-5439.ge371433f.el9.noarch 2/2 2026-02-23T11:10:22.665 INFO:teuthology.orchestra.run.trial191.stdout:Glob pattern passed to enable, but globs are not supported for this. 2026-02-23T11:10:22.666 INFO:teuthology.orchestra.run.trial191.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2026-02-23T11:10:22.666 INFO:teuthology.orchestra.run.trial191.stdout: 2026-02-23T11:10:22.889 INFO:teuthology.orchestra.run.trial191.stdout: Verifying : ceph-volume-2:20.3.0-5439.ge371433f.el9.noarch 1/2 2026-02-23T11:10:22.947 INFO:teuthology.orchestra.run.trial191.stdout: Verifying : cryptsetup-2.8.1-3.el9.x86_64 2/2 2026-02-23T11:10:22.947 INFO:teuthology.orchestra.run.trial191.stdout: 2026-02-23T11:10:22.947 INFO:teuthology.orchestra.run.trial191.stdout:Installed: 2026-02-23T11:10:22.947 INFO:teuthology.orchestra.run.trial191.stdout: ceph-volume-2:20.3.0-5439.ge371433f.el9.noarch cryptsetup-2.8.1-3.el9.x86_64 2026-02-23T11:10:22.947 INFO:teuthology.orchestra.run.trial191.stdout: 2026-02-23T11:10:22.947 INFO:teuthology.orchestra.run.trial191.stdout:Complete! 2026-02-23T11:10:22.975 DEBUG:teuthology.orchestra.run.trial191:> sudo yum -y install librados-devel 2026-02-23T11:10:23.283 INFO:teuthology.orchestra.run.trial191.stdout:Last metadata expiration check: 0:01:19 ago on Mon 23 Feb 2026 11:09:04 AM UTC. 2026-02-23T11:10:23.366 INFO:teuthology.orchestra.run.trial191.stdout:Dependencies resolved. 2026-02-23T11:10:23.367 INFO:teuthology.orchestra.run.trial191.stdout:================================================================================ 2026-02-23T11:10:23.367 INFO:teuthology.orchestra.run.trial191.stdout: Package Arch Version Repo Size 2026-02-23T11:10:23.367 INFO:teuthology.orchestra.run.trial191.stdout:================================================================================ 2026-02-23T11:10:23.367 INFO:teuthology.orchestra.run.trial191.stdout:Installing: 2026-02-23T11:10:23.367 INFO:teuthology.orchestra.run.trial191.stdout: librados-devel x86_64 2:20.3.0-5439.ge371433f.el9 ceph 124 k 2026-02-23T11:10:23.367 INFO:teuthology.orchestra.run.trial191.stdout: 2026-02-23T11:10:23.367 INFO:teuthology.orchestra.run.trial191.stdout:Transaction Summary 2026-02-23T11:10:23.367 INFO:teuthology.orchestra.run.trial191.stdout:================================================================================ 2026-02-23T11:10:23.367 INFO:teuthology.orchestra.run.trial191.stdout:Install 1 Package 2026-02-23T11:10:23.367 INFO:teuthology.orchestra.run.trial191.stdout: 2026-02-23T11:10:23.367 INFO:teuthology.orchestra.run.trial191.stdout:Total download size: 124 k 2026-02-23T11:10:23.367 INFO:teuthology.orchestra.run.trial191.stdout:Installed size: 434 k 2026-02-23T11:10:23.367 INFO:teuthology.orchestra.run.trial191.stdout:Downloading Packages: 2026-02-23T11:10:23.937 INFO:teuthology.orchestra.run.trial191.stdout:librados-devel-20.3.0-5439.ge371433f.el9.x86_64 217 kB/s | 124 kB 00:00 2026-02-23T11:10:23.938 INFO:teuthology.orchestra.run.trial191.stdout:-------------------------------------------------------------------------------- 2026-02-23T11:10:23.938 INFO:teuthology.orchestra.run.trial191.stdout:Total 217 kB/s | 124 kB 00:00 2026-02-23T11:10:23.938 INFO:teuthology.orchestra.run.trial191.stdout:Running transaction check 2026-02-23T11:10:23.941 INFO:teuthology.orchestra.run.trial191.stdout:Transaction check succeeded. 2026-02-23T11:10:23.941 INFO:teuthology.orchestra.run.trial191.stdout:Running transaction test 2026-02-23T11:10:23.972 INFO:teuthology.orchestra.run.trial191.stdout:Transaction test succeeded. 2026-02-23T11:10:23.972 INFO:teuthology.orchestra.run.trial191.stdout:Running transaction 2026-02-23T11:10:24.026 INFO:teuthology.orchestra.run.trial191.stdout: Preparing : 1/1 2026-02-23T11:10:24.042 INFO:teuthology.orchestra.run.trial191.stdout: Installing : librados-devel-2:20.3.0-5439.ge371433f.el9.x86_64 1/1 2026-02-23T11:10:24.121 INFO:teuthology.orchestra.run.trial191.stdout: Running scriptlet: librados-devel-2:20.3.0-5439.ge371433f.el9.x86_64 1/1 2026-02-23T11:10:24.178 INFO:teuthology.orchestra.run.trial191.stdout: Verifying : librados-devel-2:20.3.0-5439.ge371433f.el9.x86_64 1/1 2026-02-23T11:10:24.178 INFO:teuthology.orchestra.run.trial191.stdout: 2026-02-23T11:10:24.178 INFO:teuthology.orchestra.run.trial191.stdout:Installed: 2026-02-23T11:10:24.178 INFO:teuthology.orchestra.run.trial191.stdout: librados-devel-2:20.3.0-5439.ge371433f.el9.x86_64 2026-02-23T11:10:24.178 INFO:teuthology.orchestra.run.trial191.stdout: 2026-02-23T11:10:24.178 INFO:teuthology.orchestra.run.trial191.stdout:Complete! 2026-02-23T11:10:24.206 DEBUG:teuthology.orchestra.run.trial191:> sudo yum -y install libcephfs2 2026-02-23T11:10:24.519 INFO:teuthology.orchestra.run.trial191.stdout:Last metadata expiration check: 0:01:20 ago on Mon 23 Feb 2026 11:09:04 AM UTC. 2026-02-23T11:10:24.583 INFO:teuthology.orchestra.run.trial191.stdout:Package libcephfs2-2:20.3.0-5439.ge371433f.el9.x86_64 is already installed. 2026-02-23T11:10:24.601 INFO:teuthology.orchestra.run.trial191.stdout:Dependencies resolved. 2026-02-23T11:10:24.602 INFO:teuthology.orchestra.run.trial191.stdout:Nothing to do. 2026-02-23T11:10:24.602 INFO:teuthology.orchestra.run.trial191.stdout:Complete! 2026-02-23T11:10:24.623 DEBUG:teuthology.orchestra.run.trial191:> sudo yum -y install libcephfs-devel 2026-02-23T11:10:24.933 INFO:teuthology.orchestra.run.trial191.stdout:Last metadata expiration check: 0:01:20 ago on Mon 23 Feb 2026 11:09:04 AM UTC. 2026-02-23T11:10:25.016 INFO:teuthology.orchestra.run.trial191.stdout:Dependencies resolved. 2026-02-23T11:10:25.016 INFO:teuthology.orchestra.run.trial191.stdout:================================================================================ 2026-02-23T11:10:25.016 INFO:teuthology.orchestra.run.trial191.stdout: Package Arch Version Repo Size 2026-02-23T11:10:25.017 INFO:teuthology.orchestra.run.trial191.stdout:================================================================================ 2026-02-23T11:10:25.017 INFO:teuthology.orchestra.run.trial191.stdout:Installing: 2026-02-23T11:10:25.017 INFO:teuthology.orchestra.run.trial191.stdout: libcephfs-devel x86_64 2:20.3.0-5439.ge371433f.el9 ceph 37 k 2026-02-23T11:10:25.017 INFO:teuthology.orchestra.run.trial191.stdout:Installing dependencies: 2026-02-23T11:10:25.017 INFO:teuthology.orchestra.run.trial191.stdout: libcephfs-proxy2 x86_64 2:20.3.0-5439.ge371433f.el9 ceph 26 k 2026-02-23T11:10:25.017 INFO:teuthology.orchestra.run.trial191.stdout: 2026-02-23T11:10:25.017 INFO:teuthology.orchestra.run.trial191.stdout:Transaction Summary 2026-02-23T11:10:25.017 INFO:teuthology.orchestra.run.trial191.stdout:================================================================================ 2026-02-23T11:10:25.017 INFO:teuthology.orchestra.run.trial191.stdout:Install 2 Packages 2026-02-23T11:10:25.017 INFO:teuthology.orchestra.run.trial191.stdout: 2026-02-23T11:10:25.017 INFO:teuthology.orchestra.run.trial191.stdout:Total download size: 64 k 2026-02-23T11:10:25.018 INFO:teuthology.orchestra.run.trial191.stdout:Installed size: 231 k 2026-02-23T11:10:25.018 INFO:teuthology.orchestra.run.trial191.stdout:Downloading Packages: 2026-02-23T11:10:25.309 INFO:teuthology.orchestra.run.trial191.stdout:(1/2): libcephfs-proxy2-20.3.0-5439.ge371433f.e 90 kB/s | 26 kB 00:00 2026-02-23T11:10:25.364 INFO:teuthology.orchestra.run.trial191.stdout:(2/2): libcephfs-devel-20.3.0-5439.ge371433f.el 107 kB/s | 37 kB 00:00 2026-02-23T11:10:25.365 INFO:teuthology.orchestra.run.trial191.stdout:-------------------------------------------------------------------------------- 2026-02-23T11:10:25.365 INFO:teuthology.orchestra.run.trial191.stdout:Total 182 kB/s | 64 kB 00:00 2026-02-23T11:10:25.365 INFO:teuthology.orchestra.run.trial191.stdout:Running transaction check 2026-02-23T11:10:25.368 INFO:teuthology.orchestra.run.trial191.stdout:Transaction check succeeded. 2026-02-23T11:10:25.368 INFO:teuthology.orchestra.run.trial191.stdout:Running transaction test 2026-02-23T11:10:25.400 INFO:teuthology.orchestra.run.trial191.stdout:Transaction test succeeded. 2026-02-23T11:10:25.400 INFO:teuthology.orchestra.run.trial191.stdout:Running transaction 2026-02-23T11:10:25.448 INFO:teuthology.orchestra.run.trial191.stdout: Preparing : 1/1 2026-02-23T11:10:25.449 INFO:teuthology.orchestra.run.trial191.stdout: Installing : libcephfs-proxy2-2:20.3.0-5439.ge371433f.el9.x86_64 1/2 2026-02-23T11:10:25.464 INFO:teuthology.orchestra.run.trial191.stdout: Running scriptlet: libcephfs-proxy2-2:20.3.0-5439.ge371433f.el9.x86_64 1/2 2026-02-23T11:10:25.479 INFO:teuthology.orchestra.run.trial191.stdout: Installing : libcephfs-devel-2:20.3.0-5439.ge371433f.el9.x86_64 2/2 2026-02-23T11:10:25.537 INFO:teuthology.orchestra.run.trial191.stdout: Running scriptlet: libcephfs-devel-2:20.3.0-5439.ge371433f.el9.x86_64 2/2 2026-02-23T11:10:25.537 INFO:teuthology.orchestra.run.trial191.stdout: Verifying : libcephfs-devel-2:20.3.0-5439.ge371433f.el9.x86_64 1/2 2026-02-23T11:10:25.595 INFO:teuthology.orchestra.run.trial191.stdout: Verifying : libcephfs-proxy2-2:20.3.0-5439.ge371433f.el9.x86_64 2/2 2026-02-23T11:10:25.595 INFO:teuthology.orchestra.run.trial191.stdout: 2026-02-23T11:10:25.595 INFO:teuthology.orchestra.run.trial191.stdout:Installed: 2026-02-23T11:10:25.595 INFO:teuthology.orchestra.run.trial191.stdout: libcephfs-devel-2:20.3.0-5439.ge371433f.el9.x86_64 2026-02-23T11:10:25.595 INFO:teuthology.orchestra.run.trial191.stdout: libcephfs-proxy2-2:20.3.0-5439.ge371433f.el9.x86_64 2026-02-23T11:10:25.596 INFO:teuthology.orchestra.run.trial191.stdout: 2026-02-23T11:10:25.596 INFO:teuthology.orchestra.run.trial191.stdout:Complete! 2026-02-23T11:10:25.623 DEBUG:teuthology.orchestra.run.trial191:> sudo yum -y install librados2 2026-02-23T11:10:25.935 INFO:teuthology.orchestra.run.trial191.stdout:Last metadata expiration check: 0:01:21 ago on Mon 23 Feb 2026 11:09:04 AM UTC. 2026-02-23T11:10:25.999 INFO:teuthology.orchestra.run.trial191.stdout:Package librados2-2:20.3.0-5439.ge371433f.el9.x86_64 is already installed. 2026-02-23T11:10:26.018 INFO:teuthology.orchestra.run.trial191.stdout:Dependencies resolved. 2026-02-23T11:10:26.018 INFO:teuthology.orchestra.run.trial191.stdout:Nothing to do. 2026-02-23T11:10:26.018 INFO:teuthology.orchestra.run.trial191.stdout:Complete! 2026-02-23T11:10:26.040 DEBUG:teuthology.orchestra.run.trial191:> sudo yum -y install librbd1 2026-02-23T11:10:26.350 INFO:teuthology.orchestra.run.trial191.stdout:Last metadata expiration check: 0:01:22 ago on Mon 23 Feb 2026 11:09:04 AM UTC. 2026-02-23T11:10:26.415 INFO:teuthology.orchestra.run.trial191.stdout:Package librbd1-2:20.3.0-5439.ge371433f.el9.x86_64 is already installed. 2026-02-23T11:10:26.433 INFO:teuthology.orchestra.run.trial191.stdout:Dependencies resolved. 2026-02-23T11:10:26.433 INFO:teuthology.orchestra.run.trial191.stdout:Nothing to do. 2026-02-23T11:10:26.434 INFO:teuthology.orchestra.run.trial191.stdout:Complete! 2026-02-23T11:10:26.454 DEBUG:teuthology.orchestra.run.trial191:> sudo yum -y install python3-rados 2026-02-23T11:10:26.761 INFO:teuthology.orchestra.run.trial191.stdout:Last metadata expiration check: 0:01:22 ago on Mon 23 Feb 2026 11:09:04 AM UTC. 2026-02-23T11:10:26.825 INFO:teuthology.orchestra.run.trial191.stdout:Package python3-rados-2:20.3.0-5439.ge371433f.el9.x86_64 is already installed. 2026-02-23T11:10:26.843 INFO:teuthology.orchestra.run.trial191.stdout:Dependencies resolved. 2026-02-23T11:10:26.843 INFO:teuthology.orchestra.run.trial191.stdout:Nothing to do. 2026-02-23T11:10:26.843 INFO:teuthology.orchestra.run.trial191.stdout:Complete! 2026-02-23T11:10:26.865 DEBUG:teuthology.orchestra.run.trial191:> sudo yum -y install python3-rgw 2026-02-23T11:10:27.175 INFO:teuthology.orchestra.run.trial191.stdout:Last metadata expiration check: 0:01:23 ago on Mon 23 Feb 2026 11:09:04 AM UTC. 2026-02-23T11:10:27.239 INFO:teuthology.orchestra.run.trial191.stdout:Package python3-rgw-2:20.3.0-5439.ge371433f.el9.x86_64 is already installed. 2026-02-23T11:10:27.257 INFO:teuthology.orchestra.run.trial191.stdout:Dependencies resolved. 2026-02-23T11:10:27.258 INFO:teuthology.orchestra.run.trial191.stdout:Nothing to do. 2026-02-23T11:10:27.258 INFO:teuthology.orchestra.run.trial191.stdout:Complete! 2026-02-23T11:10:27.279 DEBUG:teuthology.orchestra.run.trial191:> sudo yum -y install python3-cephfs 2026-02-23T11:10:27.584 INFO:teuthology.orchestra.run.trial191.stdout:Last metadata expiration check: 0:01:23 ago on Mon 23 Feb 2026 11:09:04 AM UTC. 2026-02-23T11:10:27.649 INFO:teuthology.orchestra.run.trial191.stdout:Package python3-cephfs-2:20.3.0-5439.ge371433f.el9.x86_64 is already installed. 2026-02-23T11:10:27.667 INFO:teuthology.orchestra.run.trial191.stdout:Dependencies resolved. 2026-02-23T11:10:27.667 INFO:teuthology.orchestra.run.trial191.stdout:Nothing to do. 2026-02-23T11:10:27.667 INFO:teuthology.orchestra.run.trial191.stdout:Complete! 2026-02-23T11:10:27.688 DEBUG:teuthology.orchestra.run.trial191:> sudo yum -y install python3-rbd 2026-02-23T11:10:27.999 INFO:teuthology.orchestra.run.trial191.stdout:Last metadata expiration check: 0:01:23 ago on Mon 23 Feb 2026 11:09:04 AM UTC. 2026-02-23T11:10:28.063 INFO:teuthology.orchestra.run.trial191.stdout:Package python3-rbd-2:20.3.0-5439.ge371433f.el9.x86_64 is already installed. 2026-02-23T11:10:28.082 INFO:teuthology.orchestra.run.trial191.stdout:Dependencies resolved. 2026-02-23T11:10:28.083 INFO:teuthology.orchestra.run.trial191.stdout:Nothing to do. 2026-02-23T11:10:28.083 INFO:teuthology.orchestra.run.trial191.stdout:Complete! 2026-02-23T11:10:28.104 DEBUG:teuthology.orchestra.run.trial191:> sudo yum -y install rbd-fuse 2026-02-23T11:10:28.413 INFO:teuthology.orchestra.run.trial191.stdout:Last metadata expiration check: 0:01:24 ago on Mon 23 Feb 2026 11:09:04 AM UTC. 2026-02-23T11:10:28.496 INFO:teuthology.orchestra.run.trial191.stdout:Dependencies resolved. 2026-02-23T11:10:28.496 INFO:teuthology.orchestra.run.trial191.stdout:================================================================================ 2026-02-23T11:10:28.496 INFO:teuthology.orchestra.run.trial191.stdout: Package Architecture Version Repository Size 2026-02-23T11:10:28.496 INFO:teuthology.orchestra.run.trial191.stdout:================================================================================ 2026-02-23T11:10:28.496 INFO:teuthology.orchestra.run.trial191.stdout:Installing: 2026-02-23T11:10:28.496 INFO:teuthology.orchestra.run.trial191.stdout: rbd-fuse x86_64 2:20.3.0-5439.ge371433f.el9 ceph 86 k 2026-02-23T11:10:28.496 INFO:teuthology.orchestra.run.trial191.stdout: 2026-02-23T11:10:28.496 INFO:teuthology.orchestra.run.trial191.stdout:Transaction Summary 2026-02-23T11:10:28.496 INFO:teuthology.orchestra.run.trial191.stdout:================================================================================ 2026-02-23T11:10:28.496 INFO:teuthology.orchestra.run.trial191.stdout:Install 1 Package 2026-02-23T11:10:28.496 INFO:teuthology.orchestra.run.trial191.stdout: 2026-02-23T11:10:28.497 INFO:teuthology.orchestra.run.trial191.stdout:Total download size: 86 k 2026-02-23T11:10:28.497 INFO:teuthology.orchestra.run.trial191.stdout:Installed size: 236 k 2026-02-23T11:10:28.497 INFO:teuthology.orchestra.run.trial191.stdout:Downloading Packages: 2026-02-23T11:10:28.899 INFO:teuthology.orchestra.run.trial191.stdout:rbd-fuse-20.3.0-5439.ge371433f.el9.x86_64.rpm 215 kB/s | 86 kB 00:00 2026-02-23T11:10:28.899 INFO:teuthology.orchestra.run.trial191.stdout:-------------------------------------------------------------------------------- 2026-02-23T11:10:28.900 INFO:teuthology.orchestra.run.trial191.stdout:Total 214 kB/s | 86 kB 00:00 2026-02-23T11:10:28.900 INFO:teuthology.orchestra.run.trial191.stdout:Running transaction check 2026-02-23T11:10:28.904 INFO:teuthology.orchestra.run.trial191.stdout:Transaction check succeeded. 2026-02-23T11:10:28.904 INFO:teuthology.orchestra.run.trial191.stdout:Running transaction test 2026-02-23T11:10:28.936 INFO:teuthology.orchestra.run.trial191.stdout:Transaction test succeeded. 2026-02-23T11:10:28.936 INFO:teuthology.orchestra.run.trial191.stdout:Running transaction 2026-02-23T11:10:28.987 INFO:teuthology.orchestra.run.trial191.stdout: Preparing : 1/1 2026-02-23T11:10:29.003 INFO:teuthology.orchestra.run.trial191.stdout: Installing : rbd-fuse-2:20.3.0-5439.ge371433f.el9.x86_64 1/1 2026-02-23T11:10:29.090 INFO:teuthology.orchestra.run.trial191.stdout: Running scriptlet: rbd-fuse-2:20.3.0-5439.ge371433f.el9.x86_64 1/1 2026-02-23T11:10:29.148 INFO:teuthology.orchestra.run.trial191.stdout: Verifying : rbd-fuse-2:20.3.0-5439.ge371433f.el9.x86_64 1/1 2026-02-23T11:10:29.148 INFO:teuthology.orchestra.run.trial191.stdout: 2026-02-23T11:10:29.148 INFO:teuthology.orchestra.run.trial191.stdout:Installed: 2026-02-23T11:10:29.148 INFO:teuthology.orchestra.run.trial191.stdout: rbd-fuse-2:20.3.0-5439.ge371433f.el9.x86_64 2026-02-23T11:10:29.148 INFO:teuthology.orchestra.run.trial191.stdout: 2026-02-23T11:10:29.149 INFO:teuthology.orchestra.run.trial191.stdout:Complete! 2026-02-23T11:10:29.179 DEBUG:teuthology.orchestra.run.trial191:> sudo yum -y install rbd-mirror 2026-02-23T11:10:29.490 INFO:teuthology.orchestra.run.trial191.stdout:Last metadata expiration check: 0:01:25 ago on Mon 23 Feb 2026 11:09:04 AM UTC. 2026-02-23T11:10:29.573 INFO:teuthology.orchestra.run.trial191.stdout:Dependencies resolved. 2026-02-23T11:10:29.573 INFO:teuthology.orchestra.run.trial191.stdout:================================================================================ 2026-02-23T11:10:29.573 INFO:teuthology.orchestra.run.trial191.stdout: Package Arch Version Repo Size 2026-02-23T11:10:29.573 INFO:teuthology.orchestra.run.trial191.stdout:================================================================================ 2026-02-23T11:10:29.573 INFO:teuthology.orchestra.run.trial191.stdout:Installing: 2026-02-23T11:10:29.573 INFO:teuthology.orchestra.run.trial191.stdout: rbd-mirror x86_64 2:20.3.0-5439.ge371433f.el9 ceph 2.6 M 2026-02-23T11:10:29.574 INFO:teuthology.orchestra.run.trial191.stdout: 2026-02-23T11:10:29.574 INFO:teuthology.orchestra.run.trial191.stdout:Transaction Summary 2026-02-23T11:10:29.574 INFO:teuthology.orchestra.run.trial191.stdout:================================================================================ 2026-02-23T11:10:29.574 INFO:teuthology.orchestra.run.trial191.stdout:Install 1 Package 2026-02-23T11:10:29.574 INFO:teuthology.orchestra.run.trial191.stdout: 2026-02-23T11:10:29.574 INFO:teuthology.orchestra.run.trial191.stdout:Total download size: 2.6 M 2026-02-23T11:10:29.574 INFO:teuthology.orchestra.run.trial191.stdout:Installed size: 11 M 2026-02-23T11:10:29.574 INFO:teuthology.orchestra.run.trial191.stdout:Downloading Packages: 2026-02-23T11:10:31.081 INFO:teuthology.orchestra.run.trial191.stdout:rbd-mirror-20.3.0-5439.ge371433f.el9.x86_64.rpm 1.7 MB/s | 2.6 MB 00:01 2026-02-23T11:10:31.081 INFO:teuthology.orchestra.run.trial191.stdout:-------------------------------------------------------------------------------- 2026-02-23T11:10:31.081 INFO:teuthology.orchestra.run.trial191.stdout:Total 1.7 MB/s | 2.6 MB 00:01 2026-02-23T11:10:31.081 INFO:teuthology.orchestra.run.trial191.stdout:Running transaction check 2026-02-23T11:10:31.086 INFO:teuthology.orchestra.run.trial191.stdout:Transaction check succeeded. 2026-02-23T11:10:31.086 INFO:teuthology.orchestra.run.trial191.stdout:Running transaction test 2026-02-23T11:10:31.123 INFO:teuthology.orchestra.run.trial191.stdout:Transaction test succeeded. 2026-02-23T11:10:31.123 INFO:teuthology.orchestra.run.trial191.stdout:Running transaction 2026-02-23T11:10:31.293 INFO:teuthology.orchestra.run.trial191.stdout: Preparing : 1/1 2026-02-23T11:10:31.295 INFO:teuthology.orchestra.run.trial191.stdout: Installing : rbd-mirror-2:20.3.0-5439.ge371433f.el9.x86_64 1/1 2026-02-23T11:10:31.317 INFO:teuthology.orchestra.run.trial191.stdout: Running scriptlet: rbd-mirror-2:20.3.0-5439.ge371433f.el9.x86_64 1/1 2026-02-23T11:10:31.318 INFO:teuthology.orchestra.run.trial191.stdout:Glob pattern passed to enable, but globs are not supported for this. 2026-02-23T11:10:31.318 INFO:teuthology.orchestra.run.trial191.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2026-02-23T11:10:31.318 INFO:teuthology.orchestra.run.trial191.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2026-02-23T11:10:31.318 INFO:teuthology.orchestra.run.trial191.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2026-02-23T11:10:31.318 INFO:teuthology.orchestra.run.trial191.stdout: 2026-02-23T11:10:31.634 INFO:teuthology.orchestra.run.trial191.stdout: Verifying : rbd-mirror-2:20.3.0-5439.ge371433f.el9.x86_64 1/1 2026-02-23T11:10:31.634 INFO:teuthology.orchestra.run.trial191.stdout: 2026-02-23T11:10:31.634 INFO:teuthology.orchestra.run.trial191.stdout:Installed: 2026-02-23T11:10:31.634 INFO:teuthology.orchestra.run.trial191.stdout: rbd-mirror-2:20.3.0-5439.ge371433f.el9.x86_64 2026-02-23T11:10:31.634 INFO:teuthology.orchestra.run.trial191.stdout: 2026-02-23T11:10:31.634 INFO:teuthology.orchestra.run.trial191.stdout:Complete! 2026-02-23T11:10:31.661 DEBUG:teuthology.orchestra.run.trial191:> sudo yum -y install rbd-nbd 2026-02-23T11:10:31.976 INFO:teuthology.orchestra.run.trial191.stdout:Last metadata expiration check: 0:01:27 ago on Mon 23 Feb 2026 11:09:04 AM UTC. 2026-02-23T11:10:32.061 INFO:teuthology.orchestra.run.trial191.stdout:Dependencies resolved. 2026-02-23T11:10:32.061 INFO:teuthology.orchestra.run.trial191.stdout:================================================================================ 2026-02-23T11:10:32.061 INFO:teuthology.orchestra.run.trial191.stdout: Package Architecture Version Repository Size 2026-02-23T11:10:32.061 INFO:teuthology.orchestra.run.trial191.stdout:================================================================================ 2026-02-23T11:10:32.061 INFO:teuthology.orchestra.run.trial191.stdout:Installing: 2026-02-23T11:10:32.061 INFO:teuthology.orchestra.run.trial191.stdout: rbd-nbd x86_64 2:20.3.0-5439.ge371433f.el9 ceph 166 k 2026-02-23T11:10:32.061 INFO:teuthology.orchestra.run.trial191.stdout: 2026-02-23T11:10:32.062 INFO:teuthology.orchestra.run.trial191.stdout:Transaction Summary 2026-02-23T11:10:32.062 INFO:teuthology.orchestra.run.trial191.stdout:================================================================================ 2026-02-23T11:10:32.062 INFO:teuthology.orchestra.run.trial191.stdout:Install 1 Package 2026-02-23T11:10:32.062 INFO:teuthology.orchestra.run.trial191.stdout: 2026-02-23T11:10:32.062 INFO:teuthology.orchestra.run.trial191.stdout:Total download size: 166 k 2026-02-23T11:10:32.062 INFO:teuthology.orchestra.run.trial191.stdout:Installed size: 496 k 2026-02-23T11:10:32.062 INFO:teuthology.orchestra.run.trial191.stdout:Downloading Packages: 2026-02-23T11:10:32.505 INFO:teuthology.orchestra.run.trial191.stdout:rbd-nbd-20.3.0-5439.ge371433f.el9.x86_64.rpm 375 kB/s | 166 kB 00:00 2026-02-23T11:10:32.505 INFO:teuthology.orchestra.run.trial191.stdout:-------------------------------------------------------------------------------- 2026-02-23T11:10:32.505 INFO:teuthology.orchestra.run.trial191.stdout:Total 374 kB/s | 166 kB 00:00 2026-02-23T11:10:32.505 INFO:teuthology.orchestra.run.trial191.stdout:Running transaction check 2026-02-23T11:10:32.510 INFO:teuthology.orchestra.run.trial191.stdout:Transaction check succeeded. 2026-02-23T11:10:32.511 INFO:teuthology.orchestra.run.trial191.stdout:Running transaction test 2026-02-23T11:10:32.542 INFO:teuthology.orchestra.run.trial191.stdout:Transaction test succeeded. 2026-02-23T11:10:32.542 INFO:teuthology.orchestra.run.trial191.stdout:Running transaction 2026-02-23T11:10:32.598 INFO:teuthology.orchestra.run.trial191.stdout: Preparing : 1/1 2026-02-23T11:10:32.614 INFO:teuthology.orchestra.run.trial191.stdout: Installing : rbd-nbd-2:20.3.0-5439.ge371433f.el9.x86_64 1/1 2026-02-23T11:10:32.695 INFO:teuthology.orchestra.run.trial191.stdout: Running scriptlet: rbd-nbd-2:20.3.0-5439.ge371433f.el9.x86_64 1/1 2026-02-23T11:10:32.755 INFO:teuthology.orchestra.run.trial191.stdout: Verifying : rbd-nbd-2:20.3.0-5439.ge371433f.el9.x86_64 1/1 2026-02-23T11:10:32.755 INFO:teuthology.orchestra.run.trial191.stdout: 2026-02-23T11:10:32.755 INFO:teuthology.orchestra.run.trial191.stdout:Installed: 2026-02-23T11:10:32.755 INFO:teuthology.orchestra.run.trial191.stdout: rbd-nbd-2:20.3.0-5439.ge371433f.el9.x86_64 2026-02-23T11:10:32.755 INFO:teuthology.orchestra.run.trial191.stdout: 2026-02-23T11:10:32.755 INFO:teuthology.orchestra.run.trial191.stdout:Complete! 2026-02-23T11:10:32.784 DEBUG:teuthology.orchestra.run.trial191:> sudo yum -y install ceph-osd-crimson 2026-02-23T11:10:33.088 INFO:teuthology.orchestra.run.trial191.stdout:Last metadata expiration check: 0:01:29 ago on Mon 23 Feb 2026 11:09:04 AM UTC. 2026-02-23T11:10:33.153 INFO:teuthology.orchestra.run.trial191.stdout:Package ceph-osd-crimson-2:20.3.0-5439.ge371433f.el9.x86_64 is already installed. 2026-02-23T11:10:33.171 INFO:teuthology.orchestra.run.trial191.stdout:Dependencies resolved. 2026-02-23T11:10:33.172 INFO:teuthology.orchestra.run.trial191.stdout:Nothing to do. 2026-02-23T11:10:33.172 INFO:teuthology.orchestra.run.trial191.stdout:Complete! 2026-02-23T11:10:33.193 DEBUG:teuthology.orchestra.run.trial191:> sudo yum -y install ceph-osd-crimson-debuginfo 2026-02-23T11:10:33.502 INFO:teuthology.orchestra.run.trial191.stdout:Last metadata expiration check: 0:01:29 ago on Mon 23 Feb 2026 11:09:04 AM UTC. 2026-02-23T11:10:33.585 INFO:teuthology.orchestra.run.trial191.stdout:Dependencies resolved. 2026-02-23T11:10:33.586 INFO:teuthology.orchestra.run.trial191.stdout:================================================================================ 2026-02-23T11:10:33.586 INFO:teuthology.orchestra.run.trial191.stdout: Package Arch Version Repo Size 2026-02-23T11:10:33.586 INFO:teuthology.orchestra.run.trial191.stdout:================================================================================ 2026-02-23T11:10:33.586 INFO:teuthology.orchestra.run.trial191.stdout:Installing: 2026-02-23T11:10:33.586 INFO:teuthology.orchestra.run.trial191.stdout: ceph-osd-crimson-debuginfo x86_64 2:20.3.0-5439.ge371433f.el9 ceph 429 M 2026-02-23T11:10:33.586 INFO:teuthology.orchestra.run.trial191.stdout: 2026-02-23T11:10:33.586 INFO:teuthology.orchestra.run.trial191.stdout:Transaction Summary 2026-02-23T11:10:33.586 INFO:teuthology.orchestra.run.trial191.stdout:================================================================================ 2026-02-23T11:10:33.586 INFO:teuthology.orchestra.run.trial191.stdout:Install 1 Package 2026-02-23T11:10:33.587 INFO:teuthology.orchestra.run.trial191.stdout: 2026-02-23T11:10:33.587 INFO:teuthology.orchestra.run.trial191.stdout:Total download size: 429 M 2026-02-23T11:10:33.587 INFO:teuthology.orchestra.run.trial191.stdout:Installed size: 2.6 G 2026-02-23T11:10:33.587 INFO:teuthology.orchestra.run.trial191.stdout:Downloading Packages: 2026-02-23T11:13:07.273 INFO:teuthology.orchestra.run.trial191.stdout:ceph-osd-crimson-debuginfo-20.3.0-5439.ge371433 2.8 MB/s | 429 MB 02:33 2026-02-23T11:13:07.274 INFO:teuthology.orchestra.run.trial191.stdout:-------------------------------------------------------------------------------- 2026-02-23T11:13:07.274 INFO:teuthology.orchestra.run.trial191.stdout:Total 2.8 MB/s | 429 MB 02:33 2026-02-23T11:13:07.274 INFO:teuthology.orchestra.run.trial191.stdout:Running transaction check 2026-02-23T11:13:07.276 INFO:teuthology.orchestra.run.trial191.stdout:Transaction check succeeded. 2026-02-23T11:13:07.276 INFO:teuthology.orchestra.run.trial191.stdout:Running transaction test 2026-02-23T11:13:08.558 INFO:teuthology.orchestra.run.trial191.stdout:Transaction test succeeded. 2026-02-23T11:13:08.559 INFO:teuthology.orchestra.run.trial191.stdout:Running transaction 2026-02-23T11:13:31.985 INFO:teuthology.orchestra.run.trial191.stdout: Preparing : 1/1 2026-02-23T11:13:32.006 INFO:teuthology.orchestra.run.trial191.stdout: Installing : ceph-osd-crimson-debuginfo-2:20.3.0-5439.ge371433f.e 1/1 2026-02-23T11:13:33.820 INFO:teuthology.orchestra.run.trial191.stdout: Running scriptlet: ceph-osd-crimson-debuginfo-2:20.3.0-5439.ge371433f.e 1/1 2026-02-23T11:13:33.877 INFO:teuthology.orchestra.run.trial191.stdout: Verifying : ceph-osd-crimson-debuginfo-2:20.3.0-5439.ge371433f.e 1/1 2026-02-23T11:13:33.878 INFO:teuthology.orchestra.run.trial191.stdout: 2026-02-23T11:13:33.878 INFO:teuthology.orchestra.run.trial191.stdout:Installed: 2026-02-23T11:13:33.878 INFO:teuthology.orchestra.run.trial191.stdout: ceph-osd-crimson-debuginfo-2:20.3.0-5439.ge371433f.el9.x86_64 2026-02-23T11:13:33.878 INFO:teuthology.orchestra.run.trial191.stdout: 2026-02-23T11:13:33.878 INFO:teuthology.orchestra.run.trial191.stdout:Complete! 2026-02-23T11:13:33.944 DEBUG:teuthology.parallel:result is None 2026-02-23T11:13:33.945 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=debug&distros=centos%2F9%2Fx86_64&sha1=e371433fcd1390a0b82cfb18effc422c1f5d8c1d 2026-02-23T11:13:34.118 DEBUG:teuthology.orchestra.run.trial191:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2026-02-23T11:13:34.139 INFO:teuthology.orchestra.run.trial191.stdout:20.3.0-5439.ge371433f.el9 2026-02-23T11:13:34.139 INFO:teuthology.packaging:The installed version of ceph is 20.3.0-5439.ge371433f.el9 2026-02-23T11:13:34.139 INFO:teuthology.task.install:The correct ceph version 20.3.0-5439.ge371433f is installed. 2026-02-23T11:13:34.142 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2026-02-23T11:13:34.142 DEBUG:teuthology.orchestra.run.trial191:> set -ex 2026-02-23T11:13:34.142 DEBUG:teuthology.orchestra.run.trial191:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2026-02-23T11:13:34.205 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2026-02-23T11:13:34.205 DEBUG:teuthology.orchestra.run.trial191:> set -ex 2026-02-23T11:13:34.205 DEBUG:teuthology.orchestra.run.trial191:> sudo dd of=/usr/bin/daemon-helper 2026-02-23T11:13:34.268 DEBUG:teuthology.orchestra.run.trial191:> sudo chmod a=rx -- /usr/bin/daemon-helper 2026-02-23T11:13:34.334 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2026-02-23T11:13:34.335 DEBUG:teuthology.orchestra.run.trial191:> set -ex 2026-02-23T11:13:34.335 DEBUG:teuthology.orchestra.run.trial191:> sudo dd of=/usr/bin/adjust-ulimits 2026-02-23T11:13:34.398 DEBUG:teuthology.orchestra.run.trial191:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2026-02-23T11:13:34.461 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2026-02-23T11:13:34.462 DEBUG:teuthology.orchestra.run.trial191:> set -ex 2026-02-23T11:13:34.462 DEBUG:teuthology.orchestra.run.trial191:> sudo dd of=/usr/bin/stdin-killer 2026-02-23T11:13:34.525 DEBUG:teuthology.orchestra.run.trial191:> sudo chmod a=rx -- /usr/bin/stdin-killer 2026-02-23T11:13:34.588 INFO:teuthology.run_tasks:Running task pexec... 2026-02-23T11:13:34.607 INFO:teuthology.task.pexec:Executing custom commands... 2026-02-23T11:13:34.608 DEBUG:teuthology.orchestra.run.trial191:> TESTDIR=/home/ubuntu/cephtest bash -s 2026-02-23T11:13:34.631 INFO:teuthology.task.pexec:Running commands on host ubuntu@trial191.front.sepia.ceph.com 2026-02-23T11:13:34.631 INFO:teuthology.task.pexec:sudo update-alternatives --set ceph-osd /usr/bin/ceph-osd-crimson 2026-02-23T11:13:34.683 DEBUG:teuthology.parallel:result is None 2026-02-23T11:13:34.683 INFO:teuthology.run_tasks:Running task install... 2026-02-23T11:13:34.698 DEBUG:teuthology.task.install:project ceph 2026-02-23T11:13:34.698 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'debug', 'sha1': 'e371433fcd1390a0b82cfb18effc422c1f5d8c1d'}} 2026-02-23T11:13:34.698 DEBUG:teuthology.task.install:config {'flavor': 'debug', 'sha1': 'e371433fcd1390a0b82cfb18effc422c1f5d8c1d'} 2026-02-23T11:13:34.698 INFO:teuthology.task.install:Using flavor: debug 2026-02-23T11:13:34.708 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-osd', 'ceph-osd-classic', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-osd', 'ceph-osd-classic', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2026-02-23T11:13:34.708 INFO:teuthology.task.install:extra packages: [] 2026-02-23T11:13:34.708 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'debug', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': 'e371433fcd1390a0b82cfb18effc422c1f5d8c1d', 'tag': None, 'wait_for_package': False} 2026-02-23T11:13:34.709 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=debug&distros=centos%2F9%2Fx86_64&sha1=e371433fcd1390a0b82cfb18effc422c1f5d8c1d 2026-02-23T11:13:34.867 INFO:teuthology.task.install.rpm:Pulling from https://3.chacra.ceph.com/r/ceph/wip-shraddhaag-enable-cephadm-tests/e371433fcd1390a0b82cfb18effc422c1f5d8c1d/centos/9/flavors/debug/ 2026-02-23T11:13:34.867 INFO:teuthology.task.install.rpm:Package version is 20.3.0-5439.ge371433f 2026-02-23T11:13:35.252 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://3.chacra.ceph.com/r/ceph/wip-shraddhaag-enable-cephadm-tests/e371433fcd1390a0b82cfb18effc422c1f5d8c1d/centos/9/flavors/debug/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://3.chacra.ceph.com/r/ceph/wip-shraddhaag-enable-cephadm-tests/e371433fcd1390a0b82cfb18effc422c1f5d8c1d/centos/9/flavors/debug/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://3.chacra.ceph.com/r/ceph/wip-shraddhaag-enable-cephadm-tests/e371433fcd1390a0b82cfb18effc422c1f5d8c1d/centos/9/flavors/debug/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2026-02-23T11:13:35.252 DEBUG:teuthology.orchestra.run.trial191:> set -ex 2026-02-23T11:13:35.252 DEBUG:teuthology.orchestra.run.trial191:> sudo dd of=/etc/yum.repos.d/ceph.repo 2026-02-23T11:13:35.278 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-osd, ceph-osd-classic, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2026-02-23T11:13:35.278 DEBUG:teuthology.orchestra.run.trial191:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/e371433fcd1390a0b82cfb18effc422c1f5d8c1d/;g' /etc/yum.repos.d/ceph.repo ; fi 2026-02-23T11:13:35.342 DEBUG:teuthology.orchestra.run.trial191:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2026-02-23T11:13:35.408 DEBUG:teuthology.orchestra.run.trial191:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2026-02-23T11:13:35.463 INFO:teuthology.orchestra.run.trial191.stdout:check_obsoletes = 1 2026-02-23T11:13:35.473 DEBUG:teuthology.orchestra.run.trial191:> sudo yum clean all 2026-02-23T11:13:35.634 INFO:teuthology.orchestra.run.trial191.stdout:56 files removed 2026-02-23T11:13:35.650 DEBUG:teuthology.orchestra.run.trial191:> sudo yum -y install ceph-radosgw 2026-02-23T11:13:36.576 INFO:teuthology.orchestra.run.trial191.stdout:ceph packages for x86_64 118 kB/s | 92 kB 00:00 2026-02-23T11:13:37.202 INFO:teuthology.orchestra.run.trial191.stdout:ceph noarch packages 39 kB/s | 24 kB 00:00 2026-02-23T11:13:37.797 INFO:teuthology.orchestra.run.trial191.stdout:ceph source packages 3.6 kB/s | 2.1 kB 00:00 2026-02-23T11:13:38.056 INFO:teuthology.orchestra.run.trial191.stdout:CentOS Stream 9 - BaseOS 37 MB/s | 8.9 MB 00:00 2026-02-23T11:13:41.630 INFO:teuthology.orchestra.run.trial191.stdout:CentOS Stream 9 - AppStream 8.8 MB/s | 27 MB 00:03 2026-02-23T11:13:45.153 INFO:teuthology.orchestra.run.trial191.stdout:CentOS Stream 9 - CRB 6.9 MB/s | 7.8 MB 00:01 2026-02-23T11:13:47.164 INFO:teuthology.orchestra.run.trial191.stdout:CentOS Stream 9 - Extras packages 16 kB/s | 20 kB 00:01 2026-02-23T11:13:48.100 INFO:teuthology.orchestra.run.trial191.stdout:Extra Packages for Enterprise Linux 23 MB/s | 20 MB 00:00 2026-02-23T11:13:51.360 INFO:teuthology.orchestra.run.trial191.stdout:lab-extras 1.1 MB/s | 50 kB 00:00 2026-02-23T11:13:52.544 INFO:teuthology.orchestra.run.trial191.stdout:Package ceph-radosgw-2:20.3.0-5439.ge371433f.el9.x86_64 is already installed. 2026-02-23T11:13:52.563 INFO:teuthology.orchestra.run.trial191.stdout:Dependencies resolved. 2026-02-23T11:13:52.563 INFO:teuthology.orchestra.run.trial191.stdout:Nothing to do. 2026-02-23T11:13:52.563 INFO:teuthology.orchestra.run.trial191.stdout:Complete! 2026-02-23T11:13:52.589 DEBUG:teuthology.orchestra.run.trial191:> sudo yum -y install ceph-test 2026-02-23T11:13:52.900 INFO:teuthology.orchestra.run.trial191.stdout:Last metadata expiration check: 0:00:01 ago on Mon 23 Feb 2026 11:13:51 AM UTC. 2026-02-23T11:13:52.965 INFO:teuthology.orchestra.run.trial191.stdout:Package ceph-test-2:20.3.0-5439.ge371433f.el9.x86_64 is already installed. 2026-02-23T11:13:52.983 INFO:teuthology.orchestra.run.trial191.stdout:Dependencies resolved. 2026-02-23T11:13:52.984 INFO:teuthology.orchestra.run.trial191.stdout:Nothing to do. 2026-02-23T11:13:52.984 INFO:teuthology.orchestra.run.trial191.stdout:Complete! 2026-02-23T11:13:53.005 DEBUG:teuthology.orchestra.run.trial191:> sudo yum -y install ceph 2026-02-23T11:13:53.318 INFO:teuthology.orchestra.run.trial191.stdout:Last metadata expiration check: 0:00:02 ago on Mon 23 Feb 2026 11:13:51 AM UTC. 2026-02-23T11:13:53.382 INFO:teuthology.orchestra.run.trial191.stdout:Package ceph-2:20.3.0-5439.ge371433f.el9.x86_64 is already installed. 2026-02-23T11:13:53.401 INFO:teuthology.orchestra.run.trial191.stdout:Dependencies resolved. 2026-02-23T11:13:53.402 INFO:teuthology.orchestra.run.trial191.stdout:Nothing to do. 2026-02-23T11:13:53.402 INFO:teuthology.orchestra.run.trial191.stdout:Complete! 2026-02-23T11:13:53.423 DEBUG:teuthology.orchestra.run.trial191:> sudo yum -y install ceph-base 2026-02-23T11:13:53.732 INFO:teuthology.orchestra.run.trial191.stdout:Last metadata expiration check: 0:00:02 ago on Mon 23 Feb 2026 11:13:51 AM UTC. 2026-02-23T11:13:53.798 INFO:teuthology.orchestra.run.trial191.stdout:Package ceph-base-2:20.3.0-5439.ge371433f.el9.x86_64 is already installed. 2026-02-23T11:13:53.816 INFO:teuthology.orchestra.run.trial191.stdout:Dependencies resolved. 2026-02-23T11:13:53.817 INFO:teuthology.orchestra.run.trial191.stdout:Nothing to do. 2026-02-23T11:13:53.817 INFO:teuthology.orchestra.run.trial191.stdout:Complete! 2026-02-23T11:13:53.839 DEBUG:teuthology.orchestra.run.trial191:> sudo yum -y install cephadm 2026-02-23T11:13:54.148 INFO:teuthology.orchestra.run.trial191.stdout:Last metadata expiration check: 0:00:03 ago on Mon 23 Feb 2026 11:13:51 AM UTC. 2026-02-23T11:13:54.213 INFO:teuthology.orchestra.run.trial191.stdout:Package cephadm-2:20.3.0-5439.ge371433f.el9.noarch is already installed. 2026-02-23T11:13:54.232 INFO:teuthology.orchestra.run.trial191.stdout:Dependencies resolved. 2026-02-23T11:13:54.233 INFO:teuthology.orchestra.run.trial191.stdout:Nothing to do. 2026-02-23T11:13:54.233 INFO:teuthology.orchestra.run.trial191.stdout:Complete! 2026-02-23T11:13:54.254 DEBUG:teuthology.orchestra.run.trial191:> sudo yum -y install ceph-immutable-object-cache 2026-02-23T11:13:54.568 INFO:teuthology.orchestra.run.trial191.stdout:Last metadata expiration check: 0:00:03 ago on Mon 23 Feb 2026 11:13:51 AM UTC. 2026-02-23T11:13:54.632 INFO:teuthology.orchestra.run.trial191.stdout:Package ceph-immutable-object-cache-2:20.3.0-5439.ge371433f.el9.x86_64 is already installed. 2026-02-23T11:13:54.651 INFO:teuthology.orchestra.run.trial191.stdout:Dependencies resolved. 2026-02-23T11:13:54.651 INFO:teuthology.orchestra.run.trial191.stdout:Nothing to do. 2026-02-23T11:13:54.652 INFO:teuthology.orchestra.run.trial191.stdout:Complete! 2026-02-23T11:13:54.674 DEBUG:teuthology.orchestra.run.trial191:> sudo yum -y install ceph-mgr 2026-02-23T11:13:54.983 INFO:teuthology.orchestra.run.trial191.stdout:Last metadata expiration check: 0:00:03 ago on Mon 23 Feb 2026 11:13:51 AM UTC. 2026-02-23T11:13:55.048 INFO:teuthology.orchestra.run.trial191.stdout:Package ceph-mgr-2:20.3.0-5439.ge371433f.el9.x86_64 is already installed. 2026-02-23T11:13:55.066 INFO:teuthology.orchestra.run.trial191.stdout:Dependencies resolved. 2026-02-23T11:13:55.067 INFO:teuthology.orchestra.run.trial191.stdout:Nothing to do. 2026-02-23T11:13:55.067 INFO:teuthology.orchestra.run.trial191.stdout:Complete! 2026-02-23T11:13:55.089 DEBUG:teuthology.orchestra.run.trial191:> sudo yum -y install ceph-mgr-dashboard 2026-02-23T11:13:55.398 INFO:teuthology.orchestra.run.trial191.stdout:Last metadata expiration check: 0:00:04 ago on Mon 23 Feb 2026 11:13:51 AM UTC. 2026-02-23T11:13:55.462 INFO:teuthology.orchestra.run.trial191.stdout:Package ceph-mgr-dashboard-2:20.3.0-5439.ge371433f.el9.noarch is already installed. 2026-02-23T11:13:55.480 INFO:teuthology.orchestra.run.trial191.stdout:Dependencies resolved. 2026-02-23T11:13:55.481 INFO:teuthology.orchestra.run.trial191.stdout:Nothing to do. 2026-02-23T11:13:55.481 INFO:teuthology.orchestra.run.trial191.stdout:Complete! 2026-02-23T11:13:55.502 DEBUG:teuthology.orchestra.run.trial191:> sudo yum -y install ceph-mgr-diskprediction-local 2026-02-23T11:13:55.816 INFO:teuthology.orchestra.run.trial191.stdout:Last metadata expiration check: 0:00:04 ago on Mon 23 Feb 2026 11:13:51 AM UTC. 2026-02-23T11:13:55.880 INFO:teuthology.orchestra.run.trial191.stdout:Package ceph-mgr-diskprediction-local-2:20.3.0-5439.ge371433f.el9.noarch is already installed. 2026-02-23T11:13:55.899 INFO:teuthology.orchestra.run.trial191.stdout:Dependencies resolved. 2026-02-23T11:13:55.900 INFO:teuthology.orchestra.run.trial191.stdout:Nothing to do. 2026-02-23T11:13:55.900 INFO:teuthology.orchestra.run.trial191.stdout:Complete! 2026-02-23T11:13:55.921 DEBUG:teuthology.orchestra.run.trial191:> sudo yum -y install ceph-mgr-rook 2026-02-23T11:13:56.232 INFO:teuthology.orchestra.run.trial191.stdout:Last metadata expiration check: 0:00:05 ago on Mon 23 Feb 2026 11:13:51 AM UTC. 2026-02-23T11:13:56.297 INFO:teuthology.orchestra.run.trial191.stdout:Package ceph-mgr-rook-2:20.3.0-5439.ge371433f.el9.noarch is already installed. 2026-02-23T11:13:56.315 INFO:teuthology.orchestra.run.trial191.stdout:Dependencies resolved. 2026-02-23T11:13:56.316 INFO:teuthology.orchestra.run.trial191.stdout:Nothing to do. 2026-02-23T11:13:56.316 INFO:teuthology.orchestra.run.trial191.stdout:Complete! 2026-02-23T11:13:56.338 DEBUG:teuthology.orchestra.run.trial191:> sudo yum -y install ceph-mgr-cephadm 2026-02-23T11:13:56.651 INFO:teuthology.orchestra.run.trial191.stdout:Last metadata expiration check: 0:00:05 ago on Mon 23 Feb 2026 11:13:51 AM UTC. 2026-02-23T11:13:56.715 INFO:teuthology.orchestra.run.trial191.stdout:Package ceph-mgr-cephadm-2:20.3.0-5439.ge371433f.el9.noarch is already installed. 2026-02-23T11:13:56.733 INFO:teuthology.orchestra.run.trial191.stdout:Dependencies resolved. 2026-02-23T11:13:56.734 INFO:teuthology.orchestra.run.trial191.stdout:Nothing to do. 2026-02-23T11:13:56.734 INFO:teuthology.orchestra.run.trial191.stdout:Complete! 2026-02-23T11:13:56.756 DEBUG:teuthology.orchestra.run.trial191:> sudo yum -y install ceph-osd 2026-02-23T11:13:57.073 INFO:teuthology.orchestra.run.trial191.stdout:Last metadata expiration check: 0:00:06 ago on Mon 23 Feb 2026 11:13:51 AM UTC. 2026-02-23T11:13:57.138 INFO:teuthology.orchestra.run.trial191.stdout:Package ceph-osd-2:20.3.0-5439.ge371433f.el9.x86_64 is already installed. 2026-02-23T11:13:57.156 INFO:teuthology.orchestra.run.trial191.stdout:Dependencies resolved. 2026-02-23T11:13:57.157 INFO:teuthology.orchestra.run.trial191.stdout:Nothing to do. 2026-02-23T11:13:57.157 INFO:teuthology.orchestra.run.trial191.stdout:Complete! 2026-02-23T11:13:57.180 DEBUG:teuthology.orchestra.run.trial191:> sudo yum -y install ceph-osd-classic 2026-02-23T11:13:57.488 INFO:teuthology.orchestra.run.trial191.stdout:Last metadata expiration check: 0:00:06 ago on Mon 23 Feb 2026 11:13:51 AM UTC. 2026-02-23T11:13:57.553 INFO:teuthology.orchestra.run.trial191.stdout:Package ceph-osd-classic-2:20.3.0-5439.ge371433f.el9.x86_64 is already installed. 2026-02-23T11:13:57.571 INFO:teuthology.orchestra.run.trial191.stdout:Dependencies resolved. 2026-02-23T11:13:57.572 INFO:teuthology.orchestra.run.trial191.stdout:Nothing to do. 2026-02-23T11:13:57.572 INFO:teuthology.orchestra.run.trial191.stdout:Complete! 2026-02-23T11:13:57.593 DEBUG:teuthology.orchestra.run.trial191:> sudo yum -y install ceph-fuse 2026-02-23T11:13:57.906 INFO:teuthology.orchestra.run.trial191.stdout:Last metadata expiration check: 0:00:06 ago on Mon 23 Feb 2026 11:13:51 AM UTC. 2026-02-23T11:13:57.971 INFO:teuthology.orchestra.run.trial191.stdout:Package ceph-fuse-2:20.3.0-5439.ge371433f.el9.x86_64 is already installed. 2026-02-23T11:13:57.989 INFO:teuthology.orchestra.run.trial191.stdout:Dependencies resolved. 2026-02-23T11:13:57.990 INFO:teuthology.orchestra.run.trial191.stdout:Nothing to do. 2026-02-23T11:13:57.990 INFO:teuthology.orchestra.run.trial191.stdout:Complete! 2026-02-23T11:13:58.012 DEBUG:teuthology.orchestra.run.trial191:> sudo yum -y install ceph-volume 2026-02-23T11:13:58.323 INFO:teuthology.orchestra.run.trial191.stdout:Last metadata expiration check: 0:00:07 ago on Mon 23 Feb 2026 11:13:51 AM UTC. 2026-02-23T11:13:58.387 INFO:teuthology.orchestra.run.trial191.stdout:Package ceph-volume-2:20.3.0-5439.ge371433f.el9.noarch is already installed. 2026-02-23T11:13:58.406 INFO:teuthology.orchestra.run.trial191.stdout:Dependencies resolved. 2026-02-23T11:13:58.406 INFO:teuthology.orchestra.run.trial191.stdout:Nothing to do. 2026-02-23T11:13:58.406 INFO:teuthology.orchestra.run.trial191.stdout:Complete! 2026-02-23T11:13:58.428 DEBUG:teuthology.orchestra.run.trial191:> sudo yum -y install librados-devel 2026-02-23T11:13:58.736 INFO:teuthology.orchestra.run.trial191.stdout:Last metadata expiration check: 0:00:07 ago on Mon 23 Feb 2026 11:13:51 AM UTC. 2026-02-23T11:13:58.800 INFO:teuthology.orchestra.run.trial191.stdout:Package librados-devel-2:20.3.0-5439.ge371433f.el9.x86_64 is already installed. 2026-02-23T11:13:58.818 INFO:teuthology.orchestra.run.trial191.stdout:Dependencies resolved. 2026-02-23T11:13:58.819 INFO:teuthology.orchestra.run.trial191.stdout:Nothing to do. 2026-02-23T11:13:58.819 INFO:teuthology.orchestra.run.trial191.stdout:Complete! 2026-02-23T11:13:58.840 DEBUG:teuthology.orchestra.run.trial191:> sudo yum -y install libcephfs2 2026-02-23T11:13:59.151 INFO:teuthology.orchestra.run.trial191.stdout:Last metadata expiration check: 0:00:08 ago on Mon 23 Feb 2026 11:13:51 AM UTC. 2026-02-23T11:13:59.215 INFO:teuthology.orchestra.run.trial191.stdout:Package libcephfs2-2:20.3.0-5439.ge371433f.el9.x86_64 is already installed. 2026-02-23T11:13:59.233 INFO:teuthology.orchestra.run.trial191.stdout:Dependencies resolved. 2026-02-23T11:13:59.234 INFO:teuthology.orchestra.run.trial191.stdout:Nothing to do. 2026-02-23T11:13:59.234 INFO:teuthology.orchestra.run.trial191.stdout:Complete! 2026-02-23T11:13:59.256 DEBUG:teuthology.orchestra.run.trial191:> sudo yum -y install libcephfs-devel 2026-02-23T11:13:59.567 INFO:teuthology.orchestra.run.trial191.stdout:Last metadata expiration check: 0:00:08 ago on Mon 23 Feb 2026 11:13:51 AM UTC. 2026-02-23T11:13:59.632 INFO:teuthology.orchestra.run.trial191.stdout:Package libcephfs-devel-2:20.3.0-5439.ge371433f.el9.x86_64 is already installed. 2026-02-23T11:13:59.650 INFO:teuthology.orchestra.run.trial191.stdout:Dependencies resolved. 2026-02-23T11:13:59.651 INFO:teuthology.orchestra.run.trial191.stdout:Nothing to do. 2026-02-23T11:13:59.651 INFO:teuthology.orchestra.run.trial191.stdout:Complete! 2026-02-23T11:13:59.673 DEBUG:teuthology.orchestra.run.trial191:> sudo yum -y install librados2 2026-02-23T11:13:59.986 INFO:teuthology.orchestra.run.trial191.stdout:Last metadata expiration check: 0:00:08 ago on Mon 23 Feb 2026 11:13:51 AM UTC. 2026-02-23T11:14:00.051 INFO:teuthology.orchestra.run.trial191.stdout:Package librados2-2:20.3.0-5439.ge371433f.el9.x86_64 is already installed. 2026-02-23T11:14:00.070 INFO:teuthology.orchestra.run.trial191.stdout:Dependencies resolved. 2026-02-23T11:14:00.070 INFO:teuthology.orchestra.run.trial191.stdout:Nothing to do. 2026-02-23T11:14:00.071 INFO:teuthology.orchestra.run.trial191.stdout:Complete! 2026-02-23T11:14:00.093 DEBUG:teuthology.orchestra.run.trial191:> sudo yum -y install librbd1 2026-02-23T11:14:00.403 INFO:teuthology.orchestra.run.trial191.stdout:Last metadata expiration check: 0:00:09 ago on Mon 23 Feb 2026 11:13:51 AM UTC. 2026-02-23T11:14:00.467 INFO:teuthology.orchestra.run.trial191.stdout:Package librbd1-2:20.3.0-5439.ge371433f.el9.x86_64 is already installed. 2026-02-23T11:14:00.486 INFO:teuthology.orchestra.run.trial191.stdout:Dependencies resolved. 2026-02-23T11:14:00.486 INFO:teuthology.orchestra.run.trial191.stdout:Nothing to do. 2026-02-23T11:14:00.487 INFO:teuthology.orchestra.run.trial191.stdout:Complete! 2026-02-23T11:14:00.508 DEBUG:teuthology.orchestra.run.trial191:> sudo yum -y install python3-rados 2026-02-23T11:14:00.816 INFO:teuthology.orchestra.run.trial191.stdout:Last metadata expiration check: 0:00:09 ago on Mon 23 Feb 2026 11:13:51 AM UTC. 2026-02-23T11:14:00.880 INFO:teuthology.orchestra.run.trial191.stdout:Package python3-rados-2:20.3.0-5439.ge371433f.el9.x86_64 is already installed. 2026-02-23T11:14:00.899 INFO:teuthology.orchestra.run.trial191.stdout:Dependencies resolved. 2026-02-23T11:14:00.899 INFO:teuthology.orchestra.run.trial191.stdout:Nothing to do. 2026-02-23T11:14:00.899 INFO:teuthology.orchestra.run.trial191.stdout:Complete! 2026-02-23T11:14:00.921 DEBUG:teuthology.orchestra.run.trial191:> sudo yum -y install python3-rgw 2026-02-23T11:14:01.232 INFO:teuthology.orchestra.run.trial191.stdout:Last metadata expiration check: 0:00:10 ago on Mon 23 Feb 2026 11:13:51 AM UTC. 2026-02-23T11:14:01.296 INFO:teuthology.orchestra.run.trial191.stdout:Package python3-rgw-2:20.3.0-5439.ge371433f.el9.x86_64 is already installed. 2026-02-23T11:14:01.314 INFO:teuthology.orchestra.run.trial191.stdout:Dependencies resolved. 2026-02-23T11:14:01.315 INFO:teuthology.orchestra.run.trial191.stdout:Nothing to do. 2026-02-23T11:14:01.315 INFO:teuthology.orchestra.run.trial191.stdout:Complete! 2026-02-23T11:14:01.337 DEBUG:teuthology.orchestra.run.trial191:> sudo yum -y install python3-cephfs 2026-02-23T11:14:01.645 INFO:teuthology.orchestra.run.trial191.stdout:Last metadata expiration check: 0:00:10 ago on Mon 23 Feb 2026 11:13:51 AM UTC. 2026-02-23T11:14:01.709 INFO:teuthology.orchestra.run.trial191.stdout:Package python3-cephfs-2:20.3.0-5439.ge371433f.el9.x86_64 is already installed. 2026-02-23T11:14:01.727 INFO:teuthology.orchestra.run.trial191.stdout:Dependencies resolved. 2026-02-23T11:14:01.728 INFO:teuthology.orchestra.run.trial191.stdout:Nothing to do. 2026-02-23T11:14:01.728 INFO:teuthology.orchestra.run.trial191.stdout:Complete! 2026-02-23T11:14:01.749 DEBUG:teuthology.orchestra.run.trial191:> sudo yum -y install python3-rbd 2026-02-23T11:14:02.063 INFO:teuthology.orchestra.run.trial191.stdout:Last metadata expiration check: 0:00:11 ago on Mon 23 Feb 2026 11:13:51 AM UTC. 2026-02-23T11:14:02.127 INFO:teuthology.orchestra.run.trial191.stdout:Package python3-rbd-2:20.3.0-5439.ge371433f.el9.x86_64 is already installed. 2026-02-23T11:14:02.146 INFO:teuthology.orchestra.run.trial191.stdout:Dependencies resolved. 2026-02-23T11:14:02.146 INFO:teuthology.orchestra.run.trial191.stdout:Nothing to do. 2026-02-23T11:14:02.146 INFO:teuthology.orchestra.run.trial191.stdout:Complete! 2026-02-23T11:14:02.168 DEBUG:teuthology.orchestra.run.trial191:> sudo yum -y install rbd-fuse 2026-02-23T11:14:02.476 INFO:teuthology.orchestra.run.trial191.stdout:Last metadata expiration check: 0:00:11 ago on Mon 23 Feb 2026 11:13:51 AM UTC. 2026-02-23T11:14:02.541 INFO:teuthology.orchestra.run.trial191.stdout:Package rbd-fuse-2:20.3.0-5439.ge371433f.el9.x86_64 is already installed. 2026-02-23T11:14:02.559 INFO:teuthology.orchestra.run.trial191.stdout:Dependencies resolved. 2026-02-23T11:14:02.560 INFO:teuthology.orchestra.run.trial191.stdout:Nothing to do. 2026-02-23T11:14:02.560 INFO:teuthology.orchestra.run.trial191.stdout:Complete! 2026-02-23T11:14:02.582 DEBUG:teuthology.orchestra.run.trial191:> sudo yum -y install rbd-mirror 2026-02-23T11:14:02.888 INFO:teuthology.orchestra.run.trial191.stdout:Last metadata expiration check: 0:00:11 ago on Mon 23 Feb 2026 11:13:51 AM UTC. 2026-02-23T11:14:02.952 INFO:teuthology.orchestra.run.trial191.stdout:Package rbd-mirror-2:20.3.0-5439.ge371433f.el9.x86_64 is already installed. 2026-02-23T11:14:02.971 INFO:teuthology.orchestra.run.trial191.stdout:Dependencies resolved. 2026-02-23T11:14:02.971 INFO:teuthology.orchestra.run.trial191.stdout:Nothing to do. 2026-02-23T11:14:02.971 INFO:teuthology.orchestra.run.trial191.stdout:Complete! 2026-02-23T11:14:02.993 DEBUG:teuthology.orchestra.run.trial191:> sudo yum -y install rbd-nbd 2026-02-23T11:14:03.302 INFO:teuthology.orchestra.run.trial191.stdout:Last metadata expiration check: 0:00:12 ago on Mon 23 Feb 2026 11:13:51 AM UTC. 2026-02-23T11:14:03.367 INFO:teuthology.orchestra.run.trial191.stdout:Package rbd-nbd-2:20.3.0-5439.ge371433f.el9.x86_64 is already installed. 2026-02-23T11:14:03.385 INFO:teuthology.orchestra.run.trial191.stdout:Dependencies resolved. 2026-02-23T11:14:03.386 INFO:teuthology.orchestra.run.trial191.stdout:Nothing to do. 2026-02-23T11:14:03.386 INFO:teuthology.orchestra.run.trial191.stdout:Complete! 2026-02-23T11:14:03.411 DEBUG:teuthology.parallel:result is None 2026-02-23T11:14:03.411 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=debug&distros=centos%2F9%2Fx86_64&sha1=e371433fcd1390a0b82cfb18effc422c1f5d8c1d 2026-02-23T11:14:03.532 DEBUG:teuthology.orchestra.run.trial191:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2026-02-23T11:14:03.553 INFO:teuthology.orchestra.run.trial191.stdout:20.3.0-5439.ge371433f.el9 2026-02-23T11:14:03.553 INFO:teuthology.packaging:The installed version of ceph is 20.3.0-5439.ge371433f.el9 2026-02-23T11:14:03.553 INFO:teuthology.task.install:The correct ceph version 20.3.0-5439.ge371433f is installed. 2026-02-23T11:14:03.554 INFO:teuthology.task.install.util:Utilities already shipped, skip it... 2026-02-23T11:14:03.554 INFO:teuthology.run_tasks:Running task ceph... 2026-02-23T11:14:03.629 INFO:tasks.ceph:Making ceph log dir writeable by non-root... 2026-02-23T11:14:03.629 DEBUG:teuthology.orchestra.run.trial191:> sudo chmod 777 /var/log/ceph 2026-02-23T11:14:03.653 INFO:tasks.ceph:Disabling ceph logrotate... 2026-02-23T11:14:03.653 DEBUG:teuthology.orchestra.run.trial191:> sudo rm -f -- /etc/logrotate.d/ceph 2026-02-23T11:14:03.716 INFO:tasks.ceph:Creating extra log directories... 2026-02-23T11:14:03.716 DEBUG:teuthology.orchestra.run.trial191:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2026-02-23T11:14:03.788 INFO:tasks.ceph:Creating ceph cluster ceph... 2026-02-23T11:14:03.788 INFO:tasks.ceph:config {'conf': {'global': {'enable experimental unrecoverable data corrupting features': 'crimson'}, 'osd': {'crimson cpu num': 2, 'osd min pg log entries': 5, 'bluestore zero block detection': True, 'crimson osd obc lru size': 10, 'debug ms': 20, 'debug osd': 20, 'debug seastore': 20, 'debug seastore backref': 20, 'debug seastore cache': 20, 'debug seastore cleaner': 20, 'debug seastore device': 20, 'debug seastore epm': 20, 'debug seastore fixedkv tree': 20, 'debug seastore journal': 20, 'debug seastore lba': 20, 'debug seastore odata': 20, 'debug seastore omap': 20, 'debug seastore onode': 20, 'debug seastore t': 20, 'debug seastore tm': 20, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd mclock override recovery settings': True, 'osd mclock profile': 'high_recovery_ops', 'osd mclock skip benchmark': True, 'osd objectstore': 'seastore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random', 'seastore cachepin size pershard': '64M', 'seastore cachepin type': '2Q', 'seastore max concurrent transactions': 8}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 30, 'debug ms': 1, 'debug paxos': 20, 'mon scrub interval': 300, 'osd pool default crimson': True}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['but it is still running', 'overall HEALTH_', '\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(POOL_APP_NOT_ENABLED\\)'], 'cpu_profile': set(), 'cluster': 'ceph', 'mon_bind_msgr2': True, 'mon_bind_addrvec': True} 2026-02-23T11:14:03.788 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/mnt/teuthology/shraddhaag-2026-02-23_09:58:48-crimson-rados-main-distro-debug-trial/66185', 'branch': 'main', 'description': 'crimson-rados/singleton/{0-crimson_install all/osd-dump-metrics crimson-supported-all-distro/centos_latest crimson_qa_overrides objectstore/seastore/segmented$/{crimson_seastore_segmented_2q} rados}', 'email': None, 'first_in_suite': False, 'flavor': 'debug', 'job_id': '66185', 'kernel': {'branch': 'distro', 'kdb': True, 'sha1': 'distro'}, 'last_in_suite': False, 'machine_type': 'trial', 'name': 'shraddhaag-2026-02-23_09:58:48-crimson-rados-main-distro-debug-trial', 'no_nested_subset': False, 'openstack': [{'volumes': {'count': 3, 'size': 10}}], 'os_type': 'centos', 'os_version': '9.stream', 'overrides': {'admin_socket': {'branch': 'main'}, 'ceph': {'conf': {'global': {'enable experimental unrecoverable data corrupting features': 'crimson'}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 30, 'debug ms': 1, 'debug paxos': 20, 'mon scrub interval': 300, 'osd pool default crimson': True}, 'osd': {'bluestore zero block detection': True, 'crimson osd obc lru size': 10, 'debug ms': 20, 'debug osd': 20, 'debug seastore': 20, 'debug seastore backref': 20, 'debug seastore cache': 20, 'debug seastore cleaner': 20, 'debug seastore device': 20, 'debug seastore epm': 20, 'debug seastore fixedkv tree': 20, 'debug seastore journal': 20, 'debug seastore lba': 20, 'debug seastore odata': 20, 'debug seastore omap': 20, 'debug seastore onode': 20, 'debug seastore t': 20, 'debug seastore tm': 20, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd mclock override recovery settings': True, 'osd mclock profile': 'high_recovery_ops', 'osd mclock skip benchmark': True, 'osd objectstore': 'seastore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random', 'seastore cachepin size pershard': '64M', 'seastore cachepin type': '2Q', 'seastore max concurrent transactions': 8}}, 'crimson_compat': True, 'flavor': 'debug', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(POOL_APP_NOT_ENABLED\\)'], 'sha1': 'e371433fcd1390a0b82cfb18effc422c1f5d8c1d'}, 'ceph-deploy': {'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {}}}, 'install': {'ceph': {'flavor': 'debug', 'sha1': 'e371433fcd1390a0b82cfb18effc422c1f5d8c1d'}}, 'workunit': {'branch': 'wip-shraddhaag-enable-cephadm-tests', 'env': {'CRIMSON_COMPAT': '1'}, 'sha1': 'e371433fcd1390a0b82cfb18effc422c1f5d8c1d'}}, 'owner': 'scheduled_shraddhaag@soko04.front.sepia.ceph.com', 'priority': 49, 'repo': 'https://git.ceph.com/ceph-ci.git', 'roles': [['mon.a', 'mon.b', 'mon.c', 'mgr.x', 'osd.0', 'osd.1', 'osd.2', 'client.0']], 'seed': 66, 'sha1': 'e371433fcd1390a0b82cfb18effc422c1f5d8c1d', 'sleep_before_teardown': 0, 'suite': 'crimson-rados', 'suite_branch': 'wip-shraddhaag-enable-cephadm-tests', 'suite_path': '/home/teuthworker/src/github.com_shraddhaag_ceph_e371433fcd1390a0b82cfb18effc422c1f5d8c1d/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://github.com/shraddhaag/ceph.git', 'suite_sha1': 'e371433fcd1390a0b82cfb18effc422c1f5d8c1d', 'targets': {'trial191.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBHbuM0nLiGLKLeTIrVEMqc4uTbmdtFdgvjMkr94OYob6EeBfFEoLGtnro+X3jGP9hjt0mD2BTwYG4zPY4TwqZsI='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'branch': 'distro', 'kdb': True, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': {'extra_packages': {'deb': ['ceph-osd-crimson', 'ceph-osd-crimson-dbg'], 'rpm': ['ceph-osd-crimson', 'ceph-osd-crimson-debuginfo']}, 'flavor': 'debug', 'sha1': 'e371433fcd1390a0b82cfb18effc422c1f5d8c1d'}}, {'pexec': {'all': ['sudo update-alternatives --set ceph-osd /usr/bin/ceph-osd-crimson']}}, {'install': None}, {'ceph': {'conf': {'global': {'enable experimental unrecoverable data corrupting features': 'crimson'}, 'osd': {'crimson cpu num': 2, 'osd min pg log entries': 5, 'bluestore zero block detection': True, 'crimson osd obc lru size': 10, 'debug ms': 20, 'debug osd': 20, 'debug seastore': 20, 'debug seastore backref': 20, 'debug seastore cache': 20, 'debug seastore cleaner': 20, 'debug seastore device': 20, 'debug seastore epm': 20, 'debug seastore fixedkv tree': 20, 'debug seastore journal': 20, 'debug seastore lba': 20, 'debug seastore odata': 20, 'debug seastore omap': 20, 'debug seastore onode': 20, 'debug seastore t': 20, 'debug seastore tm': 20, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd mclock override recovery settings': True, 'osd mclock profile': 'high_recovery_ops', 'osd mclock skip benchmark': True, 'osd objectstore': 'seastore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random', 'seastore cachepin size pershard': '64M', 'seastore cachepin type': '2Q', 'seastore max concurrent transactions': 8}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 30, 'debug ms': 1, 'debug paxos': 20, 'mon scrub interval': 300, 'osd pool default crimson': True}}, 'log-ignorelist': ['but it is still running', 'overall HEALTH_', '\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(POOL_APP_NOT_ENABLED\\)'], 'pre-mgr-commands': ['sudo ceph config set mgr mgr_pool false --force'], 'crimson_compat': True, 'flavor': 'debug', 'sha1': 'e371433fcd1390a0b82cfb18effc422c1f5d8c1d', 'cluster': 'ceph'}}, {'workunit': {'clients': {'client.0': ['cephtool/test_crimson_dump_metrics.sh']}}}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': []}, 'teuthology_branch': 'main', 'teuthology_sha1': 'fa17720d0088c3ac28e473468bfc79eeaff5cd38', 'timestamp': '2026-02-23_09:58:48', 'tube': 'trial', 'user': 'shraddhaag', 'verbose': False, 'worker_log': '/home/teuthworker/mnt/teuthology/worker_logs/dispatcher.trial.3318653'} 2026-02-23T11:14:03.788 DEBUG:teuthology.orchestra.run.trial191:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2026-02-23T11:14:03.836 DEBUG:teuthology.orchestra.run.trial191:> sudo install -d -m0777 -- /var/run/ceph 2026-02-23T11:14:03.901 DEBUG:teuthology.orchestra.run.trial191:> set -ex 2026-02-23T11:14:03.901 DEBUG:teuthology.orchestra.run.trial191:> dd if=/scratch_devs of=/dev/stdout 2026-02-23T11:14:03.957 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2026-02-23T11:14:03.957 DEBUG:teuthology.orchestra.run.trial191:> stat /dev/vg_nvme/lv_1 2026-02-23T11:14:04.014 INFO:teuthology.orchestra.run.trial191.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2026-02-23T11:14:04.014 INFO:teuthology.orchestra.run.trial191.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2026-02-23T11:14:04.015 INFO:teuthology.orchestra.run.trial191.stdout:Device: 6h/6d Inode: 915 Links: 1 2026-02-23T11:14:04.015 INFO:teuthology.orchestra.run.trial191.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2026-02-23T11:14:04.015 INFO:teuthology.orchestra.run.trial191.stdout:Context: system_u:object_r:device_t:s0 2026-02-23T11:14:04.015 INFO:teuthology.orchestra.run.trial191.stdout:Access: 2026-02-23 11:10:31.455872810 +0000 2026-02-23T11:14:04.015 INFO:teuthology.orchestra.run.trial191.stdout:Modify: 2026-02-23 11:08:37.133214558 +0000 2026-02-23T11:14:04.015 INFO:teuthology.orchestra.run.trial191.stdout:Change: 2026-02-23 11:08:37.133214558 +0000 2026-02-23T11:14:04.015 INFO:teuthology.orchestra.run.trial191.stdout: Birth: 2026-02-23 11:08:37.133214558 +0000 2026-02-23T11:14:04.016 DEBUG:teuthology.orchestra.run.trial191:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2026-02-23T11:14:04.077 INFO:teuthology.orchestra.run.trial191.stderr:1+0 records in 2026-02-23T11:14:04.077 INFO:teuthology.orchestra.run.trial191.stderr:1+0 records out 2026-02-23T11:14:04.077 INFO:teuthology.orchestra.run.trial191.stderr:512 bytes copied, 0.000152517 s, 3.4 MB/s 2026-02-23T11:14:04.078 DEBUG:teuthology.orchestra.run.trial191:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2026-02-23T11:14:04.136 DEBUG:teuthology.orchestra.run.trial191:> stat /dev/vg_nvme/lv_2 2026-02-23T11:14:04.192 INFO:teuthology.orchestra.run.trial191.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2026-02-23T11:14:04.192 INFO:teuthology.orchestra.run.trial191.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2026-02-23T11:14:04.192 INFO:teuthology.orchestra.run.trial191.stdout:Device: 6h/6d Inode: 945 Links: 1 2026-02-23T11:14:04.192 INFO:teuthology.orchestra.run.trial191.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2026-02-23T11:14:04.193 INFO:teuthology.orchestra.run.trial191.stdout:Context: system_u:object_r:device_t:s0 2026-02-23T11:14:04.193 INFO:teuthology.orchestra.run.trial191.stdout:Access: 2026-02-23 11:10:31.455872810 +0000 2026-02-23T11:14:04.193 INFO:teuthology.orchestra.run.trial191.stdout:Modify: 2026-02-23 11:08:37.359210820 +0000 2026-02-23T11:14:04.193 INFO:teuthology.orchestra.run.trial191.stdout:Change: 2026-02-23 11:08:37.359210820 +0000 2026-02-23T11:14:04.193 INFO:teuthology.orchestra.run.trial191.stdout: Birth: 2026-02-23 11:08:37.359210820 +0000 2026-02-23T11:14:04.193 DEBUG:teuthology.orchestra.run.trial191:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2026-02-23T11:14:04.255 INFO:teuthology.orchestra.run.trial191.stderr:1+0 records in 2026-02-23T11:14:04.256 INFO:teuthology.orchestra.run.trial191.stderr:1+0 records out 2026-02-23T11:14:04.256 INFO:teuthology.orchestra.run.trial191.stderr:512 bytes copied, 0.00013866 s, 3.7 MB/s 2026-02-23T11:14:04.257 DEBUG:teuthology.orchestra.run.trial191:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2026-02-23T11:14:04.314 DEBUG:teuthology.orchestra.run.trial191:> stat /dev/vg_nvme/lv_3 2026-02-23T11:14:04.371 INFO:teuthology.orchestra.run.trial191.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2026-02-23T11:14:04.371 INFO:teuthology.orchestra.run.trial191.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2026-02-23T11:14:04.371 INFO:teuthology.orchestra.run.trial191.stdout:Device: 6h/6d Inode: 973 Links: 1 2026-02-23T11:14:04.371 INFO:teuthology.orchestra.run.trial191.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2026-02-23T11:14:04.371 INFO:teuthology.orchestra.run.trial191.stdout:Context: system_u:object_r:device_t:s0 2026-02-23T11:14:04.371 INFO:teuthology.orchestra.run.trial191.stdout:Access: 2026-02-23 11:10:31.455872810 +0000 2026-02-23T11:14:04.371 INFO:teuthology.orchestra.run.trial191.stdout:Modify: 2026-02-23 11:08:37.593206949 +0000 2026-02-23T11:14:04.372 INFO:teuthology.orchestra.run.trial191.stdout:Change: 2026-02-23 11:08:37.593206949 +0000 2026-02-23T11:14:04.372 INFO:teuthology.orchestra.run.trial191.stdout: Birth: 2026-02-23 11:08:37.593206949 +0000 2026-02-23T11:14:04.372 DEBUG:teuthology.orchestra.run.trial191:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2026-02-23T11:14:04.434 INFO:teuthology.orchestra.run.trial191.stderr:1+0 records in 2026-02-23T11:14:04.434 INFO:teuthology.orchestra.run.trial191.stderr:1+0 records out 2026-02-23T11:14:04.434 INFO:teuthology.orchestra.run.trial191.stderr:512 bytes copied, 0.000151254 s, 3.4 MB/s 2026-02-23T11:14:04.435 DEBUG:teuthology.orchestra.run.trial191:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2026-02-23T11:14:04.493 DEBUG:teuthology.orchestra.run.trial191:> stat /dev/vg_nvme/lv_4 2026-02-23T11:14:04.549 INFO:teuthology.orchestra.run.trial191.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2026-02-23T11:14:04.549 INFO:teuthology.orchestra.run.trial191.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2026-02-23T11:14:04.549 INFO:teuthology.orchestra.run.trial191.stdout:Device: 6h/6d Inode: 1000 Links: 1 2026-02-23T11:14:04.549 INFO:teuthology.orchestra.run.trial191.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2026-02-23T11:14:04.549 INFO:teuthology.orchestra.run.trial191.stdout:Context: system_u:object_r:device_t:s0 2026-02-23T11:14:04.549 INFO:teuthology.orchestra.run.trial191.stdout:Access: 2026-02-23 11:10:31.456872798 +0000 2026-02-23T11:14:04.549 INFO:teuthology.orchestra.run.trial191.stdout:Modify: 2026-02-23 11:08:37.844202798 +0000 2026-02-23T11:14:04.549 INFO:teuthology.orchestra.run.trial191.stdout:Change: 2026-02-23 11:08:37.844202798 +0000 2026-02-23T11:14:04.549 INFO:teuthology.orchestra.run.trial191.stdout: Birth: 2026-02-23 11:08:37.844202798 +0000 2026-02-23T11:14:04.550 DEBUG:teuthology.orchestra.run.trial191:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2026-02-23T11:14:04.611 INFO:teuthology.orchestra.run.trial191.stderr:1+0 records in 2026-02-23T11:14:04.611 INFO:teuthology.orchestra.run.trial191.stderr:1+0 records out 2026-02-23T11:14:04.611 INFO:teuthology.orchestra.run.trial191.stderr:512 bytes copied, 0.000143419 s, 3.6 MB/s 2026-02-23T11:14:04.612 DEBUG:teuthology.orchestra.run.trial191:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2026-02-23T11:14:04.669 INFO:tasks.ceph:osd dev map: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2026-02-23T11:14:04.670 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@trial191.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'}} 2026-02-23T11:14:04.670 INFO:tasks.ceph:Generating config... 2026-02-23T11:14:04.671 INFO:tasks.ceph:[global] enable experimental unrecoverable data corrupting features = crimson 2026-02-23T11:14:04.671 INFO:tasks.ceph:[osd] crimson cpu num = 2 2026-02-23T11:14:04.671 INFO:tasks.ceph:[osd] osd min pg log entries = 5 2026-02-23T11:14:04.671 INFO:tasks.ceph:[osd] bluestore zero block detection = True 2026-02-23T11:14:04.671 INFO:tasks.ceph:[osd] crimson osd obc lru size = 10 2026-02-23T11:14:04.671 INFO:tasks.ceph:[osd] debug ms = 20 2026-02-23T11:14:04.671 INFO:tasks.ceph:[osd] debug osd = 20 2026-02-23T11:14:04.671 INFO:tasks.ceph:[osd] debug seastore = 20 2026-02-23T11:14:04.671 INFO:tasks.ceph:[osd] debug seastore backref = 20 2026-02-23T11:14:04.671 INFO:tasks.ceph:[osd] debug seastore cache = 20 2026-02-23T11:14:04.671 INFO:tasks.ceph:[osd] debug seastore cleaner = 20 2026-02-23T11:14:04.671 INFO:tasks.ceph:[osd] debug seastore device = 20 2026-02-23T11:14:04.671 INFO:tasks.ceph:[osd] debug seastore epm = 20 2026-02-23T11:14:04.671 INFO:tasks.ceph:[osd] debug seastore fixedkv tree = 20 2026-02-23T11:14:04.671 INFO:tasks.ceph:[osd] debug seastore journal = 20 2026-02-23T11:14:04.671 INFO:tasks.ceph:[osd] debug seastore lba = 20 2026-02-23T11:14:04.671 INFO:tasks.ceph:[osd] debug seastore odata = 20 2026-02-23T11:14:04.671 INFO:tasks.ceph:[osd] debug seastore omap = 20 2026-02-23T11:14:04.671 INFO:tasks.ceph:[osd] debug seastore onode = 20 2026-02-23T11:14:04.671 INFO:tasks.ceph:[osd] debug seastore t = 20 2026-02-23T11:14:04.671 INFO:tasks.ceph:[osd] debug seastore tm = 20 2026-02-23T11:14:04.671 INFO:tasks.ceph:[osd] osd debug verify cached snaps = True 2026-02-23T11:14:04.671 INFO:tasks.ceph:[osd] osd debug verify missing on start = True 2026-02-23T11:14:04.671 INFO:tasks.ceph:[osd] osd mclock override recovery settings = True 2026-02-23T11:14:04.672 INFO:tasks.ceph:[osd] osd mclock profile = high_recovery_ops 2026-02-23T11:14:04.672 INFO:tasks.ceph:[osd] osd mclock skip benchmark = True 2026-02-23T11:14:04.672 INFO:tasks.ceph:[osd] osd objectstore = seastore 2026-02-23T11:14:04.672 INFO:tasks.ceph:[osd] osd op queue = debug_random 2026-02-23T11:14:04.672 INFO:tasks.ceph:[osd] osd op queue cut off = debug_random 2026-02-23T11:14:04.672 INFO:tasks.ceph:[osd] seastore cachepin size pershard = 64M 2026-02-23T11:14:04.672 INFO:tasks.ceph:[osd] seastore cachepin type = 2Q 2026-02-23T11:14:04.672 INFO:tasks.ceph:[osd] seastore max concurrent transactions = 8 2026-02-23T11:14:04.672 INFO:tasks.ceph:[mgr] debug mgr = 20 2026-02-23T11:14:04.672 INFO:tasks.ceph:[mgr] debug ms = 1 2026-02-23T11:14:04.672 INFO:tasks.ceph:[mon] debug mon = 30 2026-02-23T11:14:04.672 INFO:tasks.ceph:[mon] debug ms = 1 2026-02-23T11:14:04.672 INFO:tasks.ceph:[mon] debug paxos = 20 2026-02-23T11:14:04.672 INFO:tasks.ceph:[mon] mon scrub interval = 300 2026-02-23T11:14:04.672 INFO:tasks.ceph:[mon] osd pool default crimson = True 2026-02-23T11:14:04.672 INFO:tasks.ceph:Setting up mon.a... 2026-02-23T11:14:04.672 DEBUG:teuthology.orchestra.run.trial191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/ceph.keyring 2026-02-23T11:14:04.750 INFO:teuthology.orchestra.run.trial191.stdout:creating /etc/ceph/ceph.keyring 2026-02-23T11:14:04.752 DEBUG:teuthology.orchestra.run.trial191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/ceph.keyring 2026-02-23T11:14:04.836 DEBUG:teuthology.orchestra.run.trial191:> sudo chmod 0644 /etc/ceph/ceph.keyring 2026-02-23T11:14:04.860 DEBUG:tasks.ceph:Ceph mon addresses: [('mon.a', '10.20.193.191'), ('mon.b', '[v2:10.20.193.191:3301,v1:10.20.193.191:6790]'), ('mon.c', '[v2:10.20.193.191:3302,v1:10.20.193.191:6791]')] 2026-02-23T11:14:04.860 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'cephx', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=isa technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '10.20.193.191,[v2:10.20.193.191:3301,v1:10.20.193.191:6790],[v2:10.20.193.191:3302,v1:10.20.193.191:6791]', 'enable experimental unrecoverable data corrupting features': 'crimson'}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd mclock profile': 'high_recovery_ops', 'osd mclock skip benchmark': True, 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd debug trim objects': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'crimson cpu num': 2, 'osd min pg log entries': 5, 'bluestore zero block detection': True, 'crimson osd obc lru size': 10, 'debug ms': 20, 'debug osd': 20, 'debug seastore': 20, 'debug seastore backref': 20, 'debug seastore cache': 20, 'debug seastore cleaner': 20, 'debug seastore device': 20, 'debug seastore epm': 20, 'debug seastore fixedkv tree': 20, 'debug seastore journal': 20, 'debug seastore lba': 20, 'debug seastore odata': 20, 'debug seastore omap': 20, 'debug seastore onode': 20, 'debug seastore t': 20, 'debug seastore tm': 20, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd mclock override recovery settings': True, 'osd objectstore': 'seastore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random', 'seastore cachepin size pershard': '64M', 'seastore cachepin type': '2Q', 'seastore max concurrent transactions': 8}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false'}, 'mon': {'debug ms': 1, 'debug mon': 30, 'debug paxos': 20, 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'mon_down_mkfs_grace': '2m', 'mon_warn_on_filestore_osds': 'false', 'mon scrub interval': 300, 'osd pool default crimson': True}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok'}, 'mon.a': {}, 'mon.b': {}, 'mon.c': {}} 2026-02-23T11:14:04.861 DEBUG:teuthology.orchestra.run.trial191:> set -ex 2026-02-23T11:14:04.861 DEBUG:teuthology.orchestra.run.trial191:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2026-02-23T11:14:04.916 DEBUG:teuthology.orchestra.run.trial191:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --add a 10.20.193.191 --addv b '[v2:10.20.193.191:3301,v1:10.20.193.191:6790]' --addv c '[v2:10.20.193.191:3302,v1:10.20.193.191:6791]' --print /home/ubuntu/cephtest/ceph.monmap 2026-02-23T11:14:04.999 INFO:teuthology.orchestra.run.trial191.stderr:2026-02-23T11:14:04.997+0000 7f8d6c630f40 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T11:14:05.000 INFO:teuthology.orchestra.run.trial191.stderr:2026-02-23T11:14:04.997+0000 7f8d6c630f40 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T11:14:05.000 INFO:teuthology.orchestra.run.trial191.stderr:2026-02-23T11:14:04.997+0000 7f8d6c630f40 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T11:14:05.001 INFO:teuthology.orchestra.run.trial191.stdout:monmaptool: monmap file /home/ubuntu/cephtest/ceph.monmap 2026-02-23T11:14:05.001 INFO:teuthology.orchestra.run.trial191.stdout:monmaptool: generated fsid 408068a4-9c4e-43c3-a1bd-e489279d442d 2026-02-23T11:14:05.001 INFO:teuthology.orchestra.run.trial191.stdout:setting min_mon_release = tentacle 2026-02-23T11:14:05.001 INFO:teuthology.orchestra.run.trial191.stdout:epoch 0 2026-02-23T11:14:05.001 INFO:teuthology.orchestra.run.trial191.stdout:fsid 408068a4-9c4e-43c3-a1bd-e489279d442d 2026-02-23T11:14:05.002 INFO:teuthology.orchestra.run.trial191.stdout:last_changed 2026-02-23T11:14:04.999130+0000 2026-02-23T11:14:05.002 INFO:teuthology.orchestra.run.trial191.stdout:created 2026-02-23T11:14:04.999130+0000 2026-02-23T11:14:05.002 INFO:teuthology.orchestra.run.trial191.stdout:min_mon_release 20 (tentacle) 2026-02-23T11:14:05.002 INFO:teuthology.orchestra.run.trial191.stdout:election_strategy: 1 2026-02-23T11:14:05.002 INFO:teuthology.orchestra.run.trial191.stdout:0: [v2:10.20.193.191:3300/0,v1:10.20.193.191:6789/0] mon.a 2026-02-23T11:14:05.002 INFO:teuthology.orchestra.run.trial191.stdout:1: [v2:10.20.193.191:3301/0,v1:10.20.193.191:6790/0] mon.b 2026-02-23T11:14:05.002 INFO:teuthology.orchestra.run.trial191.stdout:2: [v2:10.20.193.191:3302/0,v1:10.20.193.191:6791/0] mon.c 2026-02-23T11:14:05.002 INFO:teuthology.orchestra.run.trial191.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/ceph.monmap (3 monitors) 2026-02-23T11:14:05.003 DEBUG:teuthology.orchestra.run.trial191:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2026-02-23T11:14:05.058 INFO:tasks.ceph:Writing /etc/ceph/ceph.conf for FSID 408068a4-9c4e-43c3-a1bd-e489279d442d... 2026-02-23T11:14:05.059 DEBUG:teuthology.orchestra.run.trial191:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2026-02-23T11:14:05.139 INFO:teuthology.orchestra.run.trial191.stdout:[global] 2026-02-23T11:14:05.139 INFO:teuthology.orchestra.run.trial191.stdout: chdir = "" 2026-02-23T11:14:05.139 INFO:teuthology.orchestra.run.trial191.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2026-02-23T11:14:05.139 INFO:teuthology.orchestra.run.trial191.stdout: auth supported = cephx 2026-02-23T11:14:05.139 INFO:teuthology.orchestra.run.trial191.stdout: 2026-02-23T11:14:05.139 INFO:teuthology.orchestra.run.trial191.stdout: filestore xattr use omap = true 2026-02-23T11:14:05.139 INFO:teuthology.orchestra.run.trial191.stdout: 2026-02-23T11:14:05.139 INFO:teuthology.orchestra.run.trial191.stdout: mon clock drift allowed = 1.000 2026-02-23T11:14:05.139 INFO:teuthology.orchestra.run.trial191.stdout: 2026-02-23T11:14:05.139 INFO:teuthology.orchestra.run.trial191.stdout: osd crush chooseleaf type = 0 2026-02-23T11:14:05.139 INFO:teuthology.orchestra.run.trial191.stdout: auth debug = true 2026-02-23T11:14:05.139 INFO:teuthology.orchestra.run.trial191.stdout: 2026-02-23T11:14:05.139 INFO:teuthology.orchestra.run.trial191.stdout: ms die on old message = true 2026-02-23T11:14:05.139 INFO:teuthology.orchestra.run.trial191.stdout: ms die on bug = true 2026-02-23T11:14:05.139 INFO:teuthology.orchestra.run.trial191.stdout: 2026-02-23T11:14:05.139 INFO:teuthology.orchestra.run.trial191.stdout: mon max pg per osd = 10000 # >= luminous 2026-02-23T11:14:05.139 INFO:teuthology.orchestra.run.trial191.stdout: mon pg warn max object skew = 0 2026-02-23T11:14:05.139 INFO:teuthology.orchestra.run.trial191.stdout: 2026-02-23T11:14:05.139 INFO:teuthology.orchestra.run.trial191.stdout: # disable pg_autoscaler by default for new pools 2026-02-23T11:14:05.139 INFO:teuthology.orchestra.run.trial191.stdout: osd_pool_default_pg_autoscale_mode = off 2026-02-23T11:14:05.140 INFO:teuthology.orchestra.run.trial191.stdout: 2026-02-23T11:14:05.140 INFO:teuthology.orchestra.run.trial191.stdout: osd pool default size = 2 2026-02-23T11:14:05.140 INFO:teuthology.orchestra.run.trial191.stdout: 2026-02-23T11:14:05.140 INFO:teuthology.orchestra.run.trial191.stdout: mon osd allow primary affinity = true 2026-02-23T11:14:05.140 INFO:teuthology.orchestra.run.trial191.stdout: mon osd allow pg remap = true 2026-02-23T11:14:05.140 INFO:teuthology.orchestra.run.trial191.stdout: mon warn on legacy crush tunables = false 2026-02-23T11:14:05.140 INFO:teuthology.orchestra.run.trial191.stdout: mon warn on crush straw calc version zero = false 2026-02-23T11:14:05.140 INFO:teuthology.orchestra.run.trial191.stdout: mon warn on no sortbitwise = false 2026-02-23T11:14:05.140 INFO:teuthology.orchestra.run.trial191.stdout: mon warn on osd down out interval zero = false 2026-02-23T11:14:05.140 INFO:teuthology.orchestra.run.trial191.stdout: mon warn on too few osds = false 2026-02-23T11:14:05.140 INFO:teuthology.orchestra.run.trial191.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2026-02-23T11:14:05.140 INFO:teuthology.orchestra.run.trial191.stdout: mon_warn_on_pool_no_redundancy = false 2026-02-23T11:14:05.140 INFO:teuthology.orchestra.run.trial191.stdout: mon_allow_pool_size_one = true 2026-02-23T11:14:05.140 INFO:teuthology.orchestra.run.trial191.stdout: 2026-02-23T11:14:05.140 INFO:teuthology.orchestra.run.trial191.stdout: osd pool default erasure code profile = plugin=isa technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2026-02-23T11:14:05.140 INFO:teuthology.orchestra.run.trial191.stdout: 2026-02-23T11:14:05.140 INFO:teuthology.orchestra.run.trial191.stdout: osd default data pool replay window = 5 2026-02-23T11:14:05.140 INFO:teuthology.orchestra.run.trial191.stdout: 2026-02-23T11:14:05.140 INFO:teuthology.orchestra.run.trial191.stdout: mon allow pool delete = true 2026-02-23T11:14:05.140 INFO:teuthology.orchestra.run.trial191.stdout: 2026-02-23T11:14:05.140 INFO:teuthology.orchestra.run.trial191.stdout: mon cluster log file level = debug 2026-02-23T11:14:05.140 INFO:teuthology.orchestra.run.trial191.stdout: debug asserts on shutdown = true 2026-02-23T11:14:05.140 INFO:teuthology.orchestra.run.trial191.stdout: mon health detail to clog = false 2026-02-23T11:14:05.140 INFO:teuthology.orchestra.run.trial191.stdout: mon host = "10.20.193.191,[v2:10.20.193.191:3301,v1:10.20.193.191:6790],[v2:10.20.193.191:3302,v1:10.20.193.191:6791]" 2026-02-23T11:14:05.140 INFO:teuthology.orchestra.run.trial191.stdout: enable experimental unrecoverable data corrupting features = crimson 2026-02-23T11:14:05.140 INFO:teuthology.orchestra.run.trial191.stdout: fsid = 408068a4-9c4e-43c3-a1bd-e489279d442d 2026-02-23T11:14:05.140 INFO:teuthology.orchestra.run.trial191.stdout: 2026-02-23T11:14:05.140 INFO:teuthology.orchestra.run.trial191.stdout:[osd] 2026-02-23T11:14:05.140 INFO:teuthology.orchestra.run.trial191.stdout: osd journal size = 100 2026-02-23T11:14:05.140 INFO:teuthology.orchestra.run.trial191.stdout: 2026-02-23T11:14:05.140 INFO:teuthology.orchestra.run.trial191.stdout: osd scrub load threshold = 5.0 2026-02-23T11:14:05.140 INFO:teuthology.orchestra.run.trial191.stdout: osd scrub max interval = 600 2026-02-23T11:14:05.140 INFO:teuthology.orchestra.run.trial191.stdout: osd mclock profile = high_recovery_ops 2026-02-23T11:14:05.140 INFO:teuthology.orchestra.run.trial191.stdout: osd mclock skip benchmark = True 2026-02-23T11:14:05.140 INFO:teuthology.orchestra.run.trial191.stdout: 2026-02-23T11:14:05.140 INFO:teuthology.orchestra.run.trial191.stdout: osd recover clone overlap = true 2026-02-23T11:14:05.140 INFO:teuthology.orchestra.run.trial191.stdout: osd recovery max chunk = 1048576 2026-02-23T11:14:05.140 INFO:teuthology.orchestra.run.trial191.stdout: 2026-02-23T11:14:05.140 INFO:teuthology.orchestra.run.trial191.stdout: osd debug shutdown = true 2026-02-23T11:14:05.140 INFO:teuthology.orchestra.run.trial191.stdout: osd debug op order = true 2026-02-23T11:14:05.140 INFO:teuthology.orchestra.run.trial191.stdout: osd debug verify stray on activate = true 2026-02-23T11:14:05.141 INFO:teuthology.orchestra.run.trial191.stdout: osd debug trim objects = true 2026-02-23T11:14:05.141 INFO:teuthology.orchestra.run.trial191.stdout: 2026-02-23T11:14:05.141 INFO:teuthology.orchestra.run.trial191.stdout: osd open classes on start = true 2026-02-23T11:14:05.141 INFO:teuthology.orchestra.run.trial191.stdout: osd debug pg log writeout = true 2026-02-23T11:14:05.141 INFO:teuthology.orchestra.run.trial191.stdout: 2026-02-23T11:14:05.141 INFO:teuthology.orchestra.run.trial191.stdout: osd deep scrub update digest min age = 30 2026-02-23T11:14:05.141 INFO:teuthology.orchestra.run.trial191.stdout: 2026-02-23T11:14:05.141 INFO:teuthology.orchestra.run.trial191.stdout: osd map max advance = 10 2026-02-23T11:14:05.141 INFO:teuthology.orchestra.run.trial191.stdout: 2026-02-23T11:14:05.141 INFO:teuthology.orchestra.run.trial191.stdout: journal zero on create = true 2026-02-23T11:14:05.141 INFO:teuthology.orchestra.run.trial191.stdout: 2026-02-23T11:14:05.141 INFO:teuthology.orchestra.run.trial191.stdout: filestore ondisk finisher threads = 3 2026-02-23T11:14:05.141 INFO:teuthology.orchestra.run.trial191.stdout: filestore apply finisher threads = 3 2026-02-23T11:14:05.141 INFO:teuthology.orchestra.run.trial191.stdout: 2026-02-23T11:14:05.141 INFO:teuthology.orchestra.run.trial191.stdout: bdev debug aio = true 2026-02-23T11:14:05.141 INFO:teuthology.orchestra.run.trial191.stdout: osd debug misdirected ops = true 2026-02-23T11:14:05.141 INFO:teuthology.orchestra.run.trial191.stdout: crimson cpu num = 2 2026-02-23T11:14:05.141 INFO:teuthology.orchestra.run.trial191.stdout: osd min pg log entries = 5 2026-02-23T11:14:05.141 INFO:teuthology.orchestra.run.trial191.stdout: bluestore zero block detection = True 2026-02-23T11:14:05.141 INFO:teuthology.orchestra.run.trial191.stdout: crimson osd obc lru size = 10 2026-02-23T11:14:05.141 INFO:teuthology.orchestra.run.trial191.stdout: debug ms = 20 2026-02-23T11:14:05.141 INFO:teuthology.orchestra.run.trial191.stdout: debug osd = 20 2026-02-23T11:14:05.141 INFO:teuthology.orchestra.run.trial191.stdout: debug seastore = 20 2026-02-23T11:14:05.141 INFO:teuthology.orchestra.run.trial191.stdout: debug seastore backref = 20 2026-02-23T11:14:05.141 INFO:teuthology.orchestra.run.trial191.stdout: debug seastore cache = 20 2026-02-23T11:14:05.141 INFO:teuthology.orchestra.run.trial191.stdout: debug seastore cleaner = 20 2026-02-23T11:14:05.141 INFO:teuthology.orchestra.run.trial191.stdout: debug seastore device = 20 2026-02-23T11:14:05.141 INFO:teuthology.orchestra.run.trial191.stdout: debug seastore epm = 20 2026-02-23T11:14:05.141 INFO:teuthology.orchestra.run.trial191.stdout: debug seastore fixedkv tree = 20 2026-02-23T11:14:05.141 INFO:teuthology.orchestra.run.trial191.stdout: debug seastore journal = 20 2026-02-23T11:14:05.141 INFO:teuthology.orchestra.run.trial191.stdout: debug seastore lba = 20 2026-02-23T11:14:05.141 INFO:teuthology.orchestra.run.trial191.stdout: debug seastore odata = 20 2026-02-23T11:14:05.141 INFO:teuthology.orchestra.run.trial191.stdout: debug seastore omap = 20 2026-02-23T11:14:05.141 INFO:teuthology.orchestra.run.trial191.stdout: debug seastore onode = 20 2026-02-23T11:14:05.141 INFO:teuthology.orchestra.run.trial191.stdout: debug seastore t = 20 2026-02-23T11:14:05.141 INFO:teuthology.orchestra.run.trial191.stdout: debug seastore tm = 20 2026-02-23T11:14:05.141 INFO:teuthology.orchestra.run.trial191.stdout: osd debug verify cached snaps = True 2026-02-23T11:14:05.141 INFO:teuthology.orchestra.run.trial191.stdout: osd debug verify missing on start = True 2026-02-23T11:14:05.141 INFO:teuthology.orchestra.run.trial191.stdout: osd mclock override recovery settings = True 2026-02-23T11:14:05.141 INFO:teuthology.orchestra.run.trial191.stdout: osd objectstore = seastore 2026-02-23T11:14:05.141 INFO:teuthology.orchestra.run.trial191.stdout: osd op queue = debug_random 2026-02-23T11:14:05.141 INFO:teuthology.orchestra.run.trial191.stdout: osd op queue cut off = debug_random 2026-02-23T11:14:05.141 INFO:teuthology.orchestra.run.trial191.stdout: seastore cachepin size pershard = 64M 2026-02-23T11:14:05.141 INFO:teuthology.orchestra.run.trial191.stdout: seastore cachepin type = 2Q 2026-02-23T11:14:05.142 INFO:teuthology.orchestra.run.trial191.stdout: seastore max concurrent transactions = 8 2026-02-23T11:14:05.142 INFO:teuthology.orchestra.run.trial191.stdout: 2026-02-23T11:14:05.142 INFO:teuthology.orchestra.run.trial191.stdout:[mgr] 2026-02-23T11:14:05.142 INFO:teuthology.orchestra.run.trial191.stdout: debug ms = 1 2026-02-23T11:14:05.142 INFO:teuthology.orchestra.run.trial191.stdout: debug mgr = 20 2026-02-23T11:14:05.142 INFO:teuthology.orchestra.run.trial191.stdout: debug mon = 20 2026-02-23T11:14:05.142 INFO:teuthology.orchestra.run.trial191.stdout: debug auth = 20 2026-02-23T11:14:05.142 INFO:teuthology.orchestra.run.trial191.stdout: mon reweight min pgs per osd = 4 2026-02-23T11:14:05.142 INFO:teuthology.orchestra.run.trial191.stdout: mon reweight min bytes per osd = 10 2026-02-23T11:14:05.142 INFO:teuthology.orchestra.run.trial191.stdout: mgr/telemetry/nag = false 2026-02-23T11:14:05.142 INFO:teuthology.orchestra.run.trial191.stdout: 2026-02-23T11:14:05.142 INFO:teuthology.orchestra.run.trial191.stdout:[mon] 2026-02-23T11:14:05.142 INFO:teuthology.orchestra.run.trial191.stdout: debug ms = 1 2026-02-23T11:14:05.142 INFO:teuthology.orchestra.run.trial191.stdout: debug mon = 30 2026-02-23T11:14:05.142 INFO:teuthology.orchestra.run.trial191.stdout: debug paxos = 20 2026-02-23T11:14:05.142 INFO:teuthology.orchestra.run.trial191.stdout: debug auth = 20 2026-02-23T11:14:05.142 INFO:teuthology.orchestra.run.trial191.stdout: mon data avail warn = 5 2026-02-23T11:14:05.142 INFO:teuthology.orchestra.run.trial191.stdout: mon mgr mkfs grace = 240 2026-02-23T11:14:05.142 INFO:teuthology.orchestra.run.trial191.stdout: mon reweight min pgs per osd = 4 2026-02-23T11:14:05.142 INFO:teuthology.orchestra.run.trial191.stdout: mon osd reporter subtree level = osd 2026-02-23T11:14:05.142 INFO:teuthology.orchestra.run.trial191.stdout: mon osd prime pg temp = true 2026-02-23T11:14:05.142 INFO:teuthology.orchestra.run.trial191.stdout: mon reweight min bytes per osd = 10 2026-02-23T11:14:05.142 INFO:teuthology.orchestra.run.trial191.stdout: 2026-02-23T11:14:05.142 INFO:teuthology.orchestra.run.trial191.stdout: # rotate auth tickets quickly to exercise renewal paths 2026-02-23T11:14:05.142 INFO:teuthology.orchestra.run.trial191.stdout: auth mon ticket ttl = 660 # 11m 2026-02-23T11:14:05.142 INFO:teuthology.orchestra.run.trial191.stdout: auth service ticket ttl = 240 # 4m 2026-02-23T11:14:05.142 INFO:teuthology.orchestra.run.trial191.stdout: 2026-02-23T11:14:05.142 INFO:teuthology.orchestra.run.trial191.stdout: # don't complain about insecure global_id in the test suite 2026-02-23T11:14:05.142 INFO:teuthology.orchestra.run.trial191.stdout: mon_warn_on_insecure_global_id_reclaim = false 2026-02-23T11:14:05.142 INFO:teuthology.orchestra.run.trial191.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2026-02-23T11:14:05.142 INFO:teuthology.orchestra.run.trial191.stdout: 2026-02-23T11:14:05.142 INFO:teuthology.orchestra.run.trial191.stdout: # 1m isn't quite enough 2026-02-23T11:14:05.142 INFO:teuthology.orchestra.run.trial191.stdout: mon_down_mkfs_grace = 2m 2026-02-23T11:14:05.142 INFO:teuthology.orchestra.run.trial191.stdout: 2026-02-23T11:14:05.142 INFO:teuthology.orchestra.run.trial191.stdout: mon_warn_on_filestore_osds = false 2026-02-23T11:14:05.142 INFO:teuthology.orchestra.run.trial191.stdout: mon scrub interval = 300 2026-02-23T11:14:05.142 INFO:teuthology.orchestra.run.trial191.stdout: osd pool default crimson = True 2026-02-23T11:14:05.142 INFO:teuthology.orchestra.run.trial191.stdout: 2026-02-23T11:14:05.142 INFO:teuthology.orchestra.run.trial191.stdout:[client] 2026-02-23T11:14:05.142 INFO:teuthology.orchestra.run.trial191.stdout: rgw cache enabled = true 2026-02-23T11:14:05.142 INFO:teuthology.orchestra.run.trial191.stdout: rgw enable ops log = true 2026-02-23T11:14:05.142 INFO:teuthology.orchestra.run.trial191.stdout: rgw enable usage log = true 2026-02-23T11:14:05.142 INFO:teuthology.orchestra.run.trial191.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2026-02-23T11:14:05.143 INFO:teuthology.orchestra.run.trial191.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2026-02-23T11:14:05.143 INFO:teuthology.orchestra.run.trial191.stdout:[mon.a] 2026-02-23T11:14:05.143 INFO:teuthology.orchestra.run.trial191.stdout:[mon.b] 2026-02-23T11:14:05.143 INFO:teuthology.orchestra.run.trial191.stdout:[mon.c] 2026-02-23T11:14:05.148 INFO:tasks.ceph:Creating admin key on mon.a... 2026-02-23T11:14:05.148 DEBUG:teuthology.orchestra.run.trial191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/ceph.keyring 2026-02-23T11:14:05.238 INFO:tasks.ceph:Copying monmap to all nodes... 2026-02-23T11:14:05.238 DEBUG:teuthology.orchestra.run.trial191:> set -ex 2026-02-23T11:14:05.238 DEBUG:teuthology.orchestra.run.trial191:> dd if=/etc/ceph/ceph.keyring of=/dev/stdout 2026-02-23T11:14:05.255 DEBUG:teuthology.orchestra.run.trial191:> set -ex 2026-02-23T11:14:05.255 DEBUG:teuthology.orchestra.run.trial191:> dd if=/home/ubuntu/cephtest/ceph.monmap of=/dev/stdout 2026-02-23T11:14:05.312 INFO:tasks.ceph:Sending monmap to node ubuntu@trial191.front.sepia.ceph.com 2026-02-23T11:14:05.312 DEBUG:teuthology.orchestra.run.trial191:> set -ex 2026-02-23T11:14:05.312 DEBUG:teuthology.orchestra.run.trial191:> sudo dd of=/etc/ceph/ceph.keyring 2026-02-23T11:14:05.312 DEBUG:teuthology.orchestra.run.trial191:> sudo chmod 0644 /etc/ceph/ceph.keyring 2026-02-23T11:14:05.385 DEBUG:teuthology.orchestra.run.trial191:> set -ex 2026-02-23T11:14:05.385 DEBUG:teuthology.orchestra.run.trial191:> dd of=/home/ubuntu/cephtest/ceph.monmap 2026-02-23T11:14:05.441 INFO:tasks.ceph:Setting up mon nodes... 2026-02-23T11:14:05.441 INFO:tasks.ceph:Setting up mgr nodes... 2026-02-23T11:14:05.441 DEBUG:teuthology.orchestra.run.trial191:> sudo mkdir -p /var/lib/ceph/mgr/ceph-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/ceph-x/keyring 2026-02-23T11:14:05.530 INFO:teuthology.orchestra.run.trial191.stdout:creating /var/lib/ceph/mgr/ceph-x/keyring 2026-02-23T11:14:05.533 INFO:tasks.ceph:Setting up mds nodes... 2026-02-23T11:14:05.533 INFO:tasks.ceph_client:Setting up client nodes... 2026-02-23T11:14:05.533 DEBUG:teuthology.orchestra.run.trial191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/ceph.client.0.keyring && sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2026-02-23T11:14:05.575 INFO:teuthology.orchestra.run.trial191.stdout:creating /etc/ceph/ceph.client.0.keyring 2026-02-23T11:14:05.586 INFO:tasks.ceph:Running mkfs on osd nodes... 2026-02-23T11:14:05.587 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@trial191.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'}} 2026-02-23T11:14:05.587 DEBUG:teuthology.orchestra.run.trial191:> sudo mkdir -p /var/lib/ceph/osd/ceph-0 2026-02-23T11:14:05.652 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2026-02-23T11:14:05.652 INFO:tasks.ceph:role: osd.0 2026-02-23T11:14:05.652 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@trial191.front.sepia.ceph.com 2026-02-23T11:14:05.652 DEBUG:teuthology.orchestra.run.trial191:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2026-02-23T11:14:05.717 INFO:teuthology.orchestra.run.trial191.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=45875200 blks 2026-02-23T11:14:05.718 INFO:teuthology.orchestra.run.trial191.stdout: = sectsz=4096 attr=2, projid32bit=1 2026-02-23T11:14:05.718 INFO:teuthology.orchestra.run.trial191.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2026-02-23T11:14:05.718 INFO:teuthology.orchestra.run.trial191.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2026-02-23T11:14:05.718 INFO:teuthology.orchestra.run.trial191.stdout:data = bsize=4096 blocks=183500800, imaxpct=25 2026-02-23T11:14:05.718 INFO:teuthology.orchestra.run.trial191.stdout: = sunit=0 swidth=0 blks 2026-02-23T11:14:05.718 INFO:teuthology.orchestra.run.trial191.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2026-02-23T11:14:05.718 INFO:teuthology.orchestra.run.trial191.stdout:log =internal log bsize=4096 blocks=89600, version=2 2026-02-23T11:14:05.718 INFO:teuthology.orchestra.run.trial191.stdout: = sectsz=4096 sunit=1 blks, lazy-count=1 2026-02-23T11:14:05.718 INFO:teuthology.orchestra.run.trial191.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2026-02-23T11:14:06.287 INFO:teuthology.orchestra.run.trial191.stdout:Discarding blocks...Done. 2026-02-23T11:14:06.289 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@trial191.front.sepia.ceph.com -o noatime 2026-02-23T11:14:06.289 DEBUG:teuthology.orchestra.run.trial191:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-0 2026-02-23T11:14:06.363 DEBUG:teuthology.orchestra.run.trial191:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-0 2026-02-23T11:14:06.431 DEBUG:teuthology.orchestra.run.trial191:> sudo mkdir -p /var/lib/ceph/osd/ceph-1 2026-02-23T11:14:06.495 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2026-02-23T11:14:06.495 INFO:tasks.ceph:role: osd.1 2026-02-23T11:14:06.495 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@trial191.front.sepia.ceph.com 2026-02-23T11:14:06.496 DEBUG:teuthology.orchestra.run.trial191:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2026-02-23T11:14:06.560 INFO:teuthology.orchestra.run.trial191.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=45875200 blks 2026-02-23T11:14:06.560 INFO:teuthology.orchestra.run.trial191.stdout: = sectsz=4096 attr=2, projid32bit=1 2026-02-23T11:14:06.560 INFO:teuthology.orchestra.run.trial191.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2026-02-23T11:14:06.560 INFO:teuthology.orchestra.run.trial191.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2026-02-23T11:14:06.560 INFO:teuthology.orchestra.run.trial191.stdout:data = bsize=4096 blocks=183500800, imaxpct=25 2026-02-23T11:14:06.560 INFO:teuthology.orchestra.run.trial191.stdout: = sunit=0 swidth=0 blks 2026-02-23T11:14:06.561 INFO:teuthology.orchestra.run.trial191.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2026-02-23T11:14:06.561 INFO:teuthology.orchestra.run.trial191.stdout:log =internal log bsize=4096 blocks=89600, version=2 2026-02-23T11:14:06.561 INFO:teuthology.orchestra.run.trial191.stdout: = sectsz=4096 sunit=1 blks, lazy-count=1 2026-02-23T11:14:06.561 INFO:teuthology.orchestra.run.trial191.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2026-02-23T11:14:07.119 INFO:teuthology.orchestra.run.trial191.stdout:Discarding blocks...Done. 2026-02-23T11:14:07.122 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@trial191.front.sepia.ceph.com -o noatime 2026-02-23T11:14:07.122 DEBUG:teuthology.orchestra.run.trial191:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-1 2026-02-23T11:14:07.190 DEBUG:teuthology.orchestra.run.trial191:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-1 2026-02-23T11:14:07.256 DEBUG:teuthology.orchestra.run.trial191:> sudo mkdir -p /var/lib/ceph/osd/ceph-2 2026-02-23T11:14:07.319 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2026-02-23T11:14:07.319 INFO:tasks.ceph:role: osd.2 2026-02-23T11:14:07.320 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@trial191.front.sepia.ceph.com 2026-02-23T11:14:07.320 DEBUG:teuthology.orchestra.run.trial191:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2026-02-23T11:14:07.384 INFO:teuthology.orchestra.run.trial191.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=45875200 blks 2026-02-23T11:14:07.385 INFO:teuthology.orchestra.run.trial191.stdout: = sectsz=4096 attr=2, projid32bit=1 2026-02-23T11:14:07.385 INFO:teuthology.orchestra.run.trial191.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2026-02-23T11:14:07.385 INFO:teuthology.orchestra.run.trial191.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2026-02-23T11:14:07.385 INFO:teuthology.orchestra.run.trial191.stdout:data = bsize=4096 blocks=183500800, imaxpct=25 2026-02-23T11:14:07.385 INFO:teuthology.orchestra.run.trial191.stdout: = sunit=0 swidth=0 blks 2026-02-23T11:14:07.385 INFO:teuthology.orchestra.run.trial191.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2026-02-23T11:14:07.385 INFO:teuthology.orchestra.run.trial191.stdout:log =internal log bsize=4096 blocks=89600, version=2 2026-02-23T11:14:07.385 INFO:teuthology.orchestra.run.trial191.stdout: = sectsz=4096 sunit=1 blks, lazy-count=1 2026-02-23T11:14:07.385 INFO:teuthology.orchestra.run.trial191.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2026-02-23T11:14:07.940 INFO:teuthology.orchestra.run.trial191.stdout:Discarding blocks...Done. 2026-02-23T11:14:07.942 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@trial191.front.sepia.ceph.com -o noatime 2026-02-23T11:14:07.942 DEBUG:teuthology.orchestra.run.trial191:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-2 2026-02-23T11:14:08.010 DEBUG:teuthology.orchestra.run.trial191:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-2 2026-02-23T11:14:08.076 DEBUG:teuthology.orchestra.run.trial191:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/ceph.monmap 2026-02-23T11:14:08.228 INFO:teuthology.orchestra.run.trial191.stderr:INFO  osd - OSD::main: parsing early config 2026-02-23T11:14:08.287 INFO:teuthology.orchestra.run.trial191.stderr:INFO 2026-02-23 11:14:08,286 seastar - Reactor backend: linux-aio 2026-02-23T11:14:08.300 INFO:teuthology.orchestra.run.trial191.stderr:INFO 2026-02-23 11:14:08,299 [shard 0:main] io - IO queue was unable to find a suitable maximum request length, the search was cut-off early at: 16MB 2026-02-23T11:14:08.300 INFO:teuthology.orchestra.run.trial191.stderr:INFO 2026-02-23 11:14:08,299 [shard 0:main] io - IO queue was unable to find a suitable maximum request length, the search was cut-off early at: 16MB 2026-02-23T11:14:08.303 INFO:teuthology.orchestra.run.trial191.stderr:==38998==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2026-02-23T11:14:08.357 INFO:teuthology.orchestra.run.trial191.stderr:INFO 2026-02-23 11:14:08,356 [shard 0:main] osd - bypassing the config fetch due to --no-mon-config 2026-02-23T11:14:08.357 INFO:teuthology.orchestra.run.trial191.stderr:INFO 2026-02-23 11:14:08,356 [shard 0:main] osd - Configure option_name crimson_reactor_task_quota_ms with value : 0.500000 2026-02-23T11:14:08.357 INFO:teuthology.orchestra.run.trial191.stderr:INFO 2026-02-23 11:14:08,356 [shard 0:main] osd - Configure option_name crimson_reactor_idle_poll_time_us with value : 200 2026-02-23T11:14:08.357 INFO:teuthology.orchestra.run.trial191.stderr:INFO 2026-02-23 11:14:08,356 [shard 0:main] osd - get_early_config: set --thread-affinity 0 --smp 2 2026-02-23T11:14:08.357 INFO:teuthology.orchestra.run.trial191.stderr:DEBUG 2026-02-23 11:14:08,356 [shard 0:main] seastar - reactor::drain 2026-02-23T11:14:10.412 INFO:teuthology.orchestra.run.trial191.stderr:----------------------------------------------------- 2026-02-23T11:14:10.412 INFO:teuthology.orchestra.run.trial191.stderr:Suppressions used: 2026-02-23T11:14:10.413 INFO:teuthology.orchestra.run.trial191.stderr: count bytes template 2026-02-23T11:14:10.413 INFO:teuthology.orchestra.run.trial191.stderr: 1 8 MallocExtension::Register 2026-02-23T11:14:10.413 INFO:teuthology.orchestra.run.trial191.stderr:----------------------------------------------------- 2026-02-23T11:14:10.413 INFO:teuthology.orchestra.run.trial191.stderr: 2026-02-23T11:14:10.432 INFO:teuthology.orchestra.run.trial191.stderr:INFO  osd - OSD::main: early config parsed successfully 2026-02-23T11:14:10.432 INFO:teuthology.orchestra.run.trial191.stderr:INFO  osd - OSD::main: initializing seastar app_template 2026-02-23T11:14:10.481 INFO:teuthology.orchestra.run.trial191.stderr:INFO  osd - OSD::main: registering CLI options 2026-02-23T11:14:10.481 INFO:teuthology.orchestra.run.trial191.stderr:INFO  osd - OSD::main: entering seastar runtime 2026-02-23T11:14:10.488 INFO:teuthology.orchestra.run.trial191.stderr:INFO 2026-02-23 11:14:10,487 seastar - Reactor backend: linux-aio 2026-02-23T11:14:10.495 INFO:teuthology.orchestra.run.trial191.stderr:INFO 2026-02-23 11:14:10,495 [shard 0:main] io - IO queue was unable to find a suitable maximum request length, the search was cut-off early at: 16MB 2026-02-23T11:14:10.495 INFO:teuthology.orchestra.run.trial191.stderr:INFO 2026-02-23 11:14:10,495 [shard 0:main] io - IO queue was unable to find a suitable maximum request length, the search was cut-off early at: 16MB 2026-02-23T11:14:10.498 INFO:teuthology.orchestra.run.trial191.stderr:==38996==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2026-02-23T11:14:10.498 INFO:teuthology.orchestra.run.trial191.stderr:INFO 2026-02-23 11:14:10,497 [shard 0:main] osd - OSD::main: seastar runtime started 2026-02-23T11:14:10.556 INFO:teuthology.orchestra.run.trial191.stderr:DEBUG 2026-02-23 11:14:10,555 [shard 0:main] osd - OSD::main: initializing logger output 2026-02-23T11:14:10.635 INFO:teuthology.orchestra.run.trial191.stderr:Reactor stalled for 32 ms on shard 1, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0x9f10dfc 0x9d045c3 0x9d40a13 0x9d43fc6 0x9d44234 0x9d4437c 0x9d4483b /lib64/libc.so.6+0x3fc2f /lib64/libc.so.6+0x191c1b /lib64/libasan.so.8+0xc748 /lib64/libasan.so.8+0x91c1 /lib64/libasan.so.8+0xee7dc 0x9a48dbb 0x9b02622 0x9b02738 0x9b02814 0x9b02a90 0x9b02b49 0x9b03037 0x9b0343b 0x9b036f8 0x9ca27bf 0x9c052e1 0x9c042e1 0x7be8837 0x7bf23a6 0x7a51d33 0x7a8d6c5 0x711ff75 0x6f9f19a 0x6fa2f9b 0x6fa8b45 0x7482004 0x72a89fa 0x72a8afc 0x72a8bf4 0x735cb16 0x735cbcf 0x735cd82 0x735ceba 0x735cf35 0x735cfba 0x735d115 0x7425e13 0x748208d 0x728f241 0x72a6bae 0x73b5274 0x7432e6c 0x7432ee7 0x7432f62 0x74eccd2 0x75e6286 0x9e08ac1 0x9d0cb62 0x9dc6e39 0x9e285fb 0x9f988cc 0x9f98f48 0x26ed527 0xa5ce02e /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-23T11:14:10.635 INFO:teuthology.orchestra.run.trial191.stderr:kernel callstack: 0xffffffffffffff80 0xffffffff82e24885 0xffffffff82e24a55 0xffffffff82dcb80e 0xffffffff82dd3aeb 0xffffffff82dd3f4b 0xffffffff82a910d2 0xffffffff8374d451 0xffffffff83800ba2 2026-02-23T11:14:10.757 INFO:teuthology.orchestra.run.trial191.stderr:Reactor stalled for 65 ms on shard 0, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0x9f10dfc 0x9d045c3 0x9d40a13 0x9d43fc6 0x9d44234 0x9d4437c 0x9d4483b /lib64/libc.so.6+0x3fc2f /lib64/ld-linux-x86-64.so.2+0x16ebc /lib64/libasan.so.8+0xc65b1 /lib64/libasan.so.8+0x15197 0x2a5304b 0x7d7e215 0x7d9da0c 0x7e39ed3 0x7dbe035 0x78750f3 0x7875de8 0x79b38b2 0x7833a0a 0x78a9398 0x78f345f 0x78f36d8 0x79a02cc 0x79a042b 0x79b3a15 0x7864b7c 0x799073e 0x79907ce 0x7990872 0x73eb37a 0x7409299 0x740c6cb 0x75440fc 0x9e08ac1 0x9d0cb62 0x9dc6e39 0x9e285fb 0x9e29e18 0x9c44517 0x9c44d52 0xecf43f /lib64/libc.so.6+0x2a60f /lib64/libc.so.6+0x2a6bf 0xb46f54 2026-02-23T11:14:10.796 INFO:teuthology.orchestra.run.trial191.stderr:Reactor stalled for 65 ms on shard 1, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0x9f10dfc 0x9d045c3 0x9d40a13 0x9d43fc6 0x9d44234 0x9d4437c 0x9d4483b /lib64/libc.so.6+0x3fc2f 0x2a51fd4 0x7d7e22e 0x7d9da0c 0x7e39ed3 0x7dbe035 0x78750f3 0x7875de8 0x79b38b2 0x7833a0a 0x78a9398 0x78f345f 0x78f36d8 0x79a02cc 0x79a042b 0x79b3a15 0x7864b7c 0x799073e 0x79907ce 0x7990872 0x73eb37a 0x7409299 0x740c6cb 0x75440fc 0x9e08ac1 0x9d0cb62 0x9dc6e39 0x9e285fb 0x9f988cc 0x9f98f48 0x26ed527 0xa5ce02e /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-23T11:14:10.815 INFO:teuthology.orchestra.run.trial191.stderr:Reactor stalled for 122 ms on shard 0, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0x9f10dfc 0x9d045c3 0x9d40a13 0x9d43fc6 0x9d44234 0x9d4437c 0x9d4483b /lib64/libc.so.6+0x3fc2f /lib64/libasan.so.8+0xc65c2 /lib64/libasan.so.8+0x15197 0x2a5304b 0x7d7e215 0x7d9da0c 0x7e39ed3 0x7dbe035 0x78750f3 0x7875de8 0x79b38b2 0x7833a0a 0x78a9398 0x78f345f 0x78f36d8 0x79a02cc 0x79a042b 0x79b3a15 0x7864b7c 0x799073e 0x79907ce 0x7990872 0x73eb37a 0x7409299 0x740c6cb 0x75440fc 0x9e08ac1 0x9d0cb62 0x9dc6e39 0x9e285fb 0x9e29e18 0x9c44517 0x9c44d52 0xecf43f /lib64/libc.so.6+0x2a60f /lib64/libc.so.6+0x2a6bf 0xb46f54 2026-02-23T11:14:10.854 INFO:teuthology.orchestra.run.trial191.stderr:Reactor stalled for 122 ms on shard 1, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0x9f10dfc 0x9d045c3 0x9d40a13 0x9d43fc6 0x9d44234 0x9d4437c 0x9d4483b /lib64/libc.so.6+0x3fc2f 0x2a53179 0x7d7e215 0x7d9da0c 0x7e39ed3 0x7dbe035 0x78750f3 0x7875de8 0x79b38b2 0x7833a0a 0x78a9398 0x78f345f 0x78f36d8 0x79a02cc 0x79a042b 0x79b3a15 0x7864b7c 0x799073e 0x79907ce 0x7990872 0x73eb37a 0x7409299 0x740c6cb 0x75440fc 0x9e08ac1 0x9d0cb62 0x9dc6e39 0x9e285fb 0x9f988cc 0x9f98f48 0x26ed527 0xa5ce02e /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-23T11:14:10.965 INFO:teuthology.orchestra.run.trial191.stderr:Reactor stalled for 32 ms on shard 1, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0x9f10dfc 0x9d045c3 0x9d40a13 0x9d43fc6 0x9d44234 0x9d4437c 0x9d4483b /lib64/libc.so.6+0x3fc2f 0x4d4ac61 0x9b88005 0x9c7cb87 0x9c7cd4f 0x9c87c4b 0x9c88176 0x9c9f909 0x9ca45f1 0x9c052e1 0x9c042e1 0x7b9d452 0x7bec55b 0x7bf23a6 0x7a51d33 0x7a8d6c5 0x711ff75 0x6f3ec5e 0x6f40fc6 0x747ed38 0x72a6837 0x72a6939 0x72a6a31 0x735bce2 0x735bd9b 0x735bf4e 0x735c086 0x735c101 0x735c186 0x735c2e1 0x7425af5 0x747edc1 0x728e178 0x72a6bae 0x73b5274 0x7432e6c 0x7432ee7 0x7432f62 0x74eccd2 0x75e6286 0x9e08ac1 0x9d0cb62 0x9dc6e39 0x9e285fb 0x9f988cc 0x9f98f48 0x26ed527 0xa5ce02e /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-23T11:14:11.027 INFO:teuthology.orchestra.run.trial191.stderr:Reactor stalled for 32 ms on shard 1, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0x9f10dfc 0x9d045c3 0x9d40a13 0x9d43fc6 0x9d44234 0x9d4437c 0x9d4483b /lib64/libc.so.6+0x3fc2f /lib64/libasan.so.8+0x1297e4 /lib64/libasan.so.8+0xc65d1 /lib64/libasan.so.8+0x15197 0x2a5304b 0x7d7e215 0x7d9da0c 0x7e39ed3 0x7dbe035 0x78750f3 0x7875de8 0x79b38b2 0x7833a0a 0x78a9398 0x78f345f 0x78f36d8 0x79a02cc 0x79a042b 0x79b3a15 0x7864b7c 0x799073e 0x79907ce 0x7990872 0x73eb37a 0x7409299 0x740c6cb 0x75440fc 0x9e08ac1 0x9d0cb62 0x9dc6e39 0x9e285fb 0x9f988cc 0x9f98f48 0x26ed527 0xa5ce02e /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-23T11:14:11.048 INFO:teuthology.orchestra.run.trial191.stderr:Reactor stalled for 65 ms on shard 0, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0x9f10dfc 0x9d045c3 0x9d40a13 0x9d43fc6 0x9d44234 0x9d4437c 0x9d4483b /lib64/libc.so.6+0x3fc2f /lib64/libasan.so.8+0xc65c2 /lib64/libasan.so.8+0x15197 0x2a5304b 0x7d7e215 0x7d9da0c 0x7e39ed3 0x7dbe035 0x78750f3 0x7875de8 0x79b38b2 0x7833a0a 0x78a9398 0x78f345f 0x78f36d8 0x79a02cc 0x79a042b 0x79b3a15 0x7864b7c 0x799073e 0x79907ce 0x7990872 0x73eb37a 0x7409299 0x740c6cb 0x75440fc 0x9e08ac1 0x9d0cb62 0x9dc6e39 0x9e285fb 0x9e29e18 0x9c44517 0x9c44d52 0xecf43f /lib64/libc.so.6+0x2a60f /lib64/libc.so.6+0x2a6bf 0xb46f54 2026-02-23T11:14:11.105 INFO:teuthology.orchestra.run.trial191.stderr:Reactor stalled for 122 ms on shard 0, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0x9f10dfc 0x9d045c3 0x9d40a13 0x9d43fc6 0x9d44234 0x9d4437c 0x9d4483b /lib64/libc.so.6+0x3fc2f /lib64/libasan.so.8+0x129764 /lib64/libasan.so.8+0xc65d1 /lib64/libasan.so.8+0x15197 0x2a5304b 0x7d7e215 0x7d9da0c 0x7e39ed3 0x7dbe035 0x78750f3 0x7875de8 0x79b38b2 0x7833a0a 0x78a9398 0x78f345f 0x78f36d8 0x79a02cc 0x79a042b 0x79b3a15 0x7864b7c 0x799073e 0x79907ce 0x7990872 0x73eb37a 0x7409299 0x740c6cb 0x75440fc 0x9e08ac1 0x9d0cb62 0x9dc6e39 0x9e285fb 0x9e29e18 0x9c44517 0x9c44d52 0xecf43f /lib64/libc.so.6+0x2a60f /lib64/libc.so.6+0x2a6bf 0xb46f54 2026-02-23T11:14:13.259 INFO:teuthology.orchestra.run.trial191.stderr:----------------------------------------------------- 2026-02-23T11:14:13.259 INFO:teuthology.orchestra.run.trial191.stderr:Suppressions used: 2026-02-23T11:14:13.260 INFO:teuthology.orchestra.run.trial191.stderr: count bytes template 2026-02-23T11:14:13.260 INFO:teuthology.orchestra.run.trial191.stderr: 1 8 MallocExtension::Register 2026-02-23T11:14:13.260 INFO:teuthology.orchestra.run.trial191.stderr:----------------------------------------------------- 2026-02-23T11:14:13.260 INFO:teuthology.orchestra.run.trial191.stderr: 2026-02-23T11:14:13.292 DEBUG:teuthology.orchestra.run.trial191:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2026-02-23T11:14:13.317 DEBUG:teuthology.orchestra.run.trial191:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/ceph.monmap 2026-02-23T11:14:13.469 INFO:teuthology.orchestra.run.trial191.stderr:INFO  osd - OSD::main: parsing early config 2026-02-23T11:14:13.524 INFO:teuthology.orchestra.run.trial191.stderr:INFO 2026-02-23 11:14:13,523 seastar - Reactor backend: linux-aio 2026-02-23T11:14:13.533 INFO:teuthology.orchestra.run.trial191.stderr:INFO 2026-02-23 11:14:13,532 [shard 0:main] io - IO queue was unable to find a suitable maximum request length, the search was cut-off early at: 16MB 2026-02-23T11:14:13.533 INFO:teuthology.orchestra.run.trial191.stderr:INFO 2026-02-23 11:14:13,532 [shard 0:main] io - IO queue was unable to find a suitable maximum request length, the search was cut-off early at: 16MB 2026-02-23T11:14:13.535 INFO:teuthology.orchestra.run.trial191.stderr:==39057==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2026-02-23T11:14:13.590 INFO:teuthology.orchestra.run.trial191.stderr:INFO 2026-02-23 11:14:13,589 [shard 0:main] osd - bypassing the config fetch due to --no-mon-config 2026-02-23T11:14:13.590 INFO:teuthology.orchestra.run.trial191.stderr:INFO 2026-02-23 11:14:13,589 [shard 0:main] osd - Configure option_name crimson_reactor_task_quota_ms with value : 0.500000 2026-02-23T11:14:13.590 INFO:teuthology.orchestra.run.trial191.stderr:INFO 2026-02-23 11:14:13,589 [shard 0:main] osd - Configure option_name crimson_reactor_idle_poll_time_us with value : 200 2026-02-23T11:14:13.591 INFO:teuthology.orchestra.run.trial191.stderr:INFO 2026-02-23 11:14:13,589 [shard 0:main] osd - get_early_config: set --thread-affinity 0 --smp 2 2026-02-23T11:14:13.591 INFO:teuthology.orchestra.run.trial191.stderr:DEBUG 2026-02-23 11:14:13,589 [shard 0:main] seastar - reactor::drain 2026-02-23T11:14:15.643 INFO:teuthology.orchestra.run.trial191.stderr:----------------------------------------------------- 2026-02-23T11:14:15.643 INFO:teuthology.orchestra.run.trial191.stderr:Suppressions used: 2026-02-23T11:14:15.643 INFO:teuthology.orchestra.run.trial191.stderr: count bytes template 2026-02-23T11:14:15.644 INFO:teuthology.orchestra.run.trial191.stderr: 1 8 MallocExtension::Register 2026-02-23T11:14:15.644 INFO:teuthology.orchestra.run.trial191.stderr:----------------------------------------------------- 2026-02-23T11:14:15.644 INFO:teuthology.orchestra.run.trial191.stderr: 2026-02-23T11:14:15.663 INFO:teuthology.orchestra.run.trial191.stderr:INFO  osd - OSD::main: early config parsed successfully 2026-02-23T11:14:15.663 INFO:teuthology.orchestra.run.trial191.stderr:INFO  osd - OSD::main: initializing seastar app_template 2026-02-23T11:14:15.704 INFO:teuthology.orchestra.run.trial191.stderr:INFO  osd - OSD::main: registering CLI options 2026-02-23T11:14:15.704 INFO:teuthology.orchestra.run.trial191.stderr:INFO  osd - OSD::main: entering seastar runtime 2026-02-23T11:14:15.711 INFO:teuthology.orchestra.run.trial191.stderr:INFO 2026-02-23 11:14:15,710 seastar - Reactor backend: linux-aio 2026-02-23T11:14:15.723 INFO:teuthology.orchestra.run.trial191.stderr:INFO 2026-02-23 11:14:15,722 [shard 1:main] io - IO queue was unable to find a suitable maximum request length, the search was cut-off early at: 16MB 2026-02-23T11:14:15.723 INFO:teuthology.orchestra.run.trial191.stderr:INFO 2026-02-23 11:14:15,722 [shard 1:main] io - IO queue was unable to find a suitable maximum request length, the search was cut-off early at: 16MB 2026-02-23T11:14:15.726 INFO:teuthology.orchestra.run.trial191.stderr:==39055==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2026-02-23T11:14:15.726 INFO:teuthology.orchestra.run.trial191.stderr:INFO 2026-02-23 11:14:15,725 [shard 0:main] osd - OSD::main: seastar runtime started 2026-02-23T11:14:15.783 INFO:teuthology.orchestra.run.trial191.stderr:DEBUG 2026-02-23 11:14:15,783 [shard 0:main] osd - OSD::main: initializing logger output 2026-02-23T11:14:15.863 INFO:teuthology.orchestra.run.trial191.stderr:Reactor stalled for 32 ms on shard 1, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0x9f10dfc 0x9d045c3 0x9d40a13 0x9d43fc6 0x9d44234 0x9d4437c 0x9d4483b /lib64/libc.so.6+0x3fc2f /lib64/libasan.so.8+0x1583b 0x1091317 0xf2e3ce 0x9b7dc0b 0x9b7e0d5 0x9b7e348 0x9b7e5fe 0x9b7f0ad 0x9b7f5ab 0x9b7fbdb 0x9b7fd45 0x9c8fd63 0x9ca2b6a 0x9c052e1 0x9c042e1 0x7be8837 0x7bf23a6 0x7a51d33 0x7a8d6c5 0x711ff75 0x6f9f19a 0x6fa2f9b 0x6fa8b45 0x7482004 0x72a89fa 0x72a8afc 0x72a8bf4 0x735cb16 0x735cbcf 0x735cd82 0x735ceba 0x735cf35 0x735cfba 0x735d115 0x7425e13 0x748208d 0x728f241 0x72a6bae 0x73b5274 0x7432e6c 0x7432ee7 0x7432f62 0x74eccd2 0x75e6286 0x9e08ac1 0x9d0cb62 0x9dc6e39 0x9e285fb 0x9f988cc 0x9f98f48 0x26ed527 0xa5ce02e /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-23T11:14:15.895 INFO:teuthology.orchestra.run.trial191.stderr:Reactor stalled for 65 ms on shard 0, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0x9f10dfc 0x9d045c3 0x9d40a13 0x9d43fc6 0x9d44234 0x9d4437c 0x9d4483b /lib64/libc.so.6+0x3fc2f /lib64/ld-linux-x86-64.so.2+0x16ec6 /lib64/libasan.so.8+0xc65b1 /lib64/libasan.so.8+0x15197 0x9b3e4f1 0x9c7c93f 0x9c7cbe6 0x9c7cd4f 0x9c87c4b 0x9c88176 0x9c9f909 0x9ca45f1 0x9c052e1 0x9c042e1 0x7e58655 0x7e5b450 0x8f80d69 0x8f836d7 0x787a27f 0x78aad7c 0x6fa0365 0x75590f6 0x755924a 0x9e08ac1 0x9d0cb62 0x9dc6e39 0x9e285fb 0x9e29e18 0x9c44517 0x9c44d52 0xecf43f /lib64/libc.so.6+0x2a60f /lib64/libc.so.6+0x2a6bf 0xb46f54 2026-02-23T11:14:15.934 INFO:teuthology.orchestra.run.trial191.stderr:Reactor stalled for 65 ms on shard 1, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0x9f10dfc 0x9d045c3 0x9d40a13 0x9d43fc6 0x9d44234 0x9d4437c 0x9d4483b /lib64/libc.so.6+0x3fc2f /lib64/libasan.so.8+0xfee8b /lib64/libasan.so.8+0xfd21f /lib64/libasan.so.8+0xee859 0x9c850cc 0x9c85198 0x9c9f447 0x9ca45f1 0x9c052e1 0x9c042e1 0x7e58655 0x7e5b450 0x8f80d69 0x8f836d7 0x787a27f 0x78aad7c 0x6fa0365 0x75590f6 0x755924a 0x9e08ac1 0x9d0cb62 0x9dc6e39 0x9e285fb 0x9f988cc 0x9f98f48 0x26ed527 0xa5ce02e /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-23T11:14:15.953 INFO:teuthology.orchestra.run.trial191.stderr:Reactor stalled for 32 ms on shard 0, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0x9f10dfc 0x9d045c3 0x9d40a13 0x9d43fc6 0x9d44234 0x9d4437c 0x9d4483b /lib64/libc.so.6+0x3fc2f /lib64/libasan.so.8+0x129764 /lib64/libasan.so.8+0xc65d1 /lib64/libasan.so.8+0x15197 0x2a5304b 0x7d7e215 0x7d9da0c 0x7e39ed3 0x7dbe035 0x78750f3 0x7875de8 0x79b38b2 0x7833a0a 0x78a9398 0x78f345f 0x78f36d8 0x79a02cc 0x79a042b 0x79b3a15 0x7864b7c 0x799073e 0x79907ce 0x7990872 0x73eb37a 0x7409299 0x740c6cb 0x75440fc 0x9e08ac1 0x9d0cb62 0x9dc6e39 0x9e285fb 0x9e29e18 0x9c44517 0x9c44d52 0xecf43f /lib64/libc.so.6+0x2a60f /lib64/libc.so.6+0x2a6bf 0xb46f54 2026-02-23T11:14:16.011 INFO:teuthology.orchestra.run.trial191.stderr:Reactor stalled for 90 ms on shard 0, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0x9f10dfc 0x9d045c3 0x9d40a13 0x9d43fc6 0x9d44234 0x9d4437c 0x9d4483b /lib64/libc.so.6+0x3fc2f /lib64/libasan.so.8+0x129764 /lib64/libasan.so.8+0xc65d1 /lib64/libasan.so.8+0x15197 0x2a5304b 0x7d7e215 0x7d9da0c 0x7e39ed3 0x7dbe035 0x78750f3 0x7875de8 0x79b38b2 0x7833a0a 0x78a9398 0x78f345f 0x78f36d8 0x79a02cc 0x79a042b 0x79b3a15 0x7864b7c 0x799073e 0x79907ce 0x7990872 0x73eb37a 0x7409299 0x740c6cb 0x75440fc 0x9e08ac1 0x9d0cb62 0x9dc6e39 0x9e285fb 0x9e29e18 0x9c44517 0x9c44d52 0xecf43f /lib64/libc.so.6+0x2a60f /lib64/libc.so.6+0x2a6bf 0xb46f54 2026-02-23T11:14:16.024 INFO:teuthology.orchestra.run.trial191.stderr:Reactor stalled for 65 ms on shard 1, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0x9f10dfc 0x9d045c3 0x9d40a13 0x9d43fc6 0x9d44234 0x9d4437c 0x9d4483b /lib64/libc.so.6+0x3fc2f /lib64/libasan.so.8+0xc6592 /lib64/libasan.so.8+0x15197 0x2a5304b 0x7d7e215 0x7d9da0c 0x7e39ed3 0x7dbe035 0x78750f3 0x7875de8 0x79b38b2 0x7833a0a 0x78a9398 0x78f345f 0x78f36d8 0x79a02cc 0x79a042b 0x79b3a15 0x7864b7c 0x799073e 0x79907ce 0x7990872 0x73eb37a 0x7409299 0x740c6cb 0x75440fc 0x9e08ac1 0x9d0cb62 0x9dc6e39 0x9e285fb 0x9f988cc 0x9f98f48 0x26ed527 0xa5ce02e /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-23T11:14:16.082 INFO:teuthology.orchestra.run.trial191.stderr:Reactor stalled for 122 ms on shard 1, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0x9f10dfc 0x9d045c3 0x9d40a13 0x9d43fc6 0x9d44234 0x9d4437c 0x9d4483b /lib64/libc.so.6+0x3fc2f /lib64/libasan.so.8+0xc657f /lib64/libasan.so.8+0x15197 0x2a5304b 0x7d7e215 0x7d9da0c 0x7e39ed3 0x7dbe035 0x78750f3 0x7875de8 0x79b38b2 0x7833a0a 0x78a9398 0x78f345f 0x78f36d8 0x79a02cc 0x79a042b 0x79b3a15 0x7864b7c 0x799073e 0x79907ce 0x7990872 0x73eb37a 0x7409299 0x740c6cb 0x75440fc 0x9e08ac1 0x9d0cb62 0x9dc6e39 0x9e285fb 0x9f988cc 0x9f98f48 0x26ed527 0xa5ce02e /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-23T11:14:16.194 INFO:teuthology.orchestra.run.trial191.stderr:Reactor stalled for 32 ms on shard 1, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0x9f10dfc 0x9d045c3 0x9d40a13 0x9d43fc6 0x9d44234 0x9d4437c 0x9d4483b /lib64/libc.so.6+0x3fc2f /lib64/libc.so.6+0x16b36a /lib64/libasan.so.8+0xb0541 /lib64/libasan.so.8+0xb1437 0xb69729 0x10912a0 0xf2e3ce 0x9b81eea 0x9b82829 0x9c9ebf3 0x9ca1994 0x9ca45f1 0x9c052e1 0x9c042e1 0x7b9d452 0x7bec21f 0x7bf23a6 0x7a51d33 0x7a8d6c5 0x711ff75 0x6f3ec5e 0x6f40fc6 0x747ed38 0x72a6837 0x72a6939 0x72a6a31 0x735bce2 0x735bd9b 0x735bf4e 0x735c086 0x735c101 0x735c186 0x735c2e1 0x7425af5 0x747edc1 0x728e178 0x72a6bae 0x73b5274 0x7432e6c 0x7432ee7 0x7432f62 0x74eccd2 0x75e6286 0x9e08ac1 0x9d0cb62 0x9dc6e39 0x9e285fb 0x9f988cc 0x9f98f48 0x26ed527 0xa5ce02e /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-23T11:14:16.277 INFO:teuthology.orchestra.run.trial191.stderr:Reactor stalled for 65 ms on shard 0, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0x9f10dfc 0x9d045c3 0x9d40a13 0x9d43fc6 0x9d44234 0x9d4437c 0x9d4483b /lib64/libc.so.6+0x3fc2f 0x2a52e37 0x7d7e215 0x7d9da0c 0x7e39ed3 0x7dbe035 0x78750f3 0x7875de8 0x79b38b2 0x7833a0a 0x78a9398 0x78f345f 0x78f36d8 0x79a02cc 0x79a042b 0x79b3a15 0x7864b7c 0x799073e 0x79907ce 0x7990872 0x73eb37a 0x7409299 0x740c6cb 0x75440fc 0x9e08ac1 0x9d0cb62 0x9dc6e39 0x9e285fb 0x9e29e18 0x9c44517 0x9c44d52 0xecf43f /lib64/libc.so.6+0x2a60f /lib64/libc.so.6+0x2a6bf 0xb46f54 2026-02-23T11:14:16.335 INFO:teuthology.orchestra.run.trial191.stderr:Reactor stalled for 122 ms on shard 0, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0x9f10dfc 0x9d045c3 0x9d40a13 0x9d43fc6 0x9d44234 0x9d4437c 0x9d4483b /lib64/libc.so.6+0x3fc2f 0x29fe1a8 0x2a52fec 0x7d7e215 0x7d9da0c 0x7e39ed3 0x7dbe035 0x78750f3 0x7875de8 0x79b38b2 0x7833a0a 0x78a9398 0x78f345f 0x78f36d8 0x79a02cc 0x79a042b 0x79b3a15 0x7864b7c 0x799073e 0x79907ce 0x7990872 0x73eb37a 0x7409299 0x740c6cb 0x75440fc 0x9e08ac1 0x9d0cb62 0x9dc6e39 0x9e285fb 0x9e29e18 0x9c44517 0x9c44d52 0xecf43f /lib64/libc.so.6+0x2a60f /lib64/libc.so.6+0x2a6bf 0xb46f54 2026-02-23T11:14:18.489 INFO:teuthology.orchestra.run.trial191.stderr:----------------------------------------------------- 2026-02-23T11:14:18.490 INFO:teuthology.orchestra.run.trial191.stderr:Suppressions used: 2026-02-23T11:14:18.490 INFO:teuthology.orchestra.run.trial191.stderr: count bytes template 2026-02-23T11:14:18.490 INFO:teuthology.orchestra.run.trial191.stderr: 1 8 MallocExtension::Register 2026-02-23T11:14:18.490 INFO:teuthology.orchestra.run.trial191.stderr:----------------------------------------------------- 2026-02-23T11:14:18.490 INFO:teuthology.orchestra.run.trial191.stderr: 2026-02-23T11:14:18.522 DEBUG:teuthology.orchestra.run.trial191:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2026-02-23T11:14:18.547 DEBUG:teuthology.orchestra.run.trial191:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/ceph.monmap 2026-02-23T11:14:18.700 INFO:teuthology.orchestra.run.trial191.stderr:INFO  osd - OSD::main: parsing early config 2026-02-23T11:14:18.752 INFO:teuthology.orchestra.run.trial191.stderr:INFO 2026-02-23 11:14:18,751 seastar - Reactor backend: linux-aio 2026-02-23T11:14:18.769 INFO:teuthology.orchestra.run.trial191.stderr:INFO 2026-02-23 11:14:18,768 [shard 0:main] io - IO queue was unable to find a suitable maximum request length, the search was cut-off early at: 16MB 2026-02-23T11:14:18.769 INFO:teuthology.orchestra.run.trial191.stderr:INFO 2026-02-23 11:14:18,768 [shard 0:main] io - IO queue was unable to find a suitable maximum request length, the search was cut-off early at: 16MB 2026-02-23T11:14:18.771 INFO:teuthology.orchestra.run.trial191.stderr:==39117==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2026-02-23T11:14:18.826 INFO:teuthology.orchestra.run.trial191.stderr:INFO 2026-02-23 11:14:18,825 [shard 0:main] osd - bypassing the config fetch due to --no-mon-config 2026-02-23T11:14:18.827 INFO:teuthology.orchestra.run.trial191.stderr:INFO 2026-02-23 11:14:18,825 [shard 0:main] osd - Configure option_name crimson_reactor_task_quota_ms with value : 0.500000 2026-02-23T11:14:18.827 INFO:teuthology.orchestra.run.trial191.stderr:INFO 2026-02-23 11:14:18,825 [shard 0:main] osd - Configure option_name crimson_reactor_idle_poll_time_us with value : 200 2026-02-23T11:14:18.827 INFO:teuthology.orchestra.run.trial191.stderr:INFO 2026-02-23 11:14:18,825 [shard 0:main] osd - get_early_config: set --thread-affinity 0 --smp 2 2026-02-23T11:14:18.827 INFO:teuthology.orchestra.run.trial191.stderr:DEBUG 2026-02-23 11:14:18,826 [shard 0:main] seastar - reactor::drain 2026-02-23T11:14:20.876 INFO:teuthology.orchestra.run.trial191.stderr:----------------------------------------------------- 2026-02-23T11:14:20.876 INFO:teuthology.orchestra.run.trial191.stderr:Suppressions used: 2026-02-23T11:14:20.876 INFO:teuthology.orchestra.run.trial191.stderr: count bytes template 2026-02-23T11:14:20.876 INFO:teuthology.orchestra.run.trial191.stderr: 1 8 MallocExtension::Register 2026-02-23T11:14:20.876 INFO:teuthology.orchestra.run.trial191.stderr:----------------------------------------------------- 2026-02-23T11:14:20.877 INFO:teuthology.orchestra.run.trial191.stderr: 2026-02-23T11:14:20.896 INFO:teuthology.orchestra.run.trial191.stderr:INFO  osd - OSD::main: early config parsed successfully 2026-02-23T11:14:20.896 INFO:teuthology.orchestra.run.trial191.stderr:INFO  osd - OSD::main: initializing seastar app_template 2026-02-23T11:14:20.937 INFO:teuthology.orchestra.run.trial191.stderr:INFO  osd - OSD::main: registering CLI options 2026-02-23T11:14:20.937 INFO:teuthology.orchestra.run.trial191.stderr:INFO  osd - OSD::main: entering seastar runtime 2026-02-23T11:14:20.944 INFO:teuthology.orchestra.run.trial191.stderr:INFO 2026-02-23 11:14:20,943 seastar - Reactor backend: linux-aio 2026-02-23T11:14:20.958 INFO:teuthology.orchestra.run.trial191.stderr:INFO 2026-02-23 11:14:20,958 [shard 1:main] io - IO queue was unable to find a suitable maximum request length, the search was cut-off early at: 16MB 2026-02-23T11:14:20.958 INFO:teuthology.orchestra.run.trial191.stderr:INFO 2026-02-23 11:14:20,958 [shard 1:main] io - IO queue was unable to find a suitable maximum request length, the search was cut-off early at: 16MB 2026-02-23T11:14:20.961 INFO:teuthology.orchestra.run.trial191.stderr:==39115==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2026-02-23T11:14:20.961 INFO:teuthology.orchestra.run.trial191.stderr:INFO 2026-02-23 11:14:20,960 [shard 0:main] osd - OSD::main: seastar runtime started 2026-02-23T11:14:21.024 INFO:teuthology.orchestra.run.trial191.stderr:DEBUG 2026-02-23 11:14:21,024 [shard 0:main] osd - OSD::main: initializing logger output 2026-02-23T11:14:21.105 INFO:teuthology.orchestra.run.trial191.stderr:Reactor stalled for 32 ms on shard 1, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0x9f10dfc 0x9d045c3 0x9d40a13 0x9d43fc6 0x9d44234 0x9d4437c 0x9d4483b /lib64/libc.so.6+0x3fc2f /lib64/ld-linux-x86-64.so.2+0x16ec3 /lib64/libasan.so.8+0xc65b1 /lib64/libasan.so.8+0x15197 0x1125e3c 0xf2e3c3 0x9b7dc0b 0x9b7e0d5 0x9b7e348 0x9b7e5fe 0x9b7f0ad 0x9b7f5ab 0x9b7fbdb 0x9b7fd45 0x9c8fd63 0x9ca2b6a 0x9c052e1 0x9c042e1 0x7be8837 0x7bf23a6 0x7a51d33 0x7a8d6c5 0x711ff75 0x6f9f19a 0x6fa2f9b 0x6fa8b45 0x7482004 0x72a89fa 0x72a8afc 0x72a8bf4 0x735cb16 0x735cbcf 0x735cd82 0x735ceba 0x735cf35 0x735cfba 0x735d115 0x7425e13 0x748208d 0x728f241 0x72a6bae 0x73b5274 0x7432e6c 0x7432ee7 0x7432f62 0x74eccd2 0x75e6286 0x9e08ac1 0x9d0cb62 0x9dc6e39 0x9e285fb 0x9f988cc 0x9f98f48 0x26ed527 0xa5ce02e /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-23T11:14:21.137 INFO:teuthology.orchestra.run.trial191.stderr:Reactor stalled for 65 ms on shard 0, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0x9f10dfc 0x9d045c3 0x9d40a13 0x9d43fc6 0x9d44234 0x9d4437c 0x9d4483b /lib64/libc.so.6+0x3fc2f /lib64/libasan.so.8+0xfd16f /lib64/libasan.so.8+0xedcb5 0xe1d5af 0x9b03b21 0x9c864c2 0x9c865bb 0x7e576b2 0x7e5b450 0x8f80d69 0x8f836d7 0x787a27f 0x78aad7c 0x6fa0365 0x75590f6 0x755924a 0x9e08ac1 0x9d0cb62 0x9dc6e39 0x9e285fb 0x9e29e18 0x9c44517 0x9c44d52 0xecf43f /lib64/libc.so.6+0x2a60f /lib64/libc.so.6+0x2a6bf 0xb46f54 2026-02-23T11:14:21.176 INFO:teuthology.orchestra.run.trial191.stderr:Reactor stalled for 65 ms on shard 1, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0x9f10dfc 0x9d045c3 0x9d40a13 0x9d43fc6 0x9d44234 0x9d4437c 0x9d4483b /lib64/libc.so.6+0x3fc2f 0x34fa677 0x35eacf8 0x35eadfb 0x3606c78 0x9b8b6b3 0x9b8b796 0x9c872e7 0x9c8747c 0x9c8780e 0x9c87a19 0x9c87d55 0x9c88176 0x9c9f909 0x9ca45f1 0x9c052e1 0x9c042e1 0x7e58655 0x7e5b450 0x8f80d69 0x8f836d7 0x787a27f 0x78aad7c 0x6fa0365 0x75590f6 0x755924a 0x9e08ac1 0x9d0cb62 0x9dc6e39 0x9e285fb 0x9f988cc 0x9f98f48 0x26ed527 0xa5ce02e /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-23T11:14:21.228 INFO:teuthology.orchestra.run.trial191.stderr:Reactor stalled for 65 ms on shard 0, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0x9f10dfc 0x9d045c3 0x9d40a13 0x9d43fc6 0x9d44234 0x9d4437c 0x9d4483b /lib64/libc.so.6+0x3fc2f 0x29fe1f6 0x2a52fec 0x7d7e215 0x7d9da0c 0x7e39ed3 0x7dbe035 0x78750f3 0x7875de8 0x79b38b2 0x7833a0a 0x78a9398 0x78f345f 0x78f36d8 0x79a02cc 0x79a042b 0x79b3a15 0x7864b7c 0x799073e 0x79907ce 0x7990872 0x73eb37a 0x7409299 0x740c6cb 0x75440fc 0x9e08ac1 0x9d0cb62 0x9dc6e39 0x9e285fb 0x9e29e18 0x9c44517 0x9c44d52 0xecf43f /lib64/libc.so.6+0x2a60f /lib64/libc.so.6+0x2a6bf 0xb46f54 2026-02-23T11:14:21.267 INFO:teuthology.orchestra.run.trial191.stderr:Reactor stalled for 65 ms on shard 1, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0x9f10dfc 0x9d045c3 0x9d40a13 0x9d43fc6 0x9d44234 0x9d4437c 0x9d4483b /lib64/libc.so.6+0x3fc2f /lib64/libasan.so.8+0x1518b 0x2a5304b 0x7d7e215 0x7d9da0c 0x7e39ed3 0x7dbe035 0x78750f3 0x7875de8 0x79b38b2 0x7833a0a 0x78a9398 0x78f345f 0x78f36d8 0x79a02cc 0x79a042b 0x79b3a15 0x7864b7c 0x799073e 0x79907ce 0x7990872 0x73eb37a 0x7409299 0x740c6cb 0x75440fc 0x9e08ac1 0x9d0cb62 0x9dc6e39 0x9e285fb 0x9f988cc 0x9f98f48 0x26ed527 0xa5ce02e /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-23T11:14:21.285 INFO:teuthology.orchestra.run.trial191.stderr:Reactor stalled for 122 ms on shard 0, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0x9f10dfc 0x9d045c3 0x9d40a13 0x9d43fc6 0x9d44234 0x9d4437c 0x9d4483b /lib64/libc.so.6+0x3fc2f 0x2a51fd7 0x7d7e22e 0x7d9da0c 0x7e39ed3 0x7dbe035 0x78750f3 0x7875de8 0x79b38b2 0x7833a0a 0x78a9398 0x78f345f 0x78f36d8 0x79a02cc 0x79a042b 0x79b3a15 0x7864b7c 0x799073e 0x79907ce 0x7990872 0x73eb37a 0x7409299 0x740c6cb 0x75440fc 0x9e08ac1 0x9d0cb62 0x9dc6e39 0x9e285fb 0x9e29e18 0x9c44517 0x9c44d52 0xecf43f /lib64/libc.so.6+0x2a60f /lib64/libc.so.6+0x2a6bf 0xb46f54 2026-02-23T11:14:21.324 INFO:teuthology.orchestra.run.trial191.stderr:Reactor stalled for 122 ms on shard 1, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0x9f10dfc 0x9d045c3 0x9d40a13 0x9d43fc6 0x9d44234 0x9d4437c 0x9d4483b /lib64/libc.so.6+0x3fc2f /lib64/libasan.so.8+0xc6592 /lib64/libasan.so.8+0x15197 0x2a5304b 0x7d7e215 0x7d9da0c 0x7e39ed3 0x7dbe035 0x78750f3 0x7875de8 0x79b38b2 0x7833a0a 0x78a9398 0x78f345f 0x78f36d8 0x79a02cc 0x79a042b 0x79b3a15 0x7864b7c 0x799073e 0x79907ce 0x7990872 0x73eb37a 0x7409299 0x740c6cb 0x75440fc 0x9e08ac1 0x9d0cb62 0x9dc6e39 0x9e285fb 0x9f988cc 0x9f98f48 0x26ed527 0xa5ce02e /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-23T11:14:21.439 INFO:teuthology.orchestra.run.trial191.stderr:Reactor stalled for 32 ms on shard 1, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0x9f10dfc 0x9d045c3 0x9d40a13 0x9d43fc6 0x9d44234 0x9d4437c 0x9d4483b /lib64/libc.so.6+0x3fc2f /lib64/libasan.so.8+0xc65d9 /lib64/libasan.so.8+0x15197 0x1125e3c 0xf2e3c3 0x9c9f0b7 0x9ca16d1 0x9ca45f1 0x9c052e1 0x9c042e1 0x7e201aa 0x7bebd9d 0x7bf23a6 0x7a51d33 0x7a8d6c5 0x711ff75 0x6f3ec5e 0x6f40fc6 0x747ed38 0x72a6837 0x72a6939 0x72a6a31 0x735bce2 0x735bd9b 0x735bf4e 0x735c086 0x735c101 0x735c186 0x735c2e1 0x7425af5 0x747edc1 0x728e178 0x72a6bae 0x73b5274 0x7432e6c 0x7432ee7 0x7432f62 0x74eccd2 0x75e6286 0x9e08ac1 0x9d0cb62 0x9dc6e39 0x9e285fb 0x9f988cc 0x9f98f48 0x26ed527 0xa5ce02e /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-23T11:14:21.489 INFO:teuthology.orchestra.run.trial191.stderr:Reactor stalled for 32 ms on shard 0, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0x9f10dfc 0x9d045c3 0x9d40a13 0x9d43fc6 0x9d44234 0x9d4437c 0x9d4483b /lib64/libc.so.6+0x3fc2f /lib64/libasan.so.8+0x15197 0x2a5304b 0x7d7e215 0x7d9da0c 0x7e39ed3 0x7dbe035 0x78750f3 0x7875de8 0x79b38b2 0x7833a0a 0x78a9398 0x78f345f 0x78f36d8 0x79a02cc 0x79a042b 0x79b3a15 0x7864b7c 0x799073e 0x79907ce 0x7990872 0x73eb37a 0x7409299 0x740c6cb 0x75440fc 0x9e08ac1 0x9d0cb62 0x9dc6e39 0x9e285fb 0x9e29e18 0x9c44517 0x9c44d52 0xecf43f /lib64/libc.so.6+0x2a60f /lib64/libc.so.6+0x2a6bf 0xb46f54 2026-02-23T11:14:21.547 INFO:teuthology.orchestra.run.trial191.stderr:Reactor stalled for 90 ms on shard 0, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0x9f10dfc 0x9d045c3 0x9d40a13 0x9d43fc6 0x9d44234 0x9d4437c 0x9d4483b /lib64/libc.so.6+0x3fc2f 0x2a52f9c 0x7d7e215 0x7d9da0c 0x7e39ed3 0x7dbe035 0x78750f3 0x7875de8 0x79b38b2 0x7833a0a 0x78a9398 0x78f345f 0x78f36d8 0x79a02cc 0x79a042b 0x79b3a15 0x7864b7c 0x799073e 0x79907ce 0x7990872 0x73eb37a 0x7409299 0x740c6cb 0x75440fc 0x9e08ac1 0x9d0cb62 0x9dc6e39 0x9e285fb 0x9e29e18 0x9c44517 0x9c44d52 0xecf43f /lib64/libc.so.6+0x2a60f /lib64/libc.so.6+0x2a6bf 0xb46f54 2026-02-23T11:14:23.754 INFO:teuthology.orchestra.run.trial191.stderr:----------------------------------------------------- 2026-02-23T11:14:23.754 INFO:teuthology.orchestra.run.trial191.stderr:Suppressions used: 2026-02-23T11:14:23.754 INFO:teuthology.orchestra.run.trial191.stderr: count bytes template 2026-02-23T11:14:23.754 INFO:teuthology.orchestra.run.trial191.stderr: 1 8 MallocExtension::Register 2026-02-23T11:14:23.754 INFO:teuthology.orchestra.run.trial191.stderr:----------------------------------------------------- 2026-02-23T11:14:23.754 INFO:teuthology.orchestra.run.trial191.stderr: 2026-02-23T11:14:23.787 DEBUG:teuthology.orchestra.run.trial191:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2026-02-23T11:14:23.812 INFO:tasks.ceph:Reading keys from all nodes... 2026-02-23T11:14:23.812 DEBUG:teuthology.orchestra.run.trial191:> set -ex 2026-02-23T11:14:23.812 DEBUG:teuthology.orchestra.run.trial191:> sudo dd if=/var/lib/ceph/mgr/ceph-x/keyring of=/dev/stdout 2026-02-23T11:14:23.877 DEBUG:teuthology.orchestra.run.trial191:> set -ex 2026-02-23T11:14:23.877 DEBUG:teuthology.orchestra.run.trial191:> sudo dd if=/var/lib/ceph/osd/ceph-0/keyring of=/dev/stdout 2026-02-23T11:14:23.940 DEBUG:teuthology.orchestra.run.trial191:> set -ex 2026-02-23T11:14:23.941 DEBUG:teuthology.orchestra.run.trial191:> sudo dd if=/var/lib/ceph/osd/ceph-1/keyring of=/dev/stdout 2026-02-23T11:14:24.005 DEBUG:teuthology.orchestra.run.trial191:> set -ex 2026-02-23T11:14:24.005 DEBUG:teuthology.orchestra.run.trial191:> sudo dd if=/var/lib/ceph/osd/ceph-2/keyring of=/dev/stdout 2026-02-23T11:14:24.069 DEBUG:teuthology.orchestra.run.trial191:> set -ex 2026-02-23T11:14:24.069 DEBUG:teuthology.orchestra.run.trial191:> dd if=/etc/ceph/ceph.client.0.keyring of=/dev/stdout 2026-02-23T11:14:24.125 INFO:tasks.ceph:Adding keys to all mons... 2026-02-23T11:14:24.125 DEBUG:teuthology.orchestra.run.trial191:> sudo tee -a /etc/ceph/ceph.keyring 2026-02-23T11:14:24.187 INFO:teuthology.orchestra.run.trial191.stdout:[mgr.x] 2026-02-23T11:14:24.187 INFO:teuthology.orchestra.run.trial191.stdout: key = AQB9NpxpYR+VHxAAFrEBZ+J8ct2j+NxwZMVPQQ== 2026-02-23T11:14:24.187 INFO:teuthology.orchestra.run.trial191.stdout:[osd.0] 2026-02-23T11:14:24.187 INFO:teuthology.orchestra.run.trial191.stdout: key = AQCCNpxpj0lVIRAACh4Yz86ab8o+4gJ0kyCOGg== 2026-02-23T11:14:24.187 INFO:teuthology.orchestra.run.trial191.stdout:[osd.1] 2026-02-23T11:14:24.187 INFO:teuthology.orchestra.run.trial191.stdout: key = AQCHNpxpuwjmLhAAQI3FkFjYFGw+msDNb7y3kQ== 2026-02-23T11:14:24.187 INFO:teuthology.orchestra.run.trial191.stdout:[osd.2] 2026-02-23T11:14:24.187 INFO:teuthology.orchestra.run.trial191.stdout: key = AQCNNpxpKQmvARAARWb8Ui9y6Lv6olbfXk6Muw== 2026-02-23T11:14:24.187 INFO:teuthology.orchestra.run.trial191.stdout:[client.0] 2026-02-23T11:14:24.188 INFO:teuthology.orchestra.run.trial191.stdout: key = AQB9Npxpa3REIhAAUO8l3Et4waX64cJvqy4QQA== 2026-02-23T11:14:24.189 DEBUG:teuthology.orchestra.run.trial191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2026-02-23T11:14:24.278 DEBUG:teuthology.orchestra.run.trial191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2026-02-23T11:14:24.326 DEBUG:teuthology.orchestra.run.trial191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2026-02-23T11:14:24.373 DEBUG:teuthology.orchestra.run.trial191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2026-02-23T11:14:24.422 DEBUG:teuthology.orchestra.run.trial191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2026-02-23T11:14:24.471 INFO:tasks.ceph:Running mkfs on mon nodes... 2026-02-23T11:14:24.472 DEBUG:teuthology.orchestra.run.trial191:> sudo mkdir -p /var/lib/ceph/mon/ceph-a 2026-02-23T11:14:24.496 DEBUG:teuthology.orchestra.run.trial191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i a --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2026-02-23T11:14:24.582 INFO:teuthology.orchestra.run.trial191.stderr:2026-02-23T11:14:24.580+0000 7f58c0ec4c00 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T11:14:24.582 INFO:teuthology.orchestra.run.trial191.stderr:2026-02-23T11:14:24.580+0000 7f58c0ec4c00 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T11:14:24.594 DEBUG:teuthology.orchestra.run.trial191:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-a 2026-02-23T11:14:24.619 DEBUG:teuthology.orchestra.run.trial191:> sudo mkdir -p /var/lib/ceph/mon/ceph-b 2026-02-23T11:14:24.683 DEBUG:teuthology.orchestra.run.trial191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i b --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2026-02-23T11:14:24.767 INFO:teuthology.orchestra.run.trial191.stderr:2026-02-23T11:14:24.766+0000 7fec2fe39c00 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T11:14:24.768 INFO:teuthology.orchestra.run.trial191.stderr:2026-02-23T11:14:24.766+0000 7fec2fe39c00 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T11:14:24.780 DEBUG:teuthology.orchestra.run.trial191:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-b 2026-02-23T11:14:24.804 DEBUG:teuthology.orchestra.run.trial191:> sudo mkdir -p /var/lib/ceph/mon/ceph-c 2026-02-23T11:14:24.867 DEBUG:teuthology.orchestra.run.trial191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i c --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2026-02-23T11:14:24.951 INFO:teuthology.orchestra.run.trial191.stderr:2026-02-23T11:14:24.950+0000 7f9136c95c00 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T11:14:24.952 INFO:teuthology.orchestra.run.trial191.stderr:2026-02-23T11:14:24.950+0000 7f9136c95c00 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T11:14:24.963 DEBUG:teuthology.orchestra.run.trial191:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-c 2026-02-23T11:14:24.988 DEBUG:teuthology.orchestra.run.trial191:> rm -- /home/ubuntu/cephtest/ceph.monmap 2026-02-23T11:14:25.044 INFO:tasks.ceph:Starting mon daemons in cluster ceph... 2026-02-23T11:14:25.044 INFO:tasks.ceph.mon.a:Restarting daemon 2026-02-23T11:14:25.044 DEBUG:teuthology.orchestra.run.trial191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2026-02-23T11:14:25.086 INFO:tasks.ceph.mon.a:Started 2026-02-23T11:14:25.086 INFO:tasks.ceph.mon.b:Restarting daemon 2026-02-23T11:14:25.086 DEBUG:teuthology.orchestra.run.trial191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i b 2026-02-23T11:14:25.088 INFO:tasks.ceph.mon.b:Started 2026-02-23T11:14:25.088 INFO:tasks.ceph.mon.c:Restarting daemon 2026-02-23T11:14:25.089 DEBUG:teuthology.orchestra.run.trial191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i c 2026-02-23T11:14:25.090 INFO:tasks.ceph.mon.c:Started 2026-02-23T11:14:25.091 DEBUG:teuthology.orchestra.run.trial191:> sudo ceph config set mgr mgr_pool false --force 2026-02-23T11:14:25.150 INFO:tasks.ceph.mon.a.trial191.stderr:2026-02-23T11:14:25.148+0000 7fa9976dcc00 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T11:14:25.150 INFO:tasks.ceph.mon.a.trial191.stderr:2026-02-23T11:14:25.148+0000 7fa9976dcc00 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T11:14:25.150 INFO:tasks.ceph.mon.a.trial191.stderr:2026-02-23T11:14:25.148+0000 7fa9976dcc00 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T11:14:25.153 INFO:tasks.ceph.mon.b.trial191.stderr:2026-02-23T11:14:25.151+0000 7f6b36c8ec00 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T11:14:25.153 INFO:tasks.ceph.mon.b.trial191.stderr:2026-02-23T11:14:25.151+0000 7f6b36c8ec00 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T11:14:25.153 INFO:tasks.ceph.mon.b.trial191.stderr:2026-02-23T11:14:25.151+0000 7f6b36c8ec00 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T11:14:25.154 INFO:tasks.ceph.mon.c.trial191.stderr:2026-02-23T11:14:25.152+0000 7f55222a5c00 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T11:14:25.154 INFO:tasks.ceph.mon.c.trial191.stderr:2026-02-23T11:14:25.152+0000 7f55222a5c00 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T11:14:25.154 INFO:tasks.ceph.mon.c.trial191.stderr:2026-02-23T11:14:25.152+0000 7f55222a5c00 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T11:14:25.170 INFO:teuthology.orchestra.run.trial191.stderr:2026-02-23T11:14:25.164+0000 7f364fa1a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T11:14:25.377 INFO:teuthology.orchestra.run.trial191.stderr:2026-02-23T11:14:25.376+0000 7f364fa1a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T11:14:25.421 INFO:tasks.ceph:Starting mgr daemons in cluster ceph... 2026-02-23T11:14:25.421 INFO:tasks.ceph.mgr.x:Restarting daemon 2026-02-23T11:14:25.421 DEBUG:teuthology.orchestra.run.trial191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i x 2026-02-23T11:14:25.423 INFO:tasks.ceph.mgr.x:Started 2026-02-23T11:14:25.424 DEBUG:tasks.ceph:set 0 configs 2026-02-23T11:14:25.424 DEBUG:teuthology.orchestra.run.trial191:> sudo ceph --cluster ceph config dump 2026-02-23T11:14:25.483 INFO:tasks.ceph.mgr.x.trial191.stderr:2026-02-23T11:14:25.482+0000 7fb615614fc0 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T11:14:25.486 INFO:tasks.ceph.mgr.x.trial191.stderr:2026-02-23T11:14:25.484+0000 7fb615614fc0 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T11:14:25.486 INFO:tasks.ceph.mgr.x.trial191.stderr:2026-02-23T11:14:25.485+0000 7fb615614fc0 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T11:14:25.496 INFO:teuthology.orchestra.run.trial191.stderr:2026-02-23T11:14:25.493+0000 7f19741b8640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T11:14:25.499 INFO:teuthology.orchestra.run.trial191.stderr:2026-02-23T11:14:25.497+0000 7f19741b8640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T11:14:25.532 INFO:teuthology.orchestra.run.trial191.stdout:WHO MASK LEVEL OPTION VALUE RO 2026-02-23T11:14:25.532 INFO:teuthology.orchestra.run.trial191.stdout:mgr dev mgr_pool false * 2026-02-23T11:14:25.540 INFO:tasks.ceph:Setting crush tunables to default 2026-02-23T11:14:25.540 DEBUG:teuthology.orchestra.run.trial191:> sudo ceph --cluster ceph osd crush tunables default 2026-02-23T11:14:25.614 INFO:teuthology.orchestra.run.trial191.stderr:2026-02-23T11:14:25.610+0000 7fe09b8cb640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T11:14:25.617 INFO:teuthology.orchestra.run.trial191.stderr:2026-02-23T11:14:25.616+0000 7fe09b8cb640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T11:14:25.654 INFO:teuthology.orchestra.run.trial191.stderr:adjusted tunables profile to default 2026-02-23T11:14:25.663 INFO:tasks.ceph:check_enable_crimson: True 2026-02-23T11:14:25.663 INFO:tasks.ceph:check_enable_crimson: setting set-allow-crimson 2026-02-23T11:14:25.663 DEBUG:teuthology.orchestra.run.trial191:> sudo ceph --cluster ceph osd set-allow-crimson --yes-i-really-mean-it 2026-02-23T11:14:25.739 INFO:teuthology.orchestra.run.trial191.stderr:2026-02-23T11:14:25.734+0000 7ff886f74640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T11:14:25.741 INFO:teuthology.orchestra.run.trial191.stderr:2026-02-23T11:14:25.740+0000 7ff886f74640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T11:14:25.781 INFO:tasks.ceph.mon.a.trial191.stderr:2026-02-23T11:14:25.779+0000 7fa98edf1640 -1 WARNING: experimental feature 'crimson' is enabled 2026-02-23T11:14:25.781 INFO:tasks.ceph.mon.a.trial191.stderr:Please be aware that this feature is experimental, untested, 2026-02-23T11:14:25.781 INFO:tasks.ceph.mon.a.trial191.stderr:unsupported, and may result in data corruption, data loss, 2026-02-23T11:14:25.781 INFO:tasks.ceph.mon.a.trial191.stderr:and/or irreparable damage to your cluster. Do not use 2026-02-23T11:14:25.781 INFO:tasks.ceph.mon.a.trial191.stderr:feature with important data. 2026-02-23T11:14:25.781 INFO:tasks.ceph.mon.a.trial191.stderr: 2026-02-23T11:14:26.320 INFO:tasks.ceph.mgr.x.trial191.stderr:/usr/lib64/python3.9/site-packages/scipy/__init__.py:73: UserWarning: NumPy was imported from a Python sub-interpreter but NumPy does not properly support sub-interpreters. This will likely work for most users but might cause hard to track down issues or subtle bugs. A common user of the rare sub-interpreter feature is wsgi which also allows single-interpreter mode. 2026-02-23T11:14:26.320 INFO:tasks.ceph.mgr.x.trial191.stderr:Improvements in the case of bugs are welcome, but is not on the NumPy roadmap, and full support may require significant effort to achieve. 2026-02-23T11:14:26.320 INFO:tasks.ceph.mgr.x.trial191.stderr: from numpy import show_config as show_numpy_config 2026-02-23T11:14:26.664 INFO:tasks.ceph:Starting osd daemons in cluster ceph... 2026-02-23T11:14:26.664 DEBUG:teuthology.orchestra.run.trial191:> set -ex 2026-02-23T11:14:26.664 DEBUG:teuthology.orchestra.run.trial191:> sudo dd if=/var/lib/ceph/osd/ceph-0/fsid of=/dev/stdout 2026-02-23T11:14:26.690 DEBUG:teuthology.orchestra.run.trial191:> set -ex 2026-02-23T11:14:26.690 DEBUG:teuthology.orchestra.run.trial191:> sudo dd if=/var/lib/ceph/osd/ceph-1/fsid of=/dev/stdout 2026-02-23T11:14:26.754 DEBUG:teuthology.orchestra.run.trial191:> set -ex 2026-02-23T11:14:26.755 DEBUG:teuthology.orchestra.run.trial191:> sudo dd if=/var/lib/ceph/osd/ceph-2/fsid of=/dev/stdout 2026-02-23T11:14:26.819 DEBUG:teuthology.orchestra.run.trial191:> sudo ceph --cluster ceph osd new e4f36d13-e8b8-4777-85ed-8210652f197c 0 2026-02-23T11:14:26.932 INFO:teuthology.orchestra.run.trial191.stderr:2026-02-23T11:14:26.928+0000 7f5e9ca1a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T11:14:26.936 INFO:teuthology.orchestra.run.trial191.stderr:2026-02-23T11:14:26.934+0000 7f5e9ca1a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T11:14:26.973 INFO:teuthology.orchestra.run.trial191.stdout:0 2026-02-23T11:14:26.982 DEBUG:teuthology.orchestra.run.trial191:> sudo ceph --cluster ceph osd new e39a1968-9db1-42d6-b792-e3aa32109d97 1 2026-02-23T11:14:27.056 INFO:teuthology.orchestra.run.trial191.stderr:2026-02-23T11:14:27.051+0000 7efc4025a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T11:14:27.059 INFO:teuthology.orchestra.run.trial191.stderr:2026-02-23T11:14:27.057+0000 7efc4025a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T11:14:27.098 INFO:teuthology.orchestra.run.trial191.stdout:1 2026-02-23T11:14:27.106 DEBUG:teuthology.orchestra.run.trial191:> sudo ceph --cluster ceph osd new 09f145f1-9abb-4692-b128-bf1aff6fe492 2 2026-02-23T11:14:27.177 INFO:teuthology.orchestra.run.trial191.stderr:2026-02-23T11:14:27.171+0000 7fa7b601a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T11:14:27.180 INFO:teuthology.orchestra.run.trial191.stderr:2026-02-23T11:14:27.178+0000 7fa7b601a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T11:14:27.219 INFO:teuthology.orchestra.run.trial191.stdout:2 2026-02-23T11:14:27.227 DEBUG:teuthology.orchestra.run.trial191:> sudo install -b -o ubuntu -g ubuntu /dev/null /var/log/ceph/ceph-osd.0.log 2026-02-23T11:14:27.256 INFO:tasks.ceph.osd.0:Restarting daemon 2026-02-23T11:14:27.256 DEBUG:teuthology.orchestra.run.trial191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 0 2>> /var/log/ceph/ceph-osd.0.log 2026-02-23T11:14:27.298 INFO:tasks.ceph.osd.0:Started 2026-02-23T11:14:27.298 DEBUG:teuthology.orchestra.run.trial191:> sudo install -b -o ubuntu -g ubuntu /dev/null /var/log/ceph/ceph-osd.1.log 2026-02-23T11:14:27.326 INFO:tasks.ceph.osd.1:Restarting daemon 2026-02-23T11:14:27.326 DEBUG:teuthology.orchestra.run.trial191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 1 2>> /var/log/ceph/ceph-osd.1.log 2026-02-23T11:14:27.369 INFO:tasks.ceph.osd.1:Started 2026-02-23T11:14:27.369 DEBUG:teuthology.orchestra.run.trial191:> sudo install -b -o ubuntu -g ubuntu /dev/null /var/log/ceph/ceph-osd.2.log 2026-02-23T11:14:27.395 INFO:tasks.ceph.osd.2:Restarting daemon 2026-02-23T11:14:27.396 DEBUG:teuthology.orchestra.run.trial191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 2 2>> /var/log/ceph/ceph-osd.2.log 2026-02-23T11:14:27.439 INFO:tasks.ceph.osd.2:Started 2026-02-23T11:14:27.439 DEBUG:teuthology.orchestra.run.trial191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2026-02-23T11:14:27.515 INFO:teuthology.orchestra.run.trial191.stderr:2026-02-23T11:14:27.511+0000 7f8da5ec2640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T11:14:27.518 INFO:teuthology.orchestra.run.trial191.stderr:2026-02-23T11:14:27.516+0000 7f8da5ec2640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T11:14:27.554 INFO:teuthology.orchestra.run.trial191.stdout: 2026-02-23T11:14:27.554 INFO:teuthology.orchestra.run.trial191.stdout:{"epoch":6,"fsid":"408068a4-9c4e-43c3-a1bd-e489279d442d","created":"2026-02-23T11:14:25.368540+0000","modified":"2026-02-23T11:14:27.217466+0000","last_up_change":"0.000000","last_in_change":"2026-02-23T11:14:27.217466+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[],"osds":[{"osd":0,"uuid":"e4f36d13-e8b8-4777-85ed-8210652f197c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"e39a1968-9db1-42d6-b792-e3aa32109d97","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"09f145f1-9abb-4692-b128-bf1aff6fe492","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T11:14:27.563 INFO:tasks.ceph.ceph_manager.ceph:[] 2026-02-23T11:14:27.563 INFO:tasks.ceph:Waiting for OSDs to come up 2026-02-23T11:14:27.865 DEBUG:teuthology.orchestra.run.trial191:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2026-02-23T11:14:27.934 INFO:teuthology.misc.health.trial191.stderr:2026-02-23T11:14:27.930+0000 7f71a7c11640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T11:14:27.937 INFO:teuthology.misc.health.trial191.stderr:2026-02-23T11:14:27.935+0000 7f71a7c11640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T11:14:27.970 INFO:teuthology.misc.health.trial191.stdout: 2026-02-23T11:14:27.970 INFO:teuthology.misc.health.trial191.stdout:{"epoch":6,"fsid":"408068a4-9c4e-43c3-a1bd-e489279d442d","created":"2026-02-23T11:14:25.368540+0000","modified":"2026-02-23T11:14:27.217466+0000","last_up_change":"0.000000","last_in_change":"2026-02-23T11:14:27.217466+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[],"osds":[{"osd":0,"uuid":"e4f36d13-e8b8-4777-85ed-8210652f197c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"e39a1968-9db1-42d6-b792-e3aa32109d97","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"09f145f1-9abb-4692-b128-bf1aff6fe492","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T11:14:27.978 DEBUG:teuthology.misc:0 of 3 OSDs are up 2026-02-23T11:14:28.530 INFO:tasks.ceph.mgr.x.trial191.stderr:2026-02-23T11:14:28.528+0000 7fb532a67640 -1 mgr.server handle_report got status from non-daemon mon.c 2026-02-23T11:14:28.531 INFO:tasks.ceph.mgr.x.trial191.stderr:2026-02-23T11:14:28.528+0000 7fb532a67640 -1 mgr.server handle_report got status from non-daemon mon.b 2026-02-23T11:14:34.280 DEBUG:teuthology.orchestra.run.trial191:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2026-02-23T11:14:34.355 INFO:teuthology.misc.health.trial191.stderr:2026-02-23T11:14:34.352+0000 7f9819ccb640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T11:14:34.359 INFO:teuthology.misc.health.trial191.stderr:2026-02-23T11:14:34.357+0000 7f9819ccb640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T11:14:34.484 INFO:teuthology.misc.health.trial191.stdout: 2026-02-23T11:14:34.485 INFO:teuthology.misc.health.trial191.stdout:{"epoch":7,"fsid":"408068a4-9c4e-43c3-a1bd-e489279d442d","created":"2026-02-23T11:14:25.368540+0000","modified":"2026-02-23T11:14:33.938307+0000","last_up_change":"0.000000","last_in_change":"2026-02-23T11:14:27.217466+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":3,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[],"osds":[{"osd":0,"uuid":"e4f36d13-e8b8-4777-85ed-8210652f197c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"e39a1968-9db1-42d6-b792-e3aa32109d97","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"09f145f1-9abb-4692-b128-bf1aff6fe492","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T11:14:34.495 DEBUG:teuthology.misc:0 of 3 OSDs are up 2026-02-23T11:14:40.798 DEBUG:teuthology.orchestra.run.trial191:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2026-02-23T11:14:40.867 INFO:teuthology.misc.health.trial191.stderr:2026-02-23T11:14:40.863+0000 7f5bc510b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T11:14:40.870 INFO:teuthology.misc.health.trial191.stderr:2026-02-23T11:14:40.868+0000 7f5bc510b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T11:14:40.982 INFO:teuthology.misc.health.trial191.stdout: 2026-02-23T11:14:40.982 INFO:teuthology.misc.health.trial191.stdout:{"epoch":10,"fsid":"408068a4-9c4e-43c3-a1bd-e489279d442d","created":"2026-02-23T11:14:25.368540+0000","modified":"2026-02-23T11:14:36.956000+0000","last_up_change":"2026-02-23T11:14:36.956000+0000","last_in_change":"2026-02-23T11:14:27.217466+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[],"osds":[{"osd":0,"uuid":"e4f36d13-e8b8-4777-85ed-8210652f197c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.191:6802","nonce":3272030818}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.191:6803","nonce":3272030818}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.191:6806","nonce":3272030818}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.191:6809","nonce":3272030818}]},"public_addr":"10.20.193.191:6802/3272030818","cluster_addr":"10.20.193.191:6803/3272030818","heartbeat_back_addr":"10.20.193.191:6806/3272030818","heartbeat_front_addr":"10.20.193.191:6809/3272030818","state":["exists","up"]},{"osd":1,"uuid":"e39a1968-9db1-42d6-b792-e3aa32109d97","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":10,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.191:6808","nonce":2597130977}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.191:6807","nonce":2597130977}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.191:6811","nonce":2597130977}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.191:6810","nonce":2597130977}]},"public_addr":"10.20.193.191:6808/2597130977","cluster_addr":"10.20.193.191:6807/2597130977","heartbeat_back_addr":"10.20.193.191:6811/2597130977","heartbeat_front_addr":"10.20.193.191:6810/2597130977","state":["exists","up"]},{"osd":2,"uuid":"09f145f1-9abb-4692-b128-bf1aff6fe492","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":10,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.191:6805","nonce":3503848635}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.191:6804","nonce":3503848635}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.191:6813","nonce":3503848635}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.191:6812","nonce":3503848635}]},"public_addr":"10.20.193.191:6805/3503848635","cluster_addr":"10.20.193.191:6804/3503848635","heartbeat_back_addr":"10.20.193.191:6813/3503848635","heartbeat_front_addr":"10.20.193.191:6812/3503848635","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T11:14:40.991 DEBUG:teuthology.misc:3 of 3 OSDs are up 2026-02-23T11:14:40.991 INFO:tasks.ceph:Creating RBD pool 2026-02-23T11:14:40.991 DEBUG:teuthology.orchestra.run.trial191:> sudo ceph --cluster ceph osd pool create rbd 8 2026-02-23T11:14:41.067 INFO:teuthology.orchestra.run.trial191.stderr:2026-02-23T11:14:41.061+0000 7fd5ff7a4640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T11:14:41.069 INFO:teuthology.orchestra.run.trial191.stderr:2026-02-23T11:14:41.067+0000 7fd5ff7a4640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T11:14:42.037 INFO:teuthology.orchestra.run.trial191.stderr:pool 'rbd' created 2026-02-23T11:14:42.049 DEBUG:teuthology.orchestra.run.trial191:> rbd --cluster ceph pool init rbd 2026-02-23T11:14:42.088 INFO:teuthology.orchestra.run.trial191.stderr:2026-02-23T11:14:42.087+0000 7fc34ccef240 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T11:14:42.088 INFO:teuthology.orchestra.run.trial191.stderr:2026-02-23T11:14:42.087+0000 7fc34ccef240 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T11:14:42.092 INFO:teuthology.orchestra.run.trial191.stderr:2026-02-23T11:14:42.090+0000 7fc34ccef240 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T11:14:45.075 INFO:tasks.ceph:Starting mds daemons in cluster ceph... 2026-02-23T11:14:45.076 DEBUG:teuthology.orchestra.run.trial191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config log 1 --format=json 2026-02-23T11:14:45.076 INFO:tasks.daemonwatchdog.daemon_watchdog:watchdog starting 2026-02-23T11:14:45.191 INFO:teuthology.orchestra.run.trial191.stderr:2026-02-23T11:14:45.187+0000 7f20e9fa4640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T11:14:45.193 INFO:teuthology.orchestra.run.trial191.stderr:2026-02-23T11:14:45.192+0000 7f20e9fa4640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T11:14:45.303 INFO:teuthology.orchestra.run.trial191.stdout: 2026-02-23T11:14:45.312 INFO:teuthology.orchestra.run.trial191.stdout:[{"version":2,"timestamp":"2026-02-23T11:14:25.410216+0000","name":"","changes":[{"name":"mgr/mgr_pool","new_value":"false"}]}] 2026-02-23T11:14:45.313 INFO:tasks.ceph_manager:config epoch is 2 2026-02-23T11:14:45.313 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2026-02-23T11:14:45.313 INFO:tasks.ceph.ceph_manager.ceph:waiting for mgr available 2026-02-23T11:14:45.313 DEBUG:teuthology.orchestra.run.trial191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr dump --format=json 2026-02-23T11:14:45.388 INFO:teuthology.orchestra.run.trial191.stderr:2026-02-23T11:14:45.385+0000 7f17b8da4640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T11:14:45.391 INFO:teuthology.orchestra.run.trial191.stderr:2026-02-23T11:14:45.389+0000 7f17b8da4640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T11:14:45.537 INFO:teuthology.orchestra.run.trial191.stdout: 2026-02-23T11:14:45.546 INFO:teuthology.orchestra.run.trial191.stdout:{"epoch":4,"flags":0,"active_gid":4107,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.191:6800","nonce":3819700348},{"type":"v1","addr":"10.20.193.191:6801","nonce":3819700348}]},"active_addr":"10.20.193.191:6801/3819700348","active_change":"2026-02-23T11:14:27.521544+0000","active_mgr_features":4544132024016699391,"available":true,"standbys":[],"modules":["iostat","nfs"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to, use commas to separate multiple","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"update_pg_upmap_activity":{"name":"update_pg_upmap_activity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Updates pg_upmap activity stats to be used in `balancer status detail`","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often the agent on each host will gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"The first TCP port the agent will try to bind to. Up to 1000 subsequent ports will be attempted if this port cannot be bound","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this option can facilitate debugging daemons that encounter runtime problems.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"How frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"Fraction of total system memory to divide among autotuned daemons. Converged systems hosting both Ceph and compute should set a lower value; systems with more than 10 GiB of memory per OSD may benefit from a slightly larger value so that the abundance of RAM can be fully utilized","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command persistent logging","long_desc":"","tags":[],"see_also":[]},"certificate_automated_rotation_enabled":{"name":"certificate_automated_rotation_enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"This flag controls whether cephadm automatically rotates certificates upon expiration.","long_desc":"","tags":[],"see_also":[]},"certificate_check_debug_mode":{"name":"certificate_check_debug_mode","type":"bool","level":"dev","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"FOR TESTING ONLY: This flag forces the certificate check instead of waiting for certificate_check_period.","long_desc":"","tags":[],"see_also":[]},"certificate_check_period":{"name":"certificate_check_period","type":"int","level":"advanced","flags":0,"default_value":"1","min":"0","max":"30","enum_allowed":[],"desc":"Specifies how often (in days) the certificate should be checked for validity.","long_desc":"","tags":[],"see_also":[]},"certificate_duration_days":{"name":"certificate_duration_days","type":"int","level":"advanced","flags":0,"default_value":"1095","min":"90","max":"3650","enum_allowed":[],"desc":"Specifies the duration of self-signed certificates generated and signed by the cephadm root CA","long_desc":"","tags":[],"see_also":[]},"certificate_renewal_threshold_days":{"name":"certificate_renewal_threshold_days","type":"int","level":"advanced","flags":0,"default_value":"30","min":"10","max":"90","enum_allowed":[],"desc":"Specifies the lead time in days to initiate certificate renewal before expiration.","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently Podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.28.1","min":"","max":"","enum_allowed":[],"desc":"Alertmanager container image","long_desc":"","tags":[],"see_also":[]},"container_image_alloy":{"name":"container_image_alloy","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/alloy:latest","min":"","max":"","enum_allowed":[],"desc":"Alloy container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"Elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:12.3.1","min":"","max":"","enum_allowed":[],"desc":"Grafana container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"Haproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_nginx":{"name":"container_image_nginx","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nginx:sclorg-nginx-126","min":"","max":"","enum_allowed":[],"desc":"Nginx container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.9.1","min":"","max":"","enum_allowed":[],"desc":"Node exporter container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.6","min":"","max":"","enum_allowed":[],"desc":"Nvmeof container image","long_desc":"","tags":[],"see_also":[]},"container_image_oauth2_proxy":{"name":"container_image_oauth2_proxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/oauth2-proxy/oauth2-proxy:v7.6.0","min":"","max":"","enum_allowed":[],"desc":"Oauth2 proxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v3.6.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba":{"name":"container_image_samba","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-server:devbuilds-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba_metrics":{"name":"container_image_samba_metrics","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-metrics:devbuilds-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba metrics container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"Snmp gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"Seconds for which to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"quay.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"Seconds for which to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"Seconds for which to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"Location of dashboards to include in Grafana","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"How frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether the hardware monitoring daemon be deployed on every host?","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at the debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"Max number of daemons per service per host: guard against mistakes and runaways","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"Max number of OSDs that will be drained simultaneously when OSDs are removed","long_desc":"","tags":[],"see_also":[]},"max_parallel_osd_upgrades":{"name":"max_parallel_osd_upgrades","type":"int","level":"advanced","flags":0,"default_value":"16","min":"","max":"","enum_allowed":[],"desc":"Maximum number of OSD daemons upgraded in parallel.","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"Mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default IP address for RedFish API (OOB management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"Location of alerts to include in Prometheus Alertmanager","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry URL for login purposes. This is not the default registry.","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"Cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Customized SSH config file to distribute to managed hosts","long_desc":"","tags":[],"see_also":[]},"ssh_keepalive_count_max":{"name":"ssh_keepalive_count_max","type":"int","level":"advanced","flags":0,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"How many sequential SSH connection liveness check failures before the host is marked offline","long_desc":"","tags":[],"see_also":[]},"ssh_keepalive_interval":{"name":"ssh_keepalive_interval","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"How often SSH connections are checked for liveness","long_desc":"","tags":[],"see_also":[]},"stray_daemon_check_interval":{"name":"stray_daemon_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"How frequently cephadm should check for the presence of stray daemons","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use the cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest to ensure that all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MANAGED_BY_CLUSTERS":{"name":"MANAGED_BY_CLUSTERS","type":"str","level":"advanced","flags":0,"default_value":"[]","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MULTICLUSTER_CONFIG":{"name":"MULTICLUSTER_CONFIG","type":"str","level":"advanced","flags":0,"default_value":"{}","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROM_ALERT_CREDENTIAL_CACHE_TTL":{"name":"PROM_ALERT_CREDENTIAL_CACHE_TTL","type":"int","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_HOSTNAME_PER_DAEMON":{"name":"RGW_HOSTNAME_PER_DAEMON","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crypto_caller":{"name":"crypto_caller","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sso_oauth2":{"name":"sso_oauth2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"healthcheck_history_max_entries":{"name":"healthcheck_history_max_entries","type":"int","level":"advanced","flags":1,"default_value":"1000","min":"","max":"","enum_allowed":[],"desc":"Maximum number of health check history entries to keep","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"secondary_zone_period_retry_limit":{"name":"secondary_zone_period_retry_limit","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"RGW module period update retry limit for secondary site","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"prometheus_tls_secret_name":{"name":"prometheus_tls_secret_name","type":"str","level":"advanced","flags":0,"default_value":"rook-ceph-prometheus-server-tls","min":"","max":"","enum_allowed":[],"desc":"name of tls secret in k8s for prometheus","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"smb","can_run":true,"error_string":"","module_options":{"internal_store_backend":{"name":"internal_store_backend","type":"str","level":"dev","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"set internal store backend. for develoment and testing only","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"update_orchestration":{"name":"update_orchestration","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically update orchestration when smb resources are changed","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"pause_cloning":{"name":"pause_cloning","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Pause asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"pause_purging":{"name":"pause_purging","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Pause asynchronous subvolume purge threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"squid":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"tentacle":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"force_disabled_modules":{},"last_failure_osd_epoch":0,"active_clients":[{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"10.20.193.191:0","nonce":3942611698}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"10.20.193.191:0","nonce":722117300}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"10.20.193.191:0","nonce":807472991}]}]} 2026-02-23T11:14:45.550 INFO:tasks.ceph.ceph_manager.ceph:mgr available! 2026-02-23T11:14:45.550 INFO:tasks.ceph.ceph_manager.ceph:waiting for all up 2026-02-23T11:14:45.550 DEBUG:teuthology.orchestra.run.trial191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2026-02-23T11:14:45.628 INFO:teuthology.orchestra.run.trial191.stderr:2026-02-23T11:14:45.624+0000 7f43ba10b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T11:14:45.631 INFO:teuthology.orchestra.run.trial191.stderr:2026-02-23T11:14:45.629+0000 7f43ba10b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T11:14:45.744 INFO:teuthology.orchestra.run.trial191.stdout: 2026-02-23T11:14:45.744 INFO:teuthology.orchestra.run.trial191.stdout:{"epoch":14,"fsid":"408068a4-9c4e-43c3-a1bd-e489279d442d","created":"2026-02-23T11:14:25.368540+0000","modified":"2026-02-23T11:14:45.044401+0000","last_up_change":"2026-02-23T11:14:36.956000+0000","last_in_change":"2026-02-23T11:14:27.217466+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":"rbd","create_time":"2026-02-23T11:14:41.181595+0000","flags":270337,"flags_names":"hashpspool,selfmanaged_snaps,crimson","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"14","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":2,"snap_epoch":14,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"e4f36d13-e8b8-4777-85ed-8210652f197c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":11,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.191:6802","nonce":3272030818}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.191:6803","nonce":3272030818}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.191:6806","nonce":3272030818}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.191:6809","nonce":3272030818}]},"public_addr":"10.20.193.191:6802/3272030818","cluster_addr":"10.20.193.191:6803/3272030818","heartbeat_back_addr":"10.20.193.191:6806/3272030818","heartbeat_front_addr":"10.20.193.191:6809/3272030818","state":["exists","up"]},{"osd":1,"uuid":"e39a1968-9db1-42d6-b792-e3aa32109d97","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":10,"up_thru":11,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.191:6808","nonce":2597130977}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.191:6807","nonce":2597130977}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.191:6811","nonce":2597130977}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.191:6810","nonce":2597130977}]},"public_addr":"10.20.193.191:6808/2597130977","cluster_addr":"10.20.193.191:6807/2597130977","heartbeat_back_addr":"10.20.193.191:6811/2597130977","heartbeat_front_addr":"10.20.193.191:6810/2597130977","state":["exists","up"]},{"osd":2,"uuid":"09f145f1-9abb-4692-b128-bf1aff6fe492","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":10,"up_thru":11,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.191:6805","nonce":3503848635}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.191:6804","nonce":3503848635}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.191:6813","nonce":3503848635}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.191:6812","nonce":3503848635}]},"public_addr":"10.20.193.191:6805/3503848635","cluster_addr":"10.20.193.191:6804/3503848635","heartbeat_back_addr":"10.20.193.191:6813/3503848635","heartbeat_front_addr":"10.20.193.191:6812/3503848635","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":1,"snaps":[{"begin":2,"length":1}]}],"new_removed_snaps":[{"pool":1,"snaps":[{"begin":2,"length":1}]}],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T11:14:45.754 INFO:tasks.ceph.ceph_manager.ceph:all up! 2026-02-23T11:14:45.755 DEBUG:teuthology.orchestra.run.trial191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2026-02-23T11:14:45.831 INFO:teuthology.orchestra.run.trial191.stderr:2026-02-23T11:14:45.827+0000 7f5129993640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T11:14:45.834 INFO:teuthology.orchestra.run.trial191.stderr:2026-02-23T11:14:45.832+0000 7f5129993640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T11:14:45.949 INFO:teuthology.orchestra.run.trial191.stdout: 2026-02-23T11:14:45.949 INFO:teuthology.orchestra.run.trial191.stdout:{"epoch":14,"fsid":"408068a4-9c4e-43c3-a1bd-e489279d442d","created":"2026-02-23T11:14:25.368540+0000","modified":"2026-02-23T11:14:45.044401+0000","last_up_change":"2026-02-23T11:14:36.956000+0000","last_in_change":"2026-02-23T11:14:27.217466+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":"rbd","create_time":"2026-02-23T11:14:41.181595+0000","flags":270337,"flags_names":"hashpspool,selfmanaged_snaps,crimson","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"14","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":2,"snap_epoch":14,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"e4f36d13-e8b8-4777-85ed-8210652f197c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":11,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.191:6802","nonce":3272030818}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.191:6803","nonce":3272030818}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.191:6806","nonce":3272030818}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.191:6809","nonce":3272030818}]},"public_addr":"10.20.193.191:6802/3272030818","cluster_addr":"10.20.193.191:6803/3272030818","heartbeat_back_addr":"10.20.193.191:6806/3272030818","heartbeat_front_addr":"10.20.193.191:6809/3272030818","state":["exists","up"]},{"osd":1,"uuid":"e39a1968-9db1-42d6-b792-e3aa32109d97","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":10,"up_thru":11,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.191:6808","nonce":2597130977}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.191:6807","nonce":2597130977}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.191:6811","nonce":2597130977}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.191:6810","nonce":2597130977}]},"public_addr":"10.20.193.191:6808/2597130977","cluster_addr":"10.20.193.191:6807/2597130977","heartbeat_back_addr":"10.20.193.191:6811/2597130977","heartbeat_front_addr":"10.20.193.191:6810/2597130977","state":["exists","up"]},{"osd":2,"uuid":"09f145f1-9abb-4692-b128-bf1aff6fe492","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":10,"up_thru":11,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.191:6805","nonce":3503848635}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.191:6804","nonce":3503848635}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.191:6813","nonce":3503848635}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.191:6812","nonce":3503848635}]},"public_addr":"10.20.193.191:6805/3503848635","cluster_addr":"10.20.193.191:6804/3503848635","heartbeat_back_addr":"10.20.193.191:6813/3503848635","heartbeat_front_addr":"10.20.193.191:6812/3503848635","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":1,"snaps":[{"begin":2,"length":1}]}],"new_removed_snaps":[{"pool":1,"snaps":[{"begin":2,"length":1}]}],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T11:14:45.960 DEBUG:teuthology.orchestra.run.trial191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2026-02-23T11:14:45.961 DEBUG:teuthology.orchestra.run.trial191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2026-02-23T11:14:45.961 DEBUG:teuthology.orchestra.run.trial191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2026-02-23T11:14:46.042 INFO:teuthology.orchestra.run.trial191.stderr:2026-02-23T11:14:46.037+0000 7f1966393640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T11:14:46.044 INFO:teuthology.orchestra.run.trial191.stderr:2026-02-23T11:14:46.039+0000 7f559ea5a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T11:14:46.044 INFO:teuthology.orchestra.run.trial191.stderr:2026-02-23T11:14:46.039+0000 7f8f00ecb640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T11:14:46.045 INFO:teuthology.orchestra.run.trial191.stderr:2026-02-23T11:14:46.043+0000 7f1966393640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T11:14:46.046 INFO:teuthology.orchestra.run.trial191.stderr:2026-02-23T11:14:46.045+0000 7f559ea5a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T11:14:46.047 INFO:teuthology.orchestra.run.trial191.stderr:2026-02-23T11:14:46.045+0000 7f8f00ecb640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T11:14:46.063 INFO:teuthology.orchestra.run.trial191.stdout:42949672969 2026-02-23T11:14:46.064 DEBUG:teuthology.orchestra.run.trial191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2026-02-23T11:14:46.065 INFO:teuthology.orchestra.run.trial191.stdout:42949672969 2026-02-23T11:14:46.066 DEBUG:teuthology.orchestra.run.trial191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2026-02-23T11:14:46.066 INFO:teuthology.orchestra.run.trial191.stdout:38654705674 2026-02-23T11:14:46.067 DEBUG:teuthology.orchestra.run.trial191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2026-02-23T11:14:46.184 INFO:teuthology.orchestra.run.trial191.stderr:2026-02-23T11:14:46.180+0000 7f63e7d5c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T11:14:46.185 INFO:teuthology.orchestra.run.trial191.stderr:2026-02-23T11:14:46.180+0000 7f43ca58c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T11:14:46.186 INFO:teuthology.orchestra.run.trial191.stderr:2026-02-23T11:14:46.181+0000 7fdf62b6f640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T11:14:46.187 INFO:teuthology.orchestra.run.trial191.stderr:2026-02-23T11:14:46.186+0000 7f43ca58c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T11:14:46.188 INFO:teuthology.orchestra.run.trial191.stderr:2026-02-23T11:14:46.186+0000 7f63e7d5c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T11:14:46.190 INFO:teuthology.orchestra.run.trial191.stderr:2026-02-23T11:14:46.188+0000 7fdf62b6f640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T11:14:46.300 INFO:teuthology.orchestra.run.trial191.stdout:42949672964 2026-02-23T11:14:46.302 INFO:teuthology.orchestra.run.trial191.stdout:38654705668 2026-02-23T11:14:46.305 INFO:teuthology.orchestra.run.trial191.stdout:42949672964 2026-02-23T11:14:46.311 INFO:tasks.ceph.ceph_manager.ceph:need seq 42949672969 got 42949672964 for osd.1 2026-02-23T11:14:46.313 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654705674 got 38654705668 for osd.0 2026-02-23T11:14:46.315 INFO:tasks.ceph.ceph_manager.ceph:need seq 42949672969 got 42949672964 for osd.2 2026-02-23T11:14:47.312 DEBUG:teuthology.orchestra.run.trial191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2026-02-23T11:14:47.313 DEBUG:teuthology.orchestra.run.trial191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2026-02-23T11:14:47.315 DEBUG:teuthology.orchestra.run.trial191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2026-02-23T11:14:47.393 INFO:teuthology.orchestra.run.trial191.stderr:2026-02-23T11:14:47.388+0000 7f9465db8640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T11:14:47.395 INFO:teuthology.orchestra.run.trial191.stderr:2026-02-23T11:14:47.391+0000 7f4c24b9c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T11:14:47.396 INFO:teuthology.orchestra.run.trial191.stderr:2026-02-23T11:14:47.394+0000 7f9465db8640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T11:14:47.398 INFO:teuthology.orchestra.run.trial191.stderr:2026-02-23T11:14:47.396+0000 7f4c24b9c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T11:14:47.433 INFO:teuthology.orchestra.run.trial191.stderr:2026-02-23T11:14:47.428+0000 7eff8e9ad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T11:14:47.435 INFO:teuthology.orchestra.run.trial191.stderr:2026-02-23T11:14:47.434+0000 7eff8e9ad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T11:14:47.509 INFO:teuthology.orchestra.run.trial191.stdout:38654705668 2026-02-23T11:14:47.510 INFO:teuthology.orchestra.run.trial191.stdout:42949672964 2026-02-23T11:14:47.521 INFO:tasks.ceph.ceph_manager.ceph:need seq 42949672969 got 42949672964 for osd.1 2026-02-23T11:14:47.522 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654705674 got 38654705668 for osd.0 2026-02-23T11:14:47.549 INFO:teuthology.orchestra.run.trial191.stdout:42949672964 2026-02-23T11:14:47.560 INFO:tasks.ceph.ceph_manager.ceph:need seq 42949672969 got 42949672964 for osd.2 2026-02-23T11:14:48.521 DEBUG:teuthology.orchestra.run.trial191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2026-02-23T11:14:48.522 DEBUG:teuthology.orchestra.run.trial191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2026-02-23T11:14:48.560 DEBUG:teuthology.orchestra.run.trial191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2026-02-23T11:14:48.603 INFO:teuthology.orchestra.run.trial191.stderr:2026-02-23T11:14:48.598+0000 7f93ea3c1640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T11:14:48.604 INFO:teuthology.orchestra.run.trial191.stderr:2026-02-23T11:14:48.598+0000 7fb78c39c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T11:14:48.606 INFO:teuthology.orchestra.run.trial191.stderr:2026-02-23T11:14:48.604+0000 7fb78c39c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T11:14:48.606 INFO:teuthology.orchestra.run.trial191.stderr:2026-02-23T11:14:48.605+0000 7f93ea3c1640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T11:14:48.643 INFO:teuthology.orchestra.run.trial191.stderr:2026-02-23T11:14:48.637+0000 7ff326a8b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T11:14:48.645 INFO:teuthology.orchestra.run.trial191.stderr:2026-02-23T11:14:48.644+0000 7ff326a8b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T11:14:48.719 INFO:teuthology.orchestra.run.trial191.stdout:42949672969 2026-02-23T11:14:48.721 INFO:teuthology.orchestra.run.trial191.stdout:38654705674 2026-02-23T11:14:48.731 INFO:tasks.ceph.ceph_manager.ceph:need seq 42949672969 got 42949672969 for osd.1 2026-02-23T11:14:48.731 DEBUG:teuthology.parallel:result is None 2026-02-23T11:14:48.733 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654705674 got 38654705674 for osd.0 2026-02-23T11:14:48.733 DEBUG:teuthology.parallel:result is None 2026-02-23T11:14:48.759 INFO:teuthology.orchestra.run.trial191.stdout:42949672969 2026-02-23T11:14:48.770 INFO:tasks.ceph.ceph_manager.ceph:need seq 42949672969 got 42949672969 for osd.2 2026-02-23T11:14:48.770 DEBUG:teuthology.parallel:result is None 2026-02-23T11:14:48.770 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2026-02-23T11:14:48.770 DEBUG:teuthology.orchestra.run.trial191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-23T11:14:48.883 INFO:teuthology.orchestra.run.trial191.stderr:2026-02-23T11:14:48.879+0000 7f4848fdc640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T11:14:48.886 INFO:teuthology.orchestra.run.trial191.stderr:2026-02-23T11:14:48.884+0000 7f4848fdc640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T11:14:48.996 INFO:teuthology.orchestra.run.trial191.stdout: 2026-02-23T11:14:48.996 INFO:teuthology.orchestra.run.trial191.stderr:dumped all 2026-02-23T11:14:49.005 INFO:teuthology.orchestra.run.trial191.stdout:{"pg_ready":true,"pg_map":{"version":14,"stamp":"2026-02-23T11:14:47.526258+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":3,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":156893184,"kb_used":804,"kb_used_data":804,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":156892380,"statfs":{"total":160658620416,"available":160657797120,"internally_reserved":0,"allocated":823296,"data_stored":823296,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"0.000000"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":15,"reported_epoch":14,"state":"active+clean","last_fresh":"2026-02-23T11:14:45.060134+0000","last_change":"2026-02-23T11:14:45.066249+0000","last_active":"2026-02-23T11:14:45.060134+0000","last_peered":"2026-02-23T11:14:45.060134+0000","last_clean":"2026-02-23T11:14:45.060134+0000","last_became_active":"2026-02-23T11:14:43.080217+0000","last_became_peered":"2026-02-23T11:14:43.080217+0000","last_unstale":"2026-02-23T11:14:45.060134+0000","last_undegraded":"2026-02-23T11:14:45.060134+0000","last_fullsized":"2026-02-23T11:14:45.060134+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-23T11:14:42.032806+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-23T11:14:42.032806+0000","last_clean_scrub_stamp":"2026-02-23T11:14:42.032806+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":15,"reported_epoch":14,"state":"active+clean","last_fresh":"2026-02-23T11:14:45.057488+0000","last_change":"2026-02-23T11:14:45.061801+0000","last_active":"2026-02-23T11:14:45.057488+0000","last_peered":"2026-02-23T11:14:45.057488+0000","last_clean":"2026-02-23T11:14:45.057488+0000","last_became_active":"2026-02-23T11:14:43.116310+0000","last_became_peered":"2026-02-23T11:14:43.116310+0000","last_unstale":"2026-02-23T11:14:45.057488+0000","last_undegraded":"2026-02-23T11:14:45.057488+0000","last_fullsized":"2026-02-23T11:14:45.057488+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-23T11:14:42.032806+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-23T11:14:42.032806+0000","last_clean_scrub_stamp":"2026-02-23T11:14:42.032806+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":15,"reported_epoch":14,"state":"active+clean","last_fresh":"2026-02-23T11:14:45.060107+0000","last_change":"2026-02-23T11:14:45.062322+0000","last_active":"2026-02-23T11:14:45.060107+0000","last_peered":"2026-02-23T11:14:45.060107+0000","last_clean":"2026-02-23T11:14:45.060107+0000","last_became_active":"2026-02-23T11:14:43.079925+0000","last_became_peered":"2026-02-23T11:14:43.079925+0000","last_unstale":"2026-02-23T11:14:45.060107+0000","last_undegraded":"2026-02-23T11:14:45.060107+0000","last_fullsized":"2026-02-23T11:14:45.060107+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-23T11:14:42.032806+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-23T11:14:42.032806+0000","last_clean_scrub_stamp":"2026-02-23T11:14:42.032806+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":15,"reported_epoch":14,"state":"active+clean","last_fresh":"2026-02-23T11:14:45.059784+0000","last_change":"2026-02-23T11:14:45.066152+0000","last_active":"2026-02-23T11:14:45.059784+0000","last_peered":"2026-02-23T11:14:45.059784+0000","last_clean":"2026-02-23T11:14:45.059784+0000","last_became_active":"2026-02-23T11:14:43.096768+0000","last_became_peered":"2026-02-23T11:14:43.096768+0000","last_unstale":"2026-02-23T11:14:45.059784+0000","last_undegraded":"2026-02-23T11:14:45.059784+0000","last_fullsized":"2026-02-23T11:14:45.059784+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-23T11:14:42.032806+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-23T11:14:42.032806+0000","last_clean_scrub_stamp":"2026-02-23T11:14:42.032806+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":16,"reported_epoch":14,"state":"active+clean","last_fresh":"2026-02-23T11:14:45.060509+0000","last_change":"2026-02-23T11:14:45.065912+0000","last_active":"2026-02-23T11:14:45.060509+0000","last_peered":"2026-02-23T11:14:45.060509+0000","last_clean":"2026-02-23T11:14:45.060509+0000","last_became_active":"2026-02-23T11:14:43.077394+0000","last_became_peered":"2026-02-23T11:14:43.077394+0000","last_unstale":"2026-02-23T11:14:45.060509+0000","last_undegraded":"2026-02-23T11:14:45.060509+0000","last_fullsized":"2026-02-23T11:14:45.060509+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-23T11:14:42.032806+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-23T11:14:42.032806+0000","last_clean_scrub_stamp":"2026-02-23T11:14:42.032806+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"13'1","reported_seq":17,"reported_epoch":14,"state":"active+clean","last_fresh":"2026-02-23T11:14:45.062078+0000","last_change":"2026-02-23T11:14:45.062078+0000","last_active":"2026-02-23T11:14:45.062078+0000","last_peered":"2026-02-23T11:14:45.062078+0000","last_clean":"2026-02-23T11:14:45.062078+0000","last_became_active":"2026-02-23T11:14:43.092837+0000","last_became_peered":"2026-02-23T11:14:43.092837+0000","last_unstale":"2026-02-23T11:14:45.062078+0000","last_undegraded":"2026-02-23T11:14:45.062078+0000","last_fullsized":"2026-02-23T11:14:45.062078+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-23T11:14:42.032806+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-23T11:14:42.032806+0000","last_clean_scrub_stamp":"2026-02-23T11:14:42.032806+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":15,"reported_epoch":14,"state":"active+clean","last_fresh":"2026-02-23T11:14:45.058249+0000","last_change":"2026-02-23T11:14:45.060697+0000","last_active":"2026-02-23T11:14:45.058249+0000","last_peered":"2026-02-23T11:14:45.058249+0000","last_clean":"2026-02-23T11:14:45.058249+0000","last_became_active":"2026-02-23T11:14:43.085371+0000","last_became_peered":"2026-02-23T11:14:43.085371+0000","last_unstale":"2026-02-23T11:14:45.058249+0000","last_undegraded":"2026-02-23T11:14:45.058249+0000","last_fullsized":"2026-02-23T11:14:45.058249+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-23T11:14:42.032806+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-23T11:14:42.032806+0000","last_clean_scrub_stamp":"2026-02-23T11:14:42.032806+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":15,"reported_epoch":14,"state":"active+clean","last_fresh":"2026-02-23T11:14:45.057824+0000","last_change":"2026-02-23T11:14:45.061764+0000","last_active":"2026-02-23T11:14:45.057824+0000","last_peered":"2026-02-23T11:14:45.057824+0000","last_clean":"2026-02-23T11:14:45.057824+0000","last_became_active":"2026-02-23T11:14:43.100854+0000","last_became_peered":"2026-02-23T11:14:43.100854+0000","last_unstale":"2026-02-23T11:14:45.057824+0000","last_undegraded":"2026-02-23T11:14:45.057824+0000","last_fullsized":"2026-02-23T11:14:45.057824+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-23T11:14:42.032806+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-23T11:14:42.032806+0000","last_clean_scrub_stamp":"2026-02-23T11:14:42.032806+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":160658620416,"available":160657797120,"internally_reserved":0,"allocated":823296,"data_stored":823296,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":10,"seq":42949672969,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52297728,"kb_used":244,"kb_used_data":244,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":52297484,"statfs":{"total":53552873472,"available":53552623616,"internally_reserved":0,"allocated":249856,"data_stored":249856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":10,"seq":42949672969,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52297728,"kb_used":280,"kb_used_data":280,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":52297448,"statfs":{"total":53552873472,"available":53552586752,"internally_reserved":0,"allocated":286720,"data_stored":286720,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705674,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52297728,"kb_used":280,"kb_used_data":280,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":52297448,"statfs":{"total":53552873472,"available":53552586752,"internally_reserved":0,"allocated":286720,"data_stored":286720,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":53552873472,"available":53552586752,"internally_reserved":0,"allocated":286720,"data_stored":286720,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":53552873472,"available":53552586752,"internally_reserved":0,"allocated":286720,"data_stored":286720,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":53552873472,"available":53552623616,"internally_reserved":0,"allocated":249856,"data_stored":249856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-23T11:14:49.007 DEBUG:teuthology.orchestra.run.trial191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-23T11:14:49.083 INFO:teuthology.orchestra.run.trial191.stderr:2026-02-23T11:14:49.079+0000 7fe7e925a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T11:14:49.085 INFO:teuthology.orchestra.run.trial191.stderr:2026-02-23T11:14:49.084+0000 7fe7e925a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T11:14:49.196 INFO:teuthology.orchestra.run.trial191.stdout: 2026-02-23T11:14:49.197 INFO:teuthology.orchestra.run.trial191.stderr:dumped all 2026-02-23T11:14:49.206 INFO:teuthology.orchestra.run.trial191.stdout:{"pg_ready":true,"pg_map":{"version":14,"stamp":"2026-02-23T11:14:47.526258+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":3,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":156893184,"kb_used":804,"kb_used_data":804,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":156892380,"statfs":{"total":160658620416,"available":160657797120,"internally_reserved":0,"allocated":823296,"data_stored":823296,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"0.000000"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":15,"reported_epoch":14,"state":"active+clean","last_fresh":"2026-02-23T11:14:45.060134+0000","last_change":"2026-02-23T11:14:45.066249+0000","last_active":"2026-02-23T11:14:45.060134+0000","last_peered":"2026-02-23T11:14:45.060134+0000","last_clean":"2026-02-23T11:14:45.060134+0000","last_became_active":"2026-02-23T11:14:43.080217+0000","last_became_peered":"2026-02-23T11:14:43.080217+0000","last_unstale":"2026-02-23T11:14:45.060134+0000","last_undegraded":"2026-02-23T11:14:45.060134+0000","last_fullsized":"2026-02-23T11:14:45.060134+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-23T11:14:42.032806+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-23T11:14:42.032806+0000","last_clean_scrub_stamp":"2026-02-23T11:14:42.032806+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":15,"reported_epoch":14,"state":"active+clean","last_fresh":"2026-02-23T11:14:45.057488+0000","last_change":"2026-02-23T11:14:45.061801+0000","last_active":"2026-02-23T11:14:45.057488+0000","last_peered":"2026-02-23T11:14:45.057488+0000","last_clean":"2026-02-23T11:14:45.057488+0000","last_became_active":"2026-02-23T11:14:43.116310+0000","last_became_peered":"2026-02-23T11:14:43.116310+0000","last_unstale":"2026-02-23T11:14:45.057488+0000","last_undegraded":"2026-02-23T11:14:45.057488+0000","last_fullsized":"2026-02-23T11:14:45.057488+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-23T11:14:42.032806+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-23T11:14:42.032806+0000","last_clean_scrub_stamp":"2026-02-23T11:14:42.032806+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":15,"reported_epoch":14,"state":"active+clean","last_fresh":"2026-02-23T11:14:45.060107+0000","last_change":"2026-02-23T11:14:45.062322+0000","last_active":"2026-02-23T11:14:45.060107+0000","last_peered":"2026-02-23T11:14:45.060107+0000","last_clean":"2026-02-23T11:14:45.060107+0000","last_became_active":"2026-02-23T11:14:43.079925+0000","last_became_peered":"2026-02-23T11:14:43.079925+0000","last_unstale":"2026-02-23T11:14:45.060107+0000","last_undegraded":"2026-02-23T11:14:45.060107+0000","last_fullsized":"2026-02-23T11:14:45.060107+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-23T11:14:42.032806+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-23T11:14:42.032806+0000","last_clean_scrub_stamp":"2026-02-23T11:14:42.032806+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":15,"reported_epoch":14,"state":"active+clean","last_fresh":"2026-02-23T11:14:45.059784+0000","last_change":"2026-02-23T11:14:45.066152+0000","last_active":"2026-02-23T11:14:45.059784+0000","last_peered":"2026-02-23T11:14:45.059784+0000","last_clean":"2026-02-23T11:14:45.059784+0000","last_became_active":"2026-02-23T11:14:43.096768+0000","last_became_peered":"2026-02-23T11:14:43.096768+0000","last_unstale":"2026-02-23T11:14:45.059784+0000","last_undegraded":"2026-02-23T11:14:45.059784+0000","last_fullsized":"2026-02-23T11:14:45.059784+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-23T11:14:42.032806+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-23T11:14:42.032806+0000","last_clean_scrub_stamp":"2026-02-23T11:14:42.032806+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":16,"reported_epoch":14,"state":"active+clean","last_fresh":"2026-02-23T11:14:45.060509+0000","last_change":"2026-02-23T11:14:45.065912+0000","last_active":"2026-02-23T11:14:45.060509+0000","last_peered":"2026-02-23T11:14:45.060509+0000","last_clean":"2026-02-23T11:14:45.060509+0000","last_became_active":"2026-02-23T11:14:43.077394+0000","last_became_peered":"2026-02-23T11:14:43.077394+0000","last_unstale":"2026-02-23T11:14:45.060509+0000","last_undegraded":"2026-02-23T11:14:45.060509+0000","last_fullsized":"2026-02-23T11:14:45.060509+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-23T11:14:42.032806+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-23T11:14:42.032806+0000","last_clean_scrub_stamp":"2026-02-23T11:14:42.032806+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"13'1","reported_seq":17,"reported_epoch":14,"state":"active+clean","last_fresh":"2026-02-23T11:14:45.062078+0000","last_change":"2026-02-23T11:14:45.062078+0000","last_active":"2026-02-23T11:14:45.062078+0000","last_peered":"2026-02-23T11:14:45.062078+0000","last_clean":"2026-02-23T11:14:45.062078+0000","last_became_active":"2026-02-23T11:14:43.092837+0000","last_became_peered":"2026-02-23T11:14:43.092837+0000","last_unstale":"2026-02-23T11:14:45.062078+0000","last_undegraded":"2026-02-23T11:14:45.062078+0000","last_fullsized":"2026-02-23T11:14:45.062078+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-23T11:14:42.032806+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-23T11:14:42.032806+0000","last_clean_scrub_stamp":"2026-02-23T11:14:42.032806+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":15,"reported_epoch":14,"state":"active+clean","last_fresh":"2026-02-23T11:14:45.058249+0000","last_change":"2026-02-23T11:14:45.060697+0000","last_active":"2026-02-23T11:14:45.058249+0000","last_peered":"2026-02-23T11:14:45.058249+0000","last_clean":"2026-02-23T11:14:45.058249+0000","last_became_active":"2026-02-23T11:14:43.085371+0000","last_became_peered":"2026-02-23T11:14:43.085371+0000","last_unstale":"2026-02-23T11:14:45.058249+0000","last_undegraded":"2026-02-23T11:14:45.058249+0000","last_fullsized":"2026-02-23T11:14:45.058249+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-23T11:14:42.032806+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-23T11:14:42.032806+0000","last_clean_scrub_stamp":"2026-02-23T11:14:42.032806+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":15,"reported_epoch":14,"state":"active+clean","last_fresh":"2026-02-23T11:14:45.057824+0000","last_change":"2026-02-23T11:14:45.061764+0000","last_active":"2026-02-23T11:14:45.057824+0000","last_peered":"2026-02-23T11:14:45.057824+0000","last_clean":"2026-02-23T11:14:45.057824+0000","last_became_active":"2026-02-23T11:14:43.100854+0000","last_became_peered":"2026-02-23T11:14:43.100854+0000","last_unstale":"2026-02-23T11:14:45.057824+0000","last_undegraded":"2026-02-23T11:14:45.057824+0000","last_fullsized":"2026-02-23T11:14:45.057824+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-23T11:14:42.032806+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-23T11:14:42.032806+0000","last_clean_scrub_stamp":"2026-02-23T11:14:42.032806+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":160658620416,"available":160657797120,"internally_reserved":0,"allocated":823296,"data_stored":823296,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":10,"seq":42949672969,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52297728,"kb_used":244,"kb_used_data":244,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":52297484,"statfs":{"total":53552873472,"available":53552623616,"internally_reserved":0,"allocated":249856,"data_stored":249856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":10,"seq":42949672969,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52297728,"kb_used":280,"kb_used_data":280,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":52297448,"statfs":{"total":53552873472,"available":53552586752,"internally_reserved":0,"allocated":286720,"data_stored":286720,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705674,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52297728,"kb_used":280,"kb_used_data":280,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":52297448,"statfs":{"total":53552873472,"available":53552586752,"internally_reserved":0,"allocated":286720,"data_stored":286720,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":53552873472,"available":53552586752,"internally_reserved":0,"allocated":286720,"data_stored":286720,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":53552873472,"available":53552586752,"internally_reserved":0,"allocated":286720,"data_stored":286720,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":53552873472,"available":53552623616,"internally_reserved":0,"allocated":249856,"data_stored":249856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-23T11:14:49.208 INFO:tasks.ceph.ceph_manager.ceph:clean! 2026-02-23T11:14:49.208 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2026-02-23T11:14:49.208 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy 2026-02-23T11:14:49.208 DEBUG:teuthology.orchestra.run.trial191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2026-02-23T11:14:49.282 INFO:teuthology.orchestra.run.trial191.stderr:2026-02-23T11:14:49.279+0000 7f01804cb640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T11:14:49.285 INFO:teuthology.orchestra.run.trial191.stderr:2026-02-23T11:14:49.284+0000 7f01804cb640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T11:14:49.425 INFO:teuthology.orchestra.run.trial191.stdout: 2026-02-23T11:14:49.426 INFO:teuthology.orchestra.run.trial191.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2026-02-23T11:14:49.435 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy done 2026-02-23T11:14:49.435 INFO:teuthology.run_tasks:Running task workunit... 2026-02-23T11:14:49.463 INFO:tasks.workunit:Pulling workunits from ref e371433fcd1390a0b82cfb18effc422c1f5d8c1d 2026-02-23T11:14:49.463 INFO:tasks.workunit:Making a separate scratch dir for every client... 2026-02-23T11:14:49.464 DEBUG:teuthology.orchestra.run.trial191:> stat -- /home/ubuntu/cephtest/mnt.0 2026-02-23T11:14:49.480 DEBUG:teuthology.orchestra.run:got remote process result: 1 2026-02-23T11:14:49.480 INFO:teuthology.orchestra.run.trial191.stderr:stat: cannot statx '/home/ubuntu/cephtest/mnt.0': No such file or directory 2026-02-23T11:14:49.481 DEBUG:teuthology.orchestra.run.trial191:> mkdir -- /home/ubuntu/cephtest/mnt.0 2026-02-23T11:14:49.537 INFO:tasks.workunit:Created dir /home/ubuntu/cephtest/mnt.0 2026-02-23T11:14:49.537 DEBUG:teuthology.orchestra.run.trial191:> cd -- /home/ubuntu/cephtest/mnt.0 && mkdir -- client.0 2026-02-23T11:14:49.593 INFO:tasks.workunit:timeout=3h 2026-02-23T11:14:49.593 INFO:tasks.workunit:cleanup=True 2026-02-23T11:14:49.593 DEBUG:teuthology.orchestra.run.trial191:> rm -rf /home/ubuntu/cephtest/clone.client.0 && git clone https://github.com/shraddhaag/ceph.git /home/ubuntu/cephtest/clone.client.0 && cd /home/ubuntu/cephtest/clone.client.0 && git checkout e371433fcd1390a0b82cfb18effc422c1f5d8c1d 2026-02-23T11:14:49.656 INFO:tasks.workunit.client.0.trial191.stderr:Cloning into '/home/ubuntu/cephtest/clone.client.0'... 2026-02-23T11:15:11.357 INFO:tasks.workunit.client.0.trial191.stderr:Note: switching to 'e371433fcd1390a0b82cfb18effc422c1f5d8c1d'. 2026-02-23T11:15:11.358 INFO:tasks.workunit.client.0.trial191.stderr: 2026-02-23T11:15:11.358 INFO:tasks.workunit.client.0.trial191.stderr:You are in 'detached HEAD' state. You can look around, make experimental 2026-02-23T11:15:11.358 INFO:tasks.workunit.client.0.trial191.stderr:changes and commit them, and you can discard any commits you make in this 2026-02-23T11:15:11.358 INFO:tasks.workunit.client.0.trial191.stderr:state without impacting any branches by switching back to a branch. 2026-02-23T11:15:11.358 INFO:tasks.workunit.client.0.trial191.stderr: 2026-02-23T11:15:11.358 INFO:tasks.workunit.client.0.trial191.stderr:If you want to create a new branch to retain commits you create, you may 2026-02-23T11:15:11.358 INFO:tasks.workunit.client.0.trial191.stderr:do so (now or later) by using -c with the switch command. Example: 2026-02-23T11:15:11.358 INFO:tasks.workunit.client.0.trial191.stderr: 2026-02-23T11:15:11.358 INFO:tasks.workunit.client.0.trial191.stderr: git switch -c 2026-02-23T11:15:11.358 INFO:tasks.workunit.client.0.trial191.stderr: 2026-02-23T11:15:11.359 INFO:tasks.workunit.client.0.trial191.stderr:Or undo this operation with: 2026-02-23T11:15:11.359 INFO:tasks.workunit.client.0.trial191.stderr: 2026-02-23T11:15:11.359 INFO:tasks.workunit.client.0.trial191.stderr: git switch - 2026-02-23T11:15:11.359 INFO:tasks.workunit.client.0.trial191.stderr: 2026-02-23T11:15:11.359 INFO:tasks.workunit.client.0.trial191.stderr:Turn off this advice by setting config variable advice.detachedHead to false 2026-02-23T11:15:11.359 INFO:tasks.workunit.client.0.trial191.stderr: 2026-02-23T11:15:11.359 INFO:tasks.workunit.client.0.trial191.stderr:HEAD is now at e371433fcd1 qa/suites/crimson-rados: enable cephadm tests 2026-02-23T11:15:11.360 DEBUG:teuthology.orchestra.run.trial191:> cd -- /home/ubuntu/cephtest/clone.client.0/qa/workunits && if test -e Makefile ; then make ; fi && find -executable -type f -printf '%P\0' >/home/ubuntu/cephtest/workunits.list.client.0 2026-02-23T11:15:11.418 INFO:tasks.workunit.client.0.trial191.stdout:for d in direct_io fs ; do ( cd $d ; make all ) ; done 2026-02-23T11:15:11.420 INFO:tasks.workunit.client.0.trial191.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2026-02-23T11:15:11.420 INFO:tasks.workunit.client.0.trial191.stdout:cc -Wall -Wextra -D_GNU_SOURCE direct_io_test.c -o direct_io_test 2026-02-23T11:15:11.508 INFO:tasks.workunit.client.0.trial191.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_sync_io.c -o test_sync_io 2026-02-23T11:15:11.539 INFO:tasks.workunit.client.0.trial191.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_short_dio_read.c -o test_short_dio_read 2026-02-23T11:15:11.564 INFO:tasks.workunit.client.0.trial191.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2026-02-23T11:15:11.565 INFO:tasks.workunit.client.0.trial191.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2026-02-23T11:15:11.565 INFO:tasks.workunit.client.0.trial191.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_o_trunc.c -o test_o_trunc 2026-02-23T11:15:11.590 INFO:tasks.workunit.client.0.trial191.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2026-02-23T11:15:11.594 DEBUG:teuthology.orchestra.run.trial191:> set -ex 2026-02-23T11:15:11.594 DEBUG:teuthology.orchestra.run.trial191:> dd if=/home/ubuntu/cephtest/workunits.list.client.0 of=/dev/stdout 2026-02-23T11:15:11.654 INFO:tasks.workunit:Running workunits matching cephtool/test_crimson_dump_metrics.sh on client.0... 2026-02-23T11:15:11.656 INFO:tasks.workunit:Running workunit cephtool/test_crimson_dump_metrics.sh... 2026-02-23T11:15:11.656 DEBUG:teuthology.orchestra.run.trial191:workunit test cephtool/test_crimson_dump_metrics.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=e371433fcd1390a0b82cfb18effc422c1f5d8c1d TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 CRIMSON_COMPAT=1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test_crimson_dump_metrics.sh 2026-02-23T11:15:11.716 INFO:tasks.workunit.client.0.trial191.stderr:stty: 'standard input': Inappropriate ioctl for device 2026-02-23T11:15:11.719 INFO:tasks.workunit.client.0.trial191.stderr:+ test_dump_metrics 2026-02-23T11:15:11.719 INFO:tasks.workunit.client.0.trial191.stderr:+ ceph tell osd.0 dump_metrics 2026-02-23T11:15:11.829 INFO:tasks.workunit.client.0.trial191.stderr:+ ceph tell osd.0 dump_metrics reactor_utilization 2026-02-23T11:15:11.910 INFO:tasks.workunit.client.0.trial191.stderr:++ grep -c reactor_utilization /tmp/dump_metrics_full.json 2026-02-23T11:15:11.911 INFO:tasks.workunit.client.0.trial191.stderr:+ full_count=2 2026-02-23T11:15:11.912 INFO:tasks.workunit.client.0.trial191.stderr:++ grep -c reactor_utilization /tmp/dump_metrics_reactor.json 2026-02-23T11:15:11.913 INFO:tasks.workunit.client.0.trial191.stderr:+ reactor_count=2 2026-02-23T11:15:11.913 INFO:tasks.workunit.client.0.trial191.stderr:+ '[' 2 -gt 0 ']' 2026-02-23T11:15:11.913 INFO:tasks.workunit.client.0.trial191.stderr:+ '[' 2 -gt 0 ']' 2026-02-23T11:15:11.913 INFO:tasks.workunit.client.0.trial191.stderr:+ '[' 2 -eq 2 ']' 2026-02-23T11:15:11.913 INFO:tasks.workunit.client.0.trial191.stderr:+ rm -f /tmp/dump_metrics_full.json /tmp/dump_metrics_reactor.json 2026-02-23T11:15:11.914 INFO:tasks.workunit.client.0.trial191.stderr:+ return 0 2026-02-23T11:15:11.914 INFO:tasks.workunit.client.0.trial191.stderr:+ echo OK 2026-02-23T11:15:11.915 INFO:tasks.workunit.client.0.trial191.stdout:OK 2026-02-23T11:15:11.915 INFO:teuthology.orchestra.run:Running command with timeout 3600 2026-02-23T11:15:11.916 DEBUG:teuthology.orchestra.run.trial191:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2026-02-23T11:15:11.981 INFO:tasks.workunit:Stopping ['cephtool/test_crimson_dump_metrics.sh'] on client.0... 2026-02-23T11:15:11.981 DEBUG:teuthology.orchestra.run.trial191:> sudo rm -rf -- /home/ubuntu/cephtest/workunits.list.client.0 /home/ubuntu/cephtest/clone.client.0 2026-02-23T11:15:12.322 DEBUG:teuthology.parallel:result is None 2026-02-23T11:15:12.323 DEBUG:teuthology.orchestra.run.trial191:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0 2026-02-23T11:15:12.348 INFO:tasks.workunit:Deleted dir /home/ubuntu/cephtest/mnt.0/client.0 2026-02-23T11:15:12.348 DEBUG:teuthology.orchestra.run.trial191:> rmdir -- /home/ubuntu/cephtest/mnt.0 2026-02-23T11:15:12.405 INFO:tasks.workunit:Deleted artificial mount point /home/ubuntu/cephtest/mnt.0/client.0 2026-02-23T11:15:12.405 DEBUG:teuthology.run_tasks:Unwinding manager ceph 2026-02-23T11:15:12.421 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2026-02-23T11:15:12.421 DEBUG:teuthology.orchestra.run.trial191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-23T11:15:12.522 INFO:teuthology.orchestra.run.trial191.stderr:2026-02-23T11:15:12.518+0000 7f428210b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T11:15:12.524 INFO:teuthology.orchestra.run.trial191.stderr:2026-02-23T11:15:12.523+0000 7f428210b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T11:15:12.636 INFO:teuthology.orchestra.run.trial191.stdout: 2026-02-23T11:15:12.636 INFO:teuthology.orchestra.run.trial191.stderr:dumped all 2026-02-23T11:15:12.645 INFO:teuthology.orchestra.run.trial191.stdout:{"pg_ready":true,"pg_map":{"version":26,"stamp":"2026-02-23T11:15:11.529222+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":3,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":156893184,"kb_used":804,"kb_used_data":804,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":156892380,"statfs":{"total":160658620416,"available":160657797120,"internally_reserved":0,"allocated":823296,"data_stored":823296,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001739"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":15,"reported_epoch":14,"state":"active+clean","last_fresh":"2026-02-23T11:14:45.060134+0000","last_change":"2026-02-23T11:14:45.066249+0000","last_active":"2026-02-23T11:14:45.060134+0000","last_peered":"2026-02-23T11:14:45.060134+0000","last_clean":"2026-02-23T11:14:45.060134+0000","last_became_active":"2026-02-23T11:14:43.080217+0000","last_became_peered":"2026-02-23T11:14:43.080217+0000","last_unstale":"2026-02-23T11:14:45.060134+0000","last_undegraded":"2026-02-23T11:14:45.060134+0000","last_fullsized":"2026-02-23T11:14:45.060134+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-23T11:14:42.032806+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-23T11:14:42.032806+0000","last_clean_scrub_stamp":"2026-02-23T11:14:42.032806+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":15,"reported_epoch":14,"state":"active+clean","last_fresh":"2026-02-23T11:14:45.057488+0000","last_change":"2026-02-23T11:14:45.061801+0000","last_active":"2026-02-23T11:14:45.057488+0000","last_peered":"2026-02-23T11:14:45.057488+0000","last_clean":"2026-02-23T11:14:45.057488+0000","last_became_active":"2026-02-23T11:14:43.116310+0000","last_became_peered":"2026-02-23T11:14:43.116310+0000","last_unstale":"2026-02-23T11:14:45.057488+0000","last_undegraded":"2026-02-23T11:14:45.057488+0000","last_fullsized":"2026-02-23T11:14:45.057488+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-23T11:14:42.032806+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-23T11:14:42.032806+0000","last_clean_scrub_stamp":"2026-02-23T11:14:42.032806+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":15,"reported_epoch":14,"state":"active+clean","last_fresh":"2026-02-23T11:14:45.060107+0000","last_change":"2026-02-23T11:14:45.062322+0000","last_active":"2026-02-23T11:14:45.060107+0000","last_peered":"2026-02-23T11:14:45.060107+0000","last_clean":"2026-02-23T11:14:45.060107+0000","last_became_active":"2026-02-23T11:14:43.079925+0000","last_became_peered":"2026-02-23T11:14:43.079925+0000","last_unstale":"2026-02-23T11:14:45.060107+0000","last_undegraded":"2026-02-23T11:14:45.060107+0000","last_fullsized":"2026-02-23T11:14:45.060107+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-23T11:14:42.032806+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-23T11:14:42.032806+0000","last_clean_scrub_stamp":"2026-02-23T11:14:42.032806+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":15,"reported_epoch":14,"state":"active+clean","last_fresh":"2026-02-23T11:14:45.059784+0000","last_change":"2026-02-23T11:14:45.066152+0000","last_active":"2026-02-23T11:14:45.059784+0000","last_peered":"2026-02-23T11:14:45.059784+0000","last_clean":"2026-02-23T11:14:45.059784+0000","last_became_active":"2026-02-23T11:14:43.096768+0000","last_became_peered":"2026-02-23T11:14:43.096768+0000","last_unstale":"2026-02-23T11:14:45.059784+0000","last_undegraded":"2026-02-23T11:14:45.059784+0000","last_fullsized":"2026-02-23T11:14:45.059784+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-23T11:14:42.032806+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-23T11:14:42.032806+0000","last_clean_scrub_stamp":"2026-02-23T11:14:42.032806+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":16,"reported_epoch":14,"state":"active+clean","last_fresh":"2026-02-23T11:14:45.060509+0000","last_change":"2026-02-23T11:14:45.065912+0000","last_active":"2026-02-23T11:14:45.060509+0000","last_peered":"2026-02-23T11:14:45.060509+0000","last_clean":"2026-02-23T11:14:45.060509+0000","last_became_active":"2026-02-23T11:14:43.077394+0000","last_became_peered":"2026-02-23T11:14:43.077394+0000","last_unstale":"2026-02-23T11:14:45.060509+0000","last_undegraded":"2026-02-23T11:14:45.060509+0000","last_fullsized":"2026-02-23T11:14:45.060509+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-23T11:14:42.032806+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-23T11:14:42.032806+0000","last_clean_scrub_stamp":"2026-02-23T11:14:42.032806+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"13'1","reported_seq":17,"reported_epoch":14,"state":"active+clean","last_fresh":"2026-02-23T11:14:45.062078+0000","last_change":"2026-02-23T11:14:45.062078+0000","last_active":"2026-02-23T11:14:45.062078+0000","last_peered":"2026-02-23T11:14:45.062078+0000","last_clean":"2026-02-23T11:14:45.062078+0000","last_became_active":"2026-02-23T11:14:43.092837+0000","last_became_peered":"2026-02-23T11:14:43.092837+0000","last_unstale":"2026-02-23T11:14:45.062078+0000","last_undegraded":"2026-02-23T11:14:45.062078+0000","last_fullsized":"2026-02-23T11:14:45.062078+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-23T11:14:42.032806+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-23T11:14:42.032806+0000","last_clean_scrub_stamp":"2026-02-23T11:14:42.032806+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":15,"reported_epoch":14,"state":"active+clean","last_fresh":"2026-02-23T11:14:45.058249+0000","last_change":"2026-02-23T11:14:45.060697+0000","last_active":"2026-02-23T11:14:45.058249+0000","last_peered":"2026-02-23T11:14:45.058249+0000","last_clean":"2026-02-23T11:14:45.058249+0000","last_became_active":"2026-02-23T11:14:43.085371+0000","last_became_peered":"2026-02-23T11:14:43.085371+0000","last_unstale":"2026-02-23T11:14:45.058249+0000","last_undegraded":"2026-02-23T11:14:45.058249+0000","last_fullsized":"2026-02-23T11:14:45.058249+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-23T11:14:42.032806+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-23T11:14:42.032806+0000","last_clean_scrub_stamp":"2026-02-23T11:14:42.032806+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":15,"reported_epoch":14,"state":"active+clean","last_fresh":"2026-02-23T11:14:45.057824+0000","last_change":"2026-02-23T11:14:45.061764+0000","last_active":"2026-02-23T11:14:45.057824+0000","last_peered":"2026-02-23T11:14:45.057824+0000","last_clean":"2026-02-23T11:14:45.057824+0000","last_became_active":"2026-02-23T11:14:43.100854+0000","last_became_peered":"2026-02-23T11:14:43.100854+0000","last_unstale":"2026-02-23T11:14:45.057824+0000","last_undegraded":"2026-02-23T11:14:45.057824+0000","last_fullsized":"2026-02-23T11:14:45.057824+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-23T11:14:42.032806+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-23T11:14:42.032806+0000","last_clean_scrub_stamp":"2026-02-23T11:14:42.032806+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":160658620416,"available":160657797120,"internally_reserved":0,"allocated":823296,"data_stored":823296,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":10,"seq":42949672989,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52297728,"kb_used":244,"kb_used_data":244,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":52297484,"statfs":{"total":53552873472,"available":53552623616,"internally_reserved":0,"allocated":249856,"data_stored":249856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":10,"seq":42949672989,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52297728,"kb_used":280,"kb_used_data":280,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":52297448,"statfs":{"total":53552873472,"available":53552586752,"internally_reserved":0,"allocated":286720,"data_stored":286720,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705698,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52297728,"kb_used":280,"kb_used_data":280,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":52297448,"statfs":{"total":53552873472,"available":53552586752,"internally_reserved":0,"allocated":286720,"data_stored":286720,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":53552873472,"available":53552586752,"internally_reserved":0,"allocated":286720,"data_stored":286720,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":53552873472,"available":53552586752,"internally_reserved":0,"allocated":286720,"data_stored":286720,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":53552873472,"available":53552623616,"internally_reserved":0,"allocated":249856,"data_stored":249856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-23T11:15:12.647 DEBUG:teuthology.orchestra.run.trial191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-23T11:15:12.723 INFO:teuthology.orchestra.run.trial191.stderr:2026-02-23T11:15:12.719+0000 7f731cccb640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T11:15:12.725 INFO:teuthology.orchestra.run.trial191.stderr:2026-02-23T11:15:12.724+0000 7f731cccb640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T11:15:12.836 INFO:teuthology.orchestra.run.trial191.stdout: 2026-02-23T11:15:12.837 INFO:teuthology.orchestra.run.trial191.stderr:dumped all 2026-02-23T11:15:12.845 INFO:teuthology.orchestra.run.trial191.stdout:{"pg_ready":true,"pg_map":{"version":26,"stamp":"2026-02-23T11:15:11.529222+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":3,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":156893184,"kb_used":804,"kb_used_data":804,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":156892380,"statfs":{"total":160658620416,"available":160657797120,"internally_reserved":0,"allocated":823296,"data_stored":823296,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001739"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":15,"reported_epoch":14,"state":"active+clean","last_fresh":"2026-02-23T11:14:45.060134+0000","last_change":"2026-02-23T11:14:45.066249+0000","last_active":"2026-02-23T11:14:45.060134+0000","last_peered":"2026-02-23T11:14:45.060134+0000","last_clean":"2026-02-23T11:14:45.060134+0000","last_became_active":"2026-02-23T11:14:43.080217+0000","last_became_peered":"2026-02-23T11:14:43.080217+0000","last_unstale":"2026-02-23T11:14:45.060134+0000","last_undegraded":"2026-02-23T11:14:45.060134+0000","last_fullsized":"2026-02-23T11:14:45.060134+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-23T11:14:42.032806+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-23T11:14:42.032806+0000","last_clean_scrub_stamp":"2026-02-23T11:14:42.032806+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":15,"reported_epoch":14,"state":"active+clean","last_fresh":"2026-02-23T11:14:45.057488+0000","last_change":"2026-02-23T11:14:45.061801+0000","last_active":"2026-02-23T11:14:45.057488+0000","last_peered":"2026-02-23T11:14:45.057488+0000","last_clean":"2026-02-23T11:14:45.057488+0000","last_became_active":"2026-02-23T11:14:43.116310+0000","last_became_peered":"2026-02-23T11:14:43.116310+0000","last_unstale":"2026-02-23T11:14:45.057488+0000","last_undegraded":"2026-02-23T11:14:45.057488+0000","last_fullsized":"2026-02-23T11:14:45.057488+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-23T11:14:42.032806+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-23T11:14:42.032806+0000","last_clean_scrub_stamp":"2026-02-23T11:14:42.032806+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":15,"reported_epoch":14,"state":"active+clean","last_fresh":"2026-02-23T11:14:45.060107+0000","last_change":"2026-02-23T11:14:45.062322+0000","last_active":"2026-02-23T11:14:45.060107+0000","last_peered":"2026-02-23T11:14:45.060107+0000","last_clean":"2026-02-23T11:14:45.060107+0000","last_became_active":"2026-02-23T11:14:43.079925+0000","last_became_peered":"2026-02-23T11:14:43.079925+0000","last_unstale":"2026-02-23T11:14:45.060107+0000","last_undegraded":"2026-02-23T11:14:45.060107+0000","last_fullsized":"2026-02-23T11:14:45.060107+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-23T11:14:42.032806+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-23T11:14:42.032806+0000","last_clean_scrub_stamp":"2026-02-23T11:14:42.032806+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":15,"reported_epoch":14,"state":"active+clean","last_fresh":"2026-02-23T11:14:45.059784+0000","last_change":"2026-02-23T11:14:45.066152+0000","last_active":"2026-02-23T11:14:45.059784+0000","last_peered":"2026-02-23T11:14:45.059784+0000","last_clean":"2026-02-23T11:14:45.059784+0000","last_became_active":"2026-02-23T11:14:43.096768+0000","last_became_peered":"2026-02-23T11:14:43.096768+0000","last_unstale":"2026-02-23T11:14:45.059784+0000","last_undegraded":"2026-02-23T11:14:45.059784+0000","last_fullsized":"2026-02-23T11:14:45.059784+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-23T11:14:42.032806+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-23T11:14:42.032806+0000","last_clean_scrub_stamp":"2026-02-23T11:14:42.032806+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":16,"reported_epoch":14,"state":"active+clean","last_fresh":"2026-02-23T11:14:45.060509+0000","last_change":"2026-02-23T11:14:45.065912+0000","last_active":"2026-02-23T11:14:45.060509+0000","last_peered":"2026-02-23T11:14:45.060509+0000","last_clean":"2026-02-23T11:14:45.060509+0000","last_became_active":"2026-02-23T11:14:43.077394+0000","last_became_peered":"2026-02-23T11:14:43.077394+0000","last_unstale":"2026-02-23T11:14:45.060509+0000","last_undegraded":"2026-02-23T11:14:45.060509+0000","last_fullsized":"2026-02-23T11:14:45.060509+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-23T11:14:42.032806+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-23T11:14:42.032806+0000","last_clean_scrub_stamp":"2026-02-23T11:14:42.032806+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"13'1","reported_seq":17,"reported_epoch":14,"state":"active+clean","last_fresh":"2026-02-23T11:14:45.062078+0000","last_change":"2026-02-23T11:14:45.062078+0000","last_active":"2026-02-23T11:14:45.062078+0000","last_peered":"2026-02-23T11:14:45.062078+0000","last_clean":"2026-02-23T11:14:45.062078+0000","last_became_active":"2026-02-23T11:14:43.092837+0000","last_became_peered":"2026-02-23T11:14:43.092837+0000","last_unstale":"2026-02-23T11:14:45.062078+0000","last_undegraded":"2026-02-23T11:14:45.062078+0000","last_fullsized":"2026-02-23T11:14:45.062078+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-23T11:14:42.032806+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-23T11:14:42.032806+0000","last_clean_scrub_stamp":"2026-02-23T11:14:42.032806+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":15,"reported_epoch":14,"state":"active+clean","last_fresh":"2026-02-23T11:14:45.058249+0000","last_change":"2026-02-23T11:14:45.060697+0000","last_active":"2026-02-23T11:14:45.058249+0000","last_peered":"2026-02-23T11:14:45.058249+0000","last_clean":"2026-02-23T11:14:45.058249+0000","last_became_active":"2026-02-23T11:14:43.085371+0000","last_became_peered":"2026-02-23T11:14:43.085371+0000","last_unstale":"2026-02-23T11:14:45.058249+0000","last_undegraded":"2026-02-23T11:14:45.058249+0000","last_fullsized":"2026-02-23T11:14:45.058249+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-23T11:14:42.032806+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-23T11:14:42.032806+0000","last_clean_scrub_stamp":"2026-02-23T11:14:42.032806+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":15,"reported_epoch":14,"state":"active+clean","last_fresh":"2026-02-23T11:14:45.057824+0000","last_change":"2026-02-23T11:14:45.061764+0000","last_active":"2026-02-23T11:14:45.057824+0000","last_peered":"2026-02-23T11:14:45.057824+0000","last_clean":"2026-02-23T11:14:45.057824+0000","last_became_active":"2026-02-23T11:14:43.100854+0000","last_became_peered":"2026-02-23T11:14:43.100854+0000","last_unstale":"2026-02-23T11:14:45.057824+0000","last_undegraded":"2026-02-23T11:14:45.057824+0000","last_fullsized":"2026-02-23T11:14:45.057824+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-23T11:14:42.032806+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-23T11:14:42.032806+0000","last_clean_scrub_stamp":"2026-02-23T11:14:42.032806+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":160658620416,"available":160657797120,"internally_reserved":0,"allocated":823296,"data_stored":823296,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":10,"seq":42949672989,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52297728,"kb_used":244,"kb_used_data":244,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":52297484,"statfs":{"total":53552873472,"available":53552623616,"internally_reserved":0,"allocated":249856,"data_stored":249856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":10,"seq":42949672989,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52297728,"kb_used":280,"kb_used_data":280,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":52297448,"statfs":{"total":53552873472,"available":53552586752,"internally_reserved":0,"allocated":286720,"data_stored":286720,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705698,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52297728,"kb_used":280,"kb_used_data":280,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":52297448,"statfs":{"total":53552873472,"available":53552586752,"internally_reserved":0,"allocated":286720,"data_stored":286720,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":53552873472,"available":53552586752,"internally_reserved":0,"allocated":286720,"data_stored":286720,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":53552873472,"available":53552586752,"internally_reserved":0,"allocated":286720,"data_stored":286720,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":53552873472,"available":53552623616,"internally_reserved":0,"allocated":249856,"data_stored":249856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-23T11:15:12.847 INFO:tasks.ceph.ceph_manager.ceph:clean! 2026-02-23T11:15:12.847 DEBUG:teuthology.orchestra.run.trial191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-23T11:15:12.923 INFO:teuthology.orchestra.run.trial191.stderr:2026-02-23T11:15:12.919+0000 7fde1cc46640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T11:15:12.925 INFO:teuthology.orchestra.run.trial191.stderr:2026-02-23T11:15:12.924+0000 7fde1cc46640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T11:15:13.038 INFO:teuthology.orchestra.run.trial191.stdout: 2026-02-23T11:15:13.038 INFO:teuthology.orchestra.run.trial191.stderr:dumped all 2026-02-23T11:15:13.048 INFO:teuthology.orchestra.run.trial191.stdout:{"pg_ready":true,"pg_map":{"version":26,"stamp":"2026-02-23T11:15:11.529222+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":3,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":156893184,"kb_used":804,"kb_used_data":804,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":156892380,"statfs":{"total":160658620416,"available":160657797120,"internally_reserved":0,"allocated":823296,"data_stored":823296,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001739"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":15,"reported_epoch":14,"state":"active+clean","last_fresh":"2026-02-23T11:14:45.060134+0000","last_change":"2026-02-23T11:14:45.066249+0000","last_active":"2026-02-23T11:14:45.060134+0000","last_peered":"2026-02-23T11:14:45.060134+0000","last_clean":"2026-02-23T11:14:45.060134+0000","last_became_active":"2026-02-23T11:14:43.080217+0000","last_became_peered":"2026-02-23T11:14:43.080217+0000","last_unstale":"2026-02-23T11:14:45.060134+0000","last_undegraded":"2026-02-23T11:14:45.060134+0000","last_fullsized":"2026-02-23T11:14:45.060134+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-23T11:14:42.032806+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-23T11:14:42.032806+0000","last_clean_scrub_stamp":"2026-02-23T11:14:42.032806+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":15,"reported_epoch":14,"state":"active+clean","last_fresh":"2026-02-23T11:14:45.057488+0000","last_change":"2026-02-23T11:14:45.061801+0000","last_active":"2026-02-23T11:14:45.057488+0000","last_peered":"2026-02-23T11:14:45.057488+0000","last_clean":"2026-02-23T11:14:45.057488+0000","last_became_active":"2026-02-23T11:14:43.116310+0000","last_became_peered":"2026-02-23T11:14:43.116310+0000","last_unstale":"2026-02-23T11:14:45.057488+0000","last_undegraded":"2026-02-23T11:14:45.057488+0000","last_fullsized":"2026-02-23T11:14:45.057488+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-23T11:14:42.032806+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-23T11:14:42.032806+0000","last_clean_scrub_stamp":"2026-02-23T11:14:42.032806+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":15,"reported_epoch":14,"state":"active+clean","last_fresh":"2026-02-23T11:14:45.060107+0000","last_change":"2026-02-23T11:14:45.062322+0000","last_active":"2026-02-23T11:14:45.060107+0000","last_peered":"2026-02-23T11:14:45.060107+0000","last_clean":"2026-02-23T11:14:45.060107+0000","last_became_active":"2026-02-23T11:14:43.079925+0000","last_became_peered":"2026-02-23T11:14:43.079925+0000","last_unstale":"2026-02-23T11:14:45.060107+0000","last_undegraded":"2026-02-23T11:14:45.060107+0000","last_fullsized":"2026-02-23T11:14:45.060107+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-23T11:14:42.032806+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-23T11:14:42.032806+0000","last_clean_scrub_stamp":"2026-02-23T11:14:42.032806+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":15,"reported_epoch":14,"state":"active+clean","last_fresh":"2026-02-23T11:14:45.059784+0000","last_change":"2026-02-23T11:14:45.066152+0000","last_active":"2026-02-23T11:14:45.059784+0000","last_peered":"2026-02-23T11:14:45.059784+0000","last_clean":"2026-02-23T11:14:45.059784+0000","last_became_active":"2026-02-23T11:14:43.096768+0000","last_became_peered":"2026-02-23T11:14:43.096768+0000","last_unstale":"2026-02-23T11:14:45.059784+0000","last_undegraded":"2026-02-23T11:14:45.059784+0000","last_fullsized":"2026-02-23T11:14:45.059784+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-23T11:14:42.032806+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-23T11:14:42.032806+0000","last_clean_scrub_stamp":"2026-02-23T11:14:42.032806+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":16,"reported_epoch":14,"state":"active+clean","last_fresh":"2026-02-23T11:14:45.060509+0000","last_change":"2026-02-23T11:14:45.065912+0000","last_active":"2026-02-23T11:14:45.060509+0000","last_peered":"2026-02-23T11:14:45.060509+0000","last_clean":"2026-02-23T11:14:45.060509+0000","last_became_active":"2026-02-23T11:14:43.077394+0000","last_became_peered":"2026-02-23T11:14:43.077394+0000","last_unstale":"2026-02-23T11:14:45.060509+0000","last_undegraded":"2026-02-23T11:14:45.060509+0000","last_fullsized":"2026-02-23T11:14:45.060509+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-23T11:14:42.032806+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-23T11:14:42.032806+0000","last_clean_scrub_stamp":"2026-02-23T11:14:42.032806+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"13'1","reported_seq":17,"reported_epoch":14,"state":"active+clean","last_fresh":"2026-02-23T11:14:45.062078+0000","last_change":"2026-02-23T11:14:45.062078+0000","last_active":"2026-02-23T11:14:45.062078+0000","last_peered":"2026-02-23T11:14:45.062078+0000","last_clean":"2026-02-23T11:14:45.062078+0000","last_became_active":"2026-02-23T11:14:43.092837+0000","last_became_peered":"2026-02-23T11:14:43.092837+0000","last_unstale":"2026-02-23T11:14:45.062078+0000","last_undegraded":"2026-02-23T11:14:45.062078+0000","last_fullsized":"2026-02-23T11:14:45.062078+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-23T11:14:42.032806+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-23T11:14:42.032806+0000","last_clean_scrub_stamp":"2026-02-23T11:14:42.032806+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":15,"reported_epoch":14,"state":"active+clean","last_fresh":"2026-02-23T11:14:45.058249+0000","last_change":"2026-02-23T11:14:45.060697+0000","last_active":"2026-02-23T11:14:45.058249+0000","last_peered":"2026-02-23T11:14:45.058249+0000","last_clean":"2026-02-23T11:14:45.058249+0000","last_became_active":"2026-02-23T11:14:43.085371+0000","last_became_peered":"2026-02-23T11:14:43.085371+0000","last_unstale":"2026-02-23T11:14:45.058249+0000","last_undegraded":"2026-02-23T11:14:45.058249+0000","last_fullsized":"2026-02-23T11:14:45.058249+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-23T11:14:42.032806+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-23T11:14:42.032806+0000","last_clean_scrub_stamp":"2026-02-23T11:14:42.032806+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":15,"reported_epoch":14,"state":"active+clean","last_fresh":"2026-02-23T11:14:45.057824+0000","last_change":"2026-02-23T11:14:45.061764+0000","last_active":"2026-02-23T11:14:45.057824+0000","last_peered":"2026-02-23T11:14:45.057824+0000","last_clean":"2026-02-23T11:14:45.057824+0000","last_became_active":"2026-02-23T11:14:43.100854+0000","last_became_peered":"2026-02-23T11:14:43.100854+0000","last_unstale":"2026-02-23T11:14:45.057824+0000","last_undegraded":"2026-02-23T11:14:45.057824+0000","last_fullsized":"2026-02-23T11:14:45.057824+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-23T11:14:42.032806+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-23T11:14:42.032806+0000","last_clean_scrub_stamp":"2026-02-23T11:14:42.032806+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":160658620416,"available":160657797120,"internally_reserved":0,"allocated":823296,"data_stored":823296,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":10,"seq":42949672989,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52297728,"kb_used":244,"kb_used_data":244,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":52297484,"statfs":{"total":53552873472,"available":53552623616,"internally_reserved":0,"allocated":249856,"data_stored":249856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":10,"seq":42949672989,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52297728,"kb_used":280,"kb_used_data":280,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":52297448,"statfs":{"total":53552873472,"available":53552586752,"internally_reserved":0,"allocated":286720,"data_stored":286720,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705698,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52297728,"kb_used":280,"kb_used_data":280,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":52297448,"statfs":{"total":53552873472,"available":53552586752,"internally_reserved":0,"allocated":286720,"data_stored":286720,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":53552873472,"available":53552586752,"internally_reserved":0,"allocated":286720,"data_stored":286720,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":53552873472,"available":53552586752,"internally_reserved":0,"allocated":286720,"data_stored":286720,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":53552873472,"available":53552623616,"internally_reserved":0,"allocated":249856,"data_stored":249856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-23T11:15:13.049 DEBUG:teuthology.orchestra.run.trial191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2026-02-23T11:15:13.128 INFO:teuthology.orchestra.run.trial191.stderr:2026-02-23T11:15:13.125+0000 7f45ee85a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T11:15:13.131 INFO:teuthology.orchestra.run.trial191.stderr:2026-02-23T11:15:13.129+0000 7f45ee85a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T11:15:13.245 INFO:teuthology.orchestra.run.trial191.stdout: 2026-02-23T11:15:13.245 INFO:teuthology.orchestra.run.trial191.stdout:{"epoch":14,"fsid":"408068a4-9c4e-43c3-a1bd-e489279d442d","created":"2026-02-23T11:14:25.368540+0000","modified":"2026-02-23T11:14:45.044401+0000","last_up_change":"2026-02-23T11:14:36.956000+0000","last_in_change":"2026-02-23T11:14:27.217466+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":"rbd","create_time":"2026-02-23T11:14:41.181595+0000","flags":270337,"flags_names":"hashpspool,selfmanaged_snaps,crimson","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"14","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":2,"snap_epoch":14,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"e4f36d13-e8b8-4777-85ed-8210652f197c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":11,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.191:6802","nonce":3272030818}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.191:6803","nonce":3272030818}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.191:6806","nonce":3272030818}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.191:6809","nonce":3272030818}]},"public_addr":"10.20.193.191:6802/3272030818","cluster_addr":"10.20.193.191:6803/3272030818","heartbeat_back_addr":"10.20.193.191:6806/3272030818","heartbeat_front_addr":"10.20.193.191:6809/3272030818","state":["exists","up"]},{"osd":1,"uuid":"e39a1968-9db1-42d6-b792-e3aa32109d97","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":10,"up_thru":11,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.191:6808","nonce":2597130977}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.191:6807","nonce":2597130977}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.191:6811","nonce":2597130977}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.191:6810","nonce":2597130977}]},"public_addr":"10.20.193.191:6808/2597130977","cluster_addr":"10.20.193.191:6807/2597130977","heartbeat_back_addr":"10.20.193.191:6811/2597130977","heartbeat_front_addr":"10.20.193.191:6810/2597130977","state":["exists","up"]},{"osd":2,"uuid":"09f145f1-9abb-4692-b128-bf1aff6fe492","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":10,"up_thru":11,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.191:6805","nonce":3503848635}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.191:6804","nonce":3503848635}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.191:6813","nonce":3503848635}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.191:6812","nonce":3503848635}]},"public_addr":"10.20.193.191:6805/3503848635","cluster_addr":"10.20.193.191:6804/3503848635","heartbeat_back_addr":"10.20.193.191:6813/3503848635","heartbeat_front_addr":"10.20.193.191:6812/3503848635","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":1,"snaps":[{"begin":2,"length":1}]}],"new_removed_snaps":[{"pool":1,"snaps":[{"begin":2,"length":1}]}],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T11:15:14.255 INFO:tasks.ceph:Scrubbing osd.0 2026-02-23T11:15:14.256 DEBUG:teuthology.orchestra.run.trial191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 config set osd_debug_deep_scrub_sleep 0 2026-02-23T11:15:14.337 INFO:teuthology.orchestra.run.trial191.stderr:2026-02-23T11:15:14.331+0000 7fe1437c1640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T11:15:14.339 INFO:teuthology.orchestra.run.trial191.stderr:2026-02-23T11:15:14.338+0000 7fe1437c1640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T11:15:14.349 INFO:teuthology.orchestra.run.trial191.stdout:{ 2026-02-23T11:15:14.349 INFO:teuthology.orchestra.run.trial191.stdout: "success": "" 2026-02-23T11:15:14.349 INFO:teuthology.orchestra.run.trial191.stdout:} 2026-02-23T11:15:14.357 DEBUG:teuthology.orchestra.run.trial191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 0 2026-02-23T11:15:14.433 INFO:teuthology.orchestra.run.trial191.stderr:2026-02-23T11:15:14.429+0000 7fa0faccb640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T11:15:14.436 INFO:teuthology.orchestra.run.trial191.stderr:2026-02-23T11:15:14.434+0000 7fa0faccb640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T11:15:14.547 INFO:teuthology.orchestra.run.trial191.stderr:instructed osd(s) 0 to deep-scrub 2026-02-23T11:15:14.558 INFO:tasks.ceph:Scrubbing osd.1 2026-02-23T11:15:14.558 DEBUG:teuthology.orchestra.run.trial191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 config set osd_debug_deep_scrub_sleep 0 2026-02-23T11:15:14.637 INFO:teuthology.orchestra.run.trial191.stderr:2026-02-23T11:15:14.631+0000 7fe3cb05a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T11:15:14.640 INFO:teuthology.orchestra.run.trial191.stderr:2026-02-23T11:15:14.638+0000 7fe3cb05a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T11:15:14.650 INFO:teuthology.orchestra.run.trial191.stdout:{ 2026-02-23T11:15:14.650 INFO:teuthology.orchestra.run.trial191.stdout: "success": "" 2026-02-23T11:15:14.650 INFO:teuthology.orchestra.run.trial191.stdout:} 2026-02-23T11:15:14.656 DEBUG:teuthology.orchestra.run.trial191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 1 2026-02-23T11:15:14.732 INFO:teuthology.orchestra.run.trial191.stderr:2026-02-23T11:15:14.728+0000 7f2168ac2640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T11:15:14.735 INFO:teuthology.orchestra.run.trial191.stderr:2026-02-23T11:15:14.734+0000 7f2168ac2640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T11:15:14.849 INFO:teuthology.orchestra.run.trial191.stderr:instructed osd(s) 1 to deep-scrub 2026-02-23T11:15:14.860 INFO:tasks.ceph:Scrubbing osd.2 2026-02-23T11:15:14.860 DEBUG:teuthology.orchestra.run.trial191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 config set osd_debug_deep_scrub_sleep 0 2026-02-23T11:15:14.938 INFO:teuthology.orchestra.run.trial191.stderr:2026-02-23T11:15:14.931+0000 7fefdc25a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T11:15:14.941 INFO:teuthology.orchestra.run.trial191.stderr:2026-02-23T11:15:14.939+0000 7fefdc25a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T11:15:14.950 INFO:teuthology.orchestra.run.trial191.stdout:{ 2026-02-23T11:15:14.950 INFO:teuthology.orchestra.run.trial191.stdout: "success": "" 2026-02-23T11:15:14.950 INFO:teuthology.orchestra.run.trial191.stdout:} 2026-02-23T11:15:14.957 DEBUG:teuthology.orchestra.run.trial191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 2 2026-02-23T11:15:15.035 INFO:teuthology.orchestra.run.trial191.stderr:2026-02-23T11:15:15.030+0000 7fdfdeecb640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T11:15:15.037 INFO:teuthology.orchestra.run.trial191.stderr:2026-02-23T11:15:15.036+0000 7fdfdeecb640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T11:15:15.147 INFO:teuthology.orchestra.run.trial191.stderr:instructed osd(s) 2 to deep-scrub 2026-02-23T11:15:15.160 DEBUG:teuthology.orchestra.run.trial191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-23T11:15:15.237 INFO:teuthology.orchestra.run.trial191.stderr:2026-02-23T11:15:15.233+0000 7f31a0b0b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T11:15:15.239 INFO:teuthology.orchestra.run.trial191.stderr:2026-02-23T11:15:15.238+0000 7f31a0b0b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T11:15:15.350 INFO:teuthology.orchestra.run.trial191.stdout: 2026-02-23T11:15:15.350 INFO:teuthology.orchestra.run.trial191.stderr:dumped all 2026-02-23T11:15:15.359 INFO:teuthology.orchestra.run.trial191.stdout:{"pg_ready":true,"pg_map":{"version":27,"stamp":"2026-02-23T11:15:13.529478+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":3,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":156893184,"kb_used":804,"kb_used_data":804,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":156892380,"statfs":{"total":160658620416,"available":160657797120,"internally_reserved":0,"allocated":823296,"data_stored":823296,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001795"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":15,"reported_epoch":14,"state":"active+clean","last_fresh":"2026-02-23T11:14:45.060134+0000","last_change":"2026-02-23T11:14:45.066249+0000","last_active":"2026-02-23T11:14:45.060134+0000","last_peered":"2026-02-23T11:14:45.060134+0000","last_clean":"2026-02-23T11:14:45.060134+0000","last_became_active":"2026-02-23T11:14:43.080217+0000","last_became_peered":"2026-02-23T11:14:43.080217+0000","last_unstale":"2026-02-23T11:14:45.060134+0000","last_undegraded":"2026-02-23T11:14:45.060134+0000","last_fullsized":"2026-02-23T11:14:45.060134+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-23T11:14:42.032806+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-23T11:14:42.032806+0000","last_clean_scrub_stamp":"2026-02-23T11:14:42.032806+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":15,"reported_epoch":14,"state":"active+clean","last_fresh":"2026-02-23T11:14:45.057488+0000","last_change":"2026-02-23T11:14:45.061801+0000","last_active":"2026-02-23T11:14:45.057488+0000","last_peered":"2026-02-23T11:14:45.057488+0000","last_clean":"2026-02-23T11:14:45.057488+0000","last_became_active":"2026-02-23T11:14:43.116310+0000","last_became_peered":"2026-02-23T11:14:43.116310+0000","last_unstale":"2026-02-23T11:14:45.057488+0000","last_undegraded":"2026-02-23T11:14:45.057488+0000","last_fullsized":"2026-02-23T11:14:45.057488+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-23T11:14:42.032806+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-23T11:14:42.032806+0000","last_clean_scrub_stamp":"2026-02-23T11:14:42.032806+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":15,"reported_epoch":14,"state":"active+clean","last_fresh":"2026-02-23T11:14:45.060107+0000","last_change":"2026-02-23T11:14:45.062322+0000","last_active":"2026-02-23T11:14:45.060107+0000","last_peered":"2026-02-23T11:14:45.060107+0000","last_clean":"2026-02-23T11:14:45.060107+0000","last_became_active":"2026-02-23T11:14:43.079925+0000","last_became_peered":"2026-02-23T11:14:43.079925+0000","last_unstale":"2026-02-23T11:14:45.060107+0000","last_undegraded":"2026-02-23T11:14:45.060107+0000","last_fullsized":"2026-02-23T11:14:45.060107+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-23T11:14:42.032806+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-23T11:14:42.032806+0000","last_clean_scrub_stamp":"2026-02-23T11:14:42.032806+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":15,"reported_epoch":14,"state":"active+clean","last_fresh":"2026-02-23T11:14:45.059784+0000","last_change":"2026-02-23T11:14:45.066152+0000","last_active":"2026-02-23T11:14:45.059784+0000","last_peered":"2026-02-23T11:14:45.059784+0000","last_clean":"2026-02-23T11:14:45.059784+0000","last_became_active":"2026-02-23T11:14:43.096768+0000","last_became_peered":"2026-02-23T11:14:43.096768+0000","last_unstale":"2026-02-23T11:14:45.059784+0000","last_undegraded":"2026-02-23T11:14:45.059784+0000","last_fullsized":"2026-02-23T11:14:45.059784+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-23T11:14:42.032806+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-23T11:14:42.032806+0000","last_clean_scrub_stamp":"2026-02-23T11:14:42.032806+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":16,"reported_epoch":14,"state":"active+clean","last_fresh":"2026-02-23T11:14:45.060509+0000","last_change":"2026-02-23T11:14:45.065912+0000","last_active":"2026-02-23T11:14:45.060509+0000","last_peered":"2026-02-23T11:14:45.060509+0000","last_clean":"2026-02-23T11:14:45.060509+0000","last_became_active":"2026-02-23T11:14:43.077394+0000","last_became_peered":"2026-02-23T11:14:43.077394+0000","last_unstale":"2026-02-23T11:14:45.060509+0000","last_undegraded":"2026-02-23T11:14:45.060509+0000","last_fullsized":"2026-02-23T11:14:45.060509+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-23T11:14:42.032806+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-23T11:14:42.032806+0000","last_clean_scrub_stamp":"2026-02-23T11:14:42.032806+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"13'1","reported_seq":17,"reported_epoch":14,"state":"active+clean","last_fresh":"2026-02-23T11:14:45.062078+0000","last_change":"2026-02-23T11:14:45.062078+0000","last_active":"2026-02-23T11:14:45.062078+0000","last_peered":"2026-02-23T11:14:45.062078+0000","last_clean":"2026-02-23T11:14:45.062078+0000","last_became_active":"2026-02-23T11:14:43.092837+0000","last_became_peered":"2026-02-23T11:14:43.092837+0000","last_unstale":"2026-02-23T11:14:45.062078+0000","last_undegraded":"2026-02-23T11:14:45.062078+0000","last_fullsized":"2026-02-23T11:14:45.062078+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-23T11:14:42.032806+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-23T11:14:42.032806+0000","last_clean_scrub_stamp":"2026-02-23T11:14:42.032806+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":15,"reported_epoch":14,"state":"active+clean","last_fresh":"2026-02-23T11:14:45.058249+0000","last_change":"2026-02-23T11:14:45.060697+0000","last_active":"2026-02-23T11:14:45.058249+0000","last_peered":"2026-02-23T11:14:45.058249+0000","last_clean":"2026-02-23T11:14:45.058249+0000","last_became_active":"2026-02-23T11:14:43.085371+0000","last_became_peered":"2026-02-23T11:14:43.085371+0000","last_unstale":"2026-02-23T11:14:45.058249+0000","last_undegraded":"2026-02-23T11:14:45.058249+0000","last_fullsized":"2026-02-23T11:14:45.058249+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-23T11:14:42.032806+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-23T11:14:42.032806+0000","last_clean_scrub_stamp":"2026-02-23T11:14:42.032806+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":15,"reported_epoch":14,"state":"active+clean","last_fresh":"2026-02-23T11:14:45.057824+0000","last_change":"2026-02-23T11:14:45.061764+0000","last_active":"2026-02-23T11:14:45.057824+0000","last_peered":"2026-02-23T11:14:45.057824+0000","last_clean":"2026-02-23T11:14:45.057824+0000","last_became_active":"2026-02-23T11:14:43.100854+0000","last_became_peered":"2026-02-23T11:14:43.100854+0000","last_unstale":"2026-02-23T11:14:45.057824+0000","last_undegraded":"2026-02-23T11:14:45.057824+0000","last_fullsized":"2026-02-23T11:14:45.057824+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-23T11:14:42.032806+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-23T11:14:42.032806+0000","last_clean_scrub_stamp":"2026-02-23T11:14:42.032806+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":160658620416,"available":160657797120,"internally_reserved":0,"allocated":823296,"data_stored":823296,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":10,"seq":42949672994,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52297728,"kb_used":244,"kb_used_data":244,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":52297484,"statfs":{"total":53552873472,"available":53552623616,"internally_reserved":0,"allocated":249856,"data_stored":249856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":10,"seq":42949672994,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52297728,"kb_used":280,"kb_used_data":280,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":52297448,"statfs":{"total":53552873472,"available":53552586752,"internally_reserved":0,"allocated":286720,"data_stored":286720,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705698,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52297728,"kb_used":280,"kb_used_data":280,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":52297448,"statfs":{"total":53552873472,"available":53552586752,"internally_reserved":0,"allocated":286720,"data_stored":286720,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":53552873472,"available":53552586752,"internally_reserved":0,"allocated":286720,"data_stored":286720,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":53552873472,"available":53552586752,"internally_reserved":0,"allocated":286720,"data_stored":286720,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":53552873472,"available":53552623616,"internally_reserved":0,"allocated":249856,"data_stored":249856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-23T11:15:15.364 INFO:tasks.ceph:pgid 1.7 last_scrub_stamp 2026-02-23T11:14:42.032806+0000 time.struct_time(tm_year=2026, tm_mon=2, tm_mday=23, tm_hour=11, tm_min=14, tm_sec=42, tm_wday=0, tm_yday=54, tm_isdst=-1) <= time.struct_time(tm_year=2026, tm_mon=2, tm_mday=23, tm_hour=11, tm_min=15, tm_sec=13, tm_wday=0, tm_yday=54, tm_isdst=0) 2026-02-23T11:15:15.364 INFO:tasks.ceph:pgid 1.6 last_scrub_stamp 2026-02-23T11:14:42.032806+0000 time.struct_time(tm_year=2026, tm_mon=2, tm_mday=23, tm_hour=11, tm_min=14, tm_sec=42, tm_wday=0, tm_yday=54, tm_isdst=-1) <= time.struct_time(tm_year=2026, tm_mon=2, tm_mday=23, tm_hour=11, tm_min=15, tm_sec=13, tm_wday=0, tm_yday=54, tm_isdst=0) 2026-02-23T11:15:15.365 INFO:tasks.ceph:pgid 1.5 last_scrub_stamp 2026-02-23T11:14:42.032806+0000 time.struct_time(tm_year=2026, tm_mon=2, tm_mday=23, tm_hour=11, tm_min=14, tm_sec=42, tm_wday=0, tm_yday=54, tm_isdst=-1) <= time.struct_time(tm_year=2026, tm_mon=2, tm_mday=23, tm_hour=11, tm_min=15, tm_sec=13, tm_wday=0, tm_yday=54, tm_isdst=0) 2026-02-23T11:15:15.365 INFO:tasks.ceph:pgid 1.4 last_scrub_stamp 2026-02-23T11:14:42.032806+0000 time.struct_time(tm_year=2026, tm_mon=2, tm_mday=23, tm_hour=11, tm_min=14, tm_sec=42, tm_wday=0, tm_yday=54, tm_isdst=-1) <= time.struct_time(tm_year=2026, tm_mon=2, tm_mday=23, tm_hour=11, tm_min=15, tm_sec=13, tm_wday=0, tm_yday=54, tm_isdst=0) 2026-02-23T11:15:15.365 INFO:tasks.ceph:pgid 1.3 last_scrub_stamp 2026-02-23T11:14:42.032806+0000 time.struct_time(tm_year=2026, tm_mon=2, tm_mday=23, tm_hour=11, tm_min=14, tm_sec=42, tm_wday=0, tm_yday=54, tm_isdst=-1) <= time.struct_time(tm_year=2026, tm_mon=2, tm_mday=23, tm_hour=11, tm_min=15, tm_sec=13, tm_wday=0, tm_yday=54, tm_isdst=0) 2026-02-23T11:15:15.365 INFO:tasks.ceph:pgid 1.2 last_scrub_stamp 2026-02-23T11:14:42.032806+0000 time.struct_time(tm_year=2026, tm_mon=2, tm_mday=23, tm_hour=11, tm_min=14, tm_sec=42, tm_wday=0, tm_yday=54, tm_isdst=-1) <= time.struct_time(tm_year=2026, tm_mon=2, tm_mday=23, tm_hour=11, tm_min=15, tm_sec=13, tm_wday=0, tm_yday=54, tm_isdst=0) 2026-02-23T11:15:15.365 INFO:tasks.ceph:pgid 1.1 last_scrub_stamp 2026-02-23T11:14:42.032806+0000 time.struct_time(tm_year=2026, tm_mon=2, tm_mday=23, tm_hour=11, tm_min=14, tm_sec=42, tm_wday=0, tm_yday=54, tm_isdst=-1) <= time.struct_time(tm_year=2026, tm_mon=2, tm_mday=23, tm_hour=11, tm_min=15, tm_sec=13, tm_wday=0, tm_yday=54, tm_isdst=0) 2026-02-23T11:15:15.365 INFO:tasks.ceph:pgid 1.0 last_scrub_stamp 2026-02-23T11:14:42.032806+0000 time.struct_time(tm_year=2026, tm_mon=2, tm_mday=23, tm_hour=11, tm_min=14, tm_sec=42, tm_wday=0, tm_yday=54, tm_isdst=-1) <= time.struct_time(tm_year=2026, tm_mon=2, tm_mday=23, tm_hour=11, tm_min=15, tm_sec=13, tm_wday=0, tm_yday=54, tm_isdst=0) 2026-02-23T11:15:15.366 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2026-02-23T11:15:35.366 DEBUG:teuthology.orchestra.run.trial191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-23T11:15:35.446 INFO:teuthology.orchestra.run.trial191.stderr:2026-02-23T11:15:35.442+0000 7f01f20cb640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T11:15:35.448 INFO:teuthology.orchestra.run.trial191.stderr:2026-02-23T11:15:35.447+0000 7f01f20cb640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T11:15:35.563 INFO:teuthology.orchestra.run.trial191.stdout: 2026-02-23T11:15:35.563 INFO:teuthology.orchestra.run.trial191.stderr:dumped all 2026-02-23T11:15:35.573 INFO:teuthology.orchestra.run.trial191.stdout:{"pg_ready":true,"pg_map":{"version":38,"stamp":"2026-02-23T11:15:35.533457+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":3,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":156893184,"kb_used":804,"kb_used_data":804,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":156892380,"statfs":{"total":160658620416,"available":160657797120,"internally_reserved":0,"allocated":823296,"data_stored":823296,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002164"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":17,"reported_epoch":14,"state":"active+clean","last_fresh":"2026-02-23T11:15:14.853145+0000","last_change":"2026-02-23T11:15:14.863708+0000","last_active":"2026-02-23T11:15:14.853145+0000","last_peered":"2026-02-23T11:15:14.853145+0000","last_clean":"2026-02-23T11:15:14.853145+0000","last_became_active":"2026-02-23T11:14:43.080217+0000","last_became_peered":"2026-02-23T11:14:43.080217+0000","last_unstale":"2026-02-23T11:15:14.853145+0000","last_undegraded":"2026-02-23T11:15:14.853145+0000","last_fullsized":"2026-02-23T11:15:14.853145+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-23T11:15:14.863653+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-23T11:15:14.863653+0000","last_clean_scrub_stamp":"2026-02-23T11:14:42.032806+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":17,"reported_epoch":14,"state":"active+clean","last_fresh":"2026-02-23T11:15:14.851787+0000","last_change":"2026-02-23T11:15:14.861106+0000","last_active":"2026-02-23T11:15:14.851787+0000","last_peered":"2026-02-23T11:15:14.851787+0000","last_clean":"2026-02-23T11:15:14.851787+0000","last_became_active":"2026-02-23T11:14:43.116310+0000","last_became_peered":"2026-02-23T11:14:43.116310+0000","last_unstale":"2026-02-23T11:15:14.851787+0000","last_undegraded":"2026-02-23T11:15:14.851787+0000","last_fullsized":"2026-02-23T11:15:14.851787+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-23T11:15:14.861016+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-23T11:15:14.861016+0000","last_clean_scrub_stamp":"2026-02-23T11:14:42.032806+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":17,"reported_epoch":14,"state":"active+clean","last_fresh":"2026-02-23T11:15:15.148551+0000","last_change":"2026-02-23T11:15:15.155226+0000","last_active":"2026-02-23T11:15:15.148551+0000","last_peered":"2026-02-23T11:15:15.148551+0000","last_clean":"2026-02-23T11:15:15.148551+0000","last_became_active":"2026-02-23T11:14:43.079925+0000","last_became_peered":"2026-02-23T11:14:43.079925+0000","last_unstale":"2026-02-23T11:15:15.148551+0000","last_undegraded":"2026-02-23T11:15:15.148551+0000","last_fullsized":"2026-02-23T11:15:15.148551+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-23T11:15:15.155169+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-23T11:15:15.155169+0000","last_clean_scrub_stamp":"2026-02-23T11:14:42.032806+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":17,"reported_epoch":14,"state":"active+clean","last_fresh":"2026-02-23T11:15:14.852156+0000","last_change":"2026-02-23T11:15:14.862712+0000","last_active":"2026-02-23T11:15:14.852156+0000","last_peered":"2026-02-23T11:15:14.852156+0000","last_clean":"2026-02-23T11:15:14.852156+0000","last_became_active":"2026-02-23T11:14:43.096768+0000","last_became_peered":"2026-02-23T11:14:43.096768+0000","last_unstale":"2026-02-23T11:15:14.852156+0000","last_undegraded":"2026-02-23T11:15:14.852156+0000","last_fullsized":"2026-02-23T11:15:14.852156+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-23T11:15:14.862652+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-23T11:15:14.862652+0000","last_clean_scrub_stamp":"2026-02-23T11:14:42.032806+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":18,"reported_epoch":14,"state":"active+clean","last_fresh":"2026-02-23T11:15:14.851523+0000","last_change":"2026-02-23T11:15:14.865571+0000","last_active":"2026-02-23T11:15:14.851523+0000","last_peered":"2026-02-23T11:15:14.851523+0000","last_clean":"2026-02-23T11:15:14.851523+0000","last_became_active":"2026-02-23T11:14:43.077394+0000","last_became_peered":"2026-02-23T11:14:43.077394+0000","last_unstale":"2026-02-23T11:15:14.851523+0000","last_undegraded":"2026-02-23T11:15:14.851523+0000","last_fullsized":"2026-02-23T11:15:14.851523+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"12'1","last_scrub_stamp":"2026-02-23T11:15:14.865505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-23T11:15:14.865505+0000","last_clean_scrub_stamp":"2026-02-23T11:14:42.032806+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":19,"reported_epoch":14,"state":"active+clean","last_fresh":"2026-02-23T11:15:14.547855+0000","last_change":"2026-02-23T11:15:14.558275+0000","last_active":"2026-02-23T11:15:14.547855+0000","last_peered":"2026-02-23T11:15:14.547855+0000","last_clean":"2026-02-23T11:15:14.547855+0000","last_became_active":"2026-02-23T11:14:43.092837+0000","last_became_peered":"2026-02-23T11:14:43.092837+0000","last_unstale":"2026-02-23T11:15:14.547855+0000","last_undegraded":"2026-02-23T11:15:14.547855+0000","last_fullsized":"2026-02-23T11:15:14.547855+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"14'2","last_scrub_stamp":"2026-02-23T11:15:14.558201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-23T11:15:14.558201+0000","last_clean_scrub_stamp":"2026-02-23T11:14:42.032806+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":17,"reported_epoch":14,"state":"active+clean","last_fresh":"2026-02-23T11:15:15.148147+0000","last_change":"2026-02-23T11:15:15.154561+0000","last_active":"2026-02-23T11:15:15.148147+0000","last_peered":"2026-02-23T11:15:15.148147+0000","last_clean":"2026-02-23T11:15:15.148147+0000","last_became_active":"2026-02-23T11:14:43.085371+0000","last_became_peered":"2026-02-23T11:14:43.085371+0000","last_unstale":"2026-02-23T11:15:15.148147+0000","last_undegraded":"2026-02-23T11:15:15.148147+0000","last_fullsized":"2026-02-23T11:15:15.148147+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-23T11:15:15.154471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-23T11:15:15.154471+0000","last_clean_scrub_stamp":"2026-02-23T11:14:42.032806+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":17,"reported_epoch":14,"state":"active+clean","last_fresh":"2026-02-23T11:15:14.850412+0000","last_change":"2026-02-23T11:15:14.861681+0000","last_active":"2026-02-23T11:15:14.850412+0000","last_peered":"2026-02-23T11:15:14.850412+0000","last_clean":"2026-02-23T11:15:14.850412+0000","last_became_active":"2026-02-23T11:14:43.100854+0000","last_became_peered":"2026-02-23T11:14:43.100854+0000","last_unstale":"2026-02-23T11:15:14.850412+0000","last_undegraded":"2026-02-23T11:15:14.850412+0000","last_fullsized":"2026-02-23T11:15:14.850412+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-23T11:15:14.861630+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-23T11:15:14.861630+0000","last_clean_scrub_stamp":"2026-02-23T11:14:42.032806+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":160658620416,"available":160657797120,"internally_reserved":0,"allocated":823296,"data_stored":823296,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":10,"seq":42949673014,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52297728,"kb_used":244,"kb_used_data":244,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":52297484,"statfs":{"total":53552873472,"available":53552623616,"internally_reserved":0,"allocated":249856,"data_stored":249856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":10,"seq":42949673014,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52297728,"kb_used":280,"kb_used_data":280,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":52297448,"statfs":{"total":53552873472,"available":53552586752,"internally_reserved":0,"allocated":286720,"data_stored":286720,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705718,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52297728,"kb_used":280,"kb_used_data":280,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":52297448,"statfs":{"total":53552873472,"available":53552586752,"internally_reserved":0,"allocated":286720,"data_stored":286720,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":53552873472,"available":53552586752,"internally_reserved":0,"allocated":286720,"data_stored":286720,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":53552873472,"available":53552586752,"internally_reserved":0,"allocated":286720,"data_stored":286720,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":53552873472,"available":53552623616,"internally_reserved":0,"allocated":249856,"data_stored":249856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-23T11:15:35.575 DEBUG:teuthology.orchestra.run.trial191:> sudo ceph --cluster ceph config set global mon_health_to_clog false 2026-02-23T11:15:35.651 INFO:teuthology.orchestra.run.trial191.stderr:2026-02-23T11:15:35.645+0000 7f5dc225a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T11:15:35.653 INFO:teuthology.orchestra.run.trial191.stderr:2026-02-23T11:15:35.652+0000 7f5dc225a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T11:15:35.780 INFO:teuthology.misc:Shutting down mds daemons... 2026-02-23T11:15:35.780 INFO:teuthology.misc:Shutting down osd daemons... 2026-02-23T11:15:35.780 DEBUG:tasks.ceph.osd.0:waiting for process to exit 2026-02-23T11:15:35.780 INFO:teuthology.orchestra.run:waiting for 300 2026-02-23T11:15:35.826 INFO:tasks.ceph.osd.0:Stopped 2026-02-23T11:15:35.826 DEBUG:tasks.ceph.osd.1:waiting for process to exit 2026-02-23T11:15:35.826 INFO:teuthology.orchestra.run:waiting for 300 2026-02-23T11:15:35.875 INFO:tasks.ceph.osd.1:Stopped 2026-02-23T11:15:35.875 DEBUG:tasks.ceph.osd.2:waiting for process to exit 2026-02-23T11:15:35.875 INFO:teuthology.orchestra.run:waiting for 300 2026-02-23T11:15:35.935 INFO:tasks.ceph.osd.2:Stopped 2026-02-23T11:15:35.935 INFO:teuthology.misc:Shutting down mgr daemons... 2026-02-23T11:15:35.936 DEBUG:tasks.ceph.mgr.x:waiting for process to exit 2026-02-23T11:15:35.936 INFO:teuthology.orchestra.run:waiting for 300 2026-02-23T11:15:35.966 INFO:tasks.ceph.mgr.x:Stopped 2026-02-23T11:15:35.966 INFO:teuthology.misc:Shutting down mon daemons... 2026-02-23T11:15:35.967 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2026-02-23T11:15:35.967 INFO:teuthology.orchestra.run:waiting for 300 2026-02-23T11:15:36.016 INFO:tasks.ceph.mon.a:Stopped 2026-02-23T11:15:36.016 DEBUG:tasks.ceph.mon.b:waiting for process to exit 2026-02-23T11:15:36.016 INFO:teuthology.orchestra.run:waiting for 300 2026-02-23T11:15:36.064 INFO:tasks.ceph.mon.b:Stopped 2026-02-23T11:15:36.064 DEBUG:tasks.ceph.mon.c:waiting for process to exit 2026-02-23T11:15:36.065 INFO:teuthology.orchestra.run:waiting for 300 2026-02-23T11:15:36.114 INFO:tasks.ceph.mon.c:Stopped 2026-02-23T11:15:36.115 INFO:tasks.ceph:Checking cluster log for badness... 2026-02-23T11:15:36.115 DEBUG:teuthology.orchestra.run.trial191:> sudo grep -E '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/ceph.log | grep -E -v 'but it is still running' | grep -E -v 'overall HEALTH_' | grep -E -v '\(MDS_ALL_DOWN\)' | grep -E -v '\(MDS_UP_LESS_THAN_MAX\)' | grep -E -v '\(POOL_APP_NOT_ENABLED\)' | head -n 1 2026-02-23T11:15:36.180 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-0 on ubuntu@trial191.front.sepia.ceph.com 2026-02-23T11:15:36.181 DEBUG:teuthology.orchestra.run.trial191:> sync && sudo umount -f /var/lib/ceph/osd/ceph-0 2026-02-23T11:15:36.251 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-1 on ubuntu@trial191.front.sepia.ceph.com 2026-02-23T11:15:36.252 DEBUG:teuthology.orchestra.run.trial191:> sync && sudo umount -f /var/lib/ceph/osd/ceph-1 2026-02-23T11:15:36.370 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-2 on ubuntu@trial191.front.sepia.ceph.com 2026-02-23T11:15:36.370 DEBUG:teuthology.orchestra.run.trial191:> sync && sudo umount -f /var/lib/ceph/osd/ceph-2 2026-02-23T11:15:36.397 INFO:tasks.ceph:Archiving mon data... 2026-02-23T11:15:36.399 DEBUG:teuthology.misc:Transferring archived files from trial191:/var/lib/ceph/mon/ceph-a to /home/teuthworker/mnt/teuthology/shraddhaag-2026-02-23_09:58:48-crimson-rados-main-distro-debug-trial/66185/data/mon.a.tgz 2026-02-23T11:15:36.399 DEBUG:teuthology.orchestra.run.trial191:> mktemp 2026-02-23T11:15:36.452 INFO:teuthology.orchestra.run.trial191.stdout:/tmp/tmp.wAGpDyxJDj 2026-02-23T11:15:36.452 DEBUG:teuthology.orchestra.run.trial191:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-a -- . > /tmp/tmp.wAGpDyxJDj 2026-02-23T11:15:36.534 DEBUG:teuthology.orchestra.run.trial191:> sudo chmod 0666 /tmp/tmp.wAGpDyxJDj 2026-02-23T11:15:36.659 DEBUG:teuthology.orchestra.remote:trial191:/tmp/tmp.wAGpDyxJDj is 138KB 2026-02-23T11:15:36.711 DEBUG:teuthology.orchestra.run.trial191:> rm -fr /tmp/tmp.wAGpDyxJDj 2026-02-23T11:15:36.727 DEBUG:teuthology.misc:Transferring archived files from trial191:/var/lib/ceph/mon/ceph-b to /home/teuthworker/mnt/teuthology/shraddhaag-2026-02-23_09:58:48-crimson-rados-main-distro-debug-trial/66185/data/mon.b.tgz 2026-02-23T11:15:36.728 DEBUG:teuthology.orchestra.run.trial191:> mktemp 2026-02-23T11:15:36.782 INFO:teuthology.orchestra.run.trial191.stdout:/tmp/tmp.bUtjzrFSvP 2026-02-23T11:15:36.783 DEBUG:teuthology.orchestra.run.trial191:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-b -- . > /tmp/tmp.bUtjzrFSvP 2026-02-23T11:15:36.867 DEBUG:teuthology.orchestra.run.trial191:> sudo chmod 0666 /tmp/tmp.bUtjzrFSvP 2026-02-23T11:15:36.987 DEBUG:teuthology.orchestra.remote:trial191:/tmp/tmp.bUtjzrFSvP is 192KB 2026-02-23T11:15:37.034 DEBUG:teuthology.orchestra.run.trial191:> rm -fr /tmp/tmp.bUtjzrFSvP 2026-02-23T11:15:37.049 DEBUG:teuthology.misc:Transferring archived files from trial191:/var/lib/ceph/mon/ceph-c to /home/teuthworker/mnt/teuthology/shraddhaag-2026-02-23_09:58:48-crimson-rados-main-distro-debug-trial/66185/data/mon.c.tgz 2026-02-23T11:15:37.049 DEBUG:teuthology.orchestra.run.trial191:> mktemp 2026-02-23T11:15:37.105 INFO:teuthology.orchestra.run.trial191.stdout:/tmp/tmp.9aRzmEewTI 2026-02-23T11:15:37.105 DEBUG:teuthology.orchestra.run.trial191:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-c -- . > /tmp/tmp.9aRzmEewTI 2026-02-23T11:15:37.190 DEBUG:teuthology.orchestra.run.trial191:> sudo chmod 0666 /tmp/tmp.9aRzmEewTI 2026-02-23T11:15:37.312 DEBUG:teuthology.orchestra.remote:trial191:/tmp/tmp.9aRzmEewTI is 192KB 2026-02-23T11:15:37.360 DEBUG:teuthology.orchestra.run.trial191:> rm -fr /tmp/tmp.9aRzmEewTI 2026-02-23T11:15:37.376 INFO:tasks.ceph:Cleaning ceph cluster... 2026-02-23T11:15:37.376 DEBUG:teuthology.orchestra.run.trial191:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2026-02-23T11:15:37.495 INFO:teuthology.util.scanner:summary_data or yaml_file is empty! 2026-02-23T11:15:37.495 INFO:tasks.ceph:Archiving crash dumps... 2026-02-23T11:15:37.498 DEBUG:teuthology.misc:Transferring archived files from trial191:/var/lib/ceph/crash to /home/teuthworker/mnt/teuthology/shraddhaag-2026-02-23_09:58:48-crimson-rados-main-distro-debug-trial/66185/remote/trial191/crash 2026-02-23T11:15:37.499 DEBUG:teuthology.orchestra.run.trial191:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2026-02-23T11:15:37.558 INFO:tasks.ceph:Compressing logs... 2026-02-23T11:15:37.558 DEBUG:teuthology.orchestra.run.trial191:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2026-02-23T11:15:37.581 INFO:teuthology.orchestra.run.trial191.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40162.log 2026-02-23T11:15:37.581 INFO:teuthology.orchestra.run.trial191.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.2.log 2026-02-23T11:15:37.598 INFO:teuthology.orchestra.run.trial191.stderr:/var/log/ceph/ceph-client.admin.40162.log: gzip -5 --verbose -- /var/log/ceph/ceph-mon.a.log 2026-02-23T11:15:37.599 INFO:teuthology.orchestra.run.trial191.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.40162.log.gz 2026-02-23T11:15:37.599 INFO:teuthology.orchestra.run.trial191.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40887.log 2026-02-23T11:15:37.599 INFO:teuthology.orchestra.run.trial191.stderr:/var/log/ceph/ceph-osd.2.log: /var/log/ceph/ceph-mon.a.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41522.log 2026-02-23T11:15:37.599 INFO:teuthology.orchestra.run.trial191.stderr:/var/log/ceph/ceph-client.admin.40887.log: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.40887.log.gz 2026-02-23T11:15:37.599 INFO:teuthology.orchestra.run.trial191.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.0.log 2026-02-23T11:15:37.599 INFO:teuthology.orchestra.run.trial191.stderr:/var/log/ceph/ceph-client.admin.41522.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41227.log 2026-02-23T11:15:37.599 INFO:teuthology.orchestra.run.trial191.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.41522.log.gz 2026-02-23T11:15:37.599 INFO:teuthology.orchestra.run.trial191.stderr:/var/log/ceph/ceph-osd.0.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42756.log 2026-02-23T11:15:37.599 INFO:teuthology.orchestra.run.trial191.stderr:/var/log/ceph/ceph-client.admin.41227.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42609.log 2026-02-23T11:15:37.599 INFO:teuthology.orchestra.run.trial191.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.41227.log.gz 2026-02-23T11:15:37.599 INFO:teuthology.orchestra.run.trial191.stderr:/var/log/ceph/ceph-client.admin.42756.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40284.log 2026-02-23T11:15:37.600 INFO:teuthology.orchestra.run.trial191.stderr: 50.8% -- replaced with /var/log/ceph/ceph-client.admin.42756.log.gz 2026-02-23T11:15:37.600 INFO:teuthology.orchestra.run.trial191.stderr:/var/log/ceph/ceph-client.admin.42609.log: gzip -5 --verbose -- /var/log/ceph/ceph.tmp-client.admin.38363.log 2026-02-23T11:15:37.600 INFO:teuthology.orchestra.run.trial191.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.42609.log.gz 2026-02-23T11:15:37.600 INFO:teuthology.orchestra.run.trial191.stderr:/var/log/ceph/ceph-client.admin.40284.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42707.log 2026-02-23T11:15:37.600 INFO:teuthology.orchestra.run.trial191.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.40284.log.gz 2026-02-23T11:15:37.600 INFO:teuthology.orchestra.run.trial191.stderr:/var/log/ceph/ceph.tmp-client.admin.38363.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41125.log 67.5% 2026-02-23T11:15:37.600 INFO:teuthology.orchestra.run.trial191.stderr: -- replaced with /var/log/ceph/ceph.tmp-client.admin.38363.log.gz 2026-02-23T11:15:37.600 INFO:teuthology.orchestra.run.trial191.stderr:/var/log/ceph/ceph-client.admin.42707.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41642.log 2026-02-23T11:15:37.600 INFO:teuthology.orchestra.run.trial191.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.42707.log.gz 2026-02-23T11:15:37.600 INFO:teuthology.orchestra.run.trial191.stderr:/var/log/ceph/ceph-client.admin.41125.log: gzip -5 --verbose -- /var/log/ceph/ceph-mon.c.log 2026-02-23T11:15:37.600 INFO:teuthology.orchestra.run.trial191.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.41125.log.gz 2026-02-23T11:15:37.600 INFO:teuthology.orchestra.run.trial191.stderr:/var/log/ceph/ceph-client.admin.41642.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40817.log 2026-02-23T11:15:37.600 INFO:teuthology.orchestra.run.trial191.stderr: 49.2% -- replaced with /var/log/ceph/ceph-client.admin.41642.log.gz 2026-02-23T11:15:37.601 INFO:teuthology.orchestra.run.trial191.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42805.log 2026-02-23T11:15:37.601 INFO:teuthology.orchestra.run.trial191.stderr:/var/log/ceph/ceph-mon.c.log: /var/log/ceph/ceph-client.admin.40817.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40705.log 2026-02-23T11:15:37.601 INFO:teuthology.orchestra.run.trial191.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.40817.log.gz 2026-02-23T11:15:37.601 INFO:teuthology.orchestra.run.trial191.stderr:/var/log/ceph/ceph-client.admin.42805.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41076.log 2026-02-23T11:15:37.601 INFO:teuthology.orchestra.run.trial191.stderr: 50.8% -- replaced with /var/log/ceph/ceph-client.admin.42805.log.gz 2026-02-23T11:15:37.601 INFO:teuthology.orchestra.run.trial191.stderr:/var/log/ceph/ceph-client.admin.40705.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42413.log 2026-02-23T11:15:37.601 INFO:teuthology.orchestra.run.trial191.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.40705.log.gz 2026-02-23T11:15:37.601 INFO:teuthology.orchestra.run.trial191.stderr:/var/log/ceph/ceph-client.admin.41076.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40382.log 2026-02-23T11:15:37.601 INFO:teuthology.orchestra.run.trial191.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.41076.log.gz 2026-02-23T11:15:37.601 INFO:teuthology.orchestra.run.trial191.stderr:/var/log/ceph/ceph-client.admin.42413.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40765.log 2026-02-23T11:15:37.601 INFO:teuthology.orchestra.run.trial191.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.42413.log.gz 2026-02-23T11:15:37.601 INFO:teuthology.orchestra.run.trial191.stderr:/var/log/ceph/ceph-client.admin.40382.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41027.log 2026-02-23T11:15:37.602 INFO:teuthology.orchestra.run.trial191.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.40382.log.gz 2026-02-23T11:15:37.602 INFO:teuthology.orchestra.run.trial191.stderr:/var/log/ceph/ceph-client.admin.40765.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39854.log 2026-02-23T11:15:37.602 INFO:teuthology.orchestra.run.trial191.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.40765.log.gz 2026-02-23T11:15:37.602 INFO:teuthology.orchestra.run.trial191.stderr:/var/log/ceph/ceph-client.admin.41027.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40115.log 2026-02-23T11:15:37.602 INFO:teuthology.orchestra.run.trial191.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.41027.log.gz 2026-02-23T11:15:37.602 INFO:teuthology.orchestra.run.trial191.stderr:/var/log/ceph/ceph-client.admin.39854.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41669.log 2026-02-23T11:15:37.602 INFO:teuthology.orchestra.run.trial191.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.39854.log.gz 2026-02-23T11:15:37.602 INFO:teuthology.orchestra.run.trial191.stderr:/var/log/ceph/ceph-client.admin.40115.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42140.log 2026-02-23T11:15:37.602 INFO:teuthology.orchestra.run.trial191.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.40115.log.gz 2026-02-23T11:15:37.602 INFO:teuthology.orchestra.run.trial191.stderr:/var/log/ceph/ceph-client.admin.41669.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42164.log 2026-02-23T11:15:37.602 INFO:teuthology.orchestra.run.trial191.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.41669.log.gz 2026-02-23T11:15:37.602 INFO:teuthology.orchestra.run.trial191.stderr:/var/log/ceph/ceph-client.admin.42140.log: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.42140.log.gz 2026-02-23T11:15:37.602 INFO:teuthology.orchestra.run.trial191.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2026-02-23T11:15:37.603 INFO:teuthology.orchestra.run.trial191.stderr:/var/log/ceph/ceph-client.admin.42164.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40978.log 2026-02-23T11:15:37.603 INFO:teuthology.orchestra.run.trial191.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.42164.log.gz 2026-02-23T11:15:37.603 INFO:teuthology.orchestra.run.trial191.stderr:/var/log/ceph/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/ceph-mon.b.log 2026-02-23T11:15:37.603 INFO:teuthology.orchestra.run.trial191.stderr:/var/log/ceph/ceph-client.admin.40978.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42854.log 2026-02-23T11:15:37.603 INFO:teuthology.orchestra.run.trial191.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.40978.log.gz 2026-02-23T11:15:37.603 INFO:teuthology.orchestra.run.trial191.stderr:/var/log/ceph/ceph-mon.b.log: 94.4% -- replaced with /var/log/ceph/ceph.audit.log.gz 2026-02-23T11:15:37.603 INFO:teuthology.orchestra.run.trial191.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40044.log 2026-02-23T11:15:37.603 INFO:teuthology.orchestra.run.trial191.stderr:/var/log/ceph/ceph-client.admin.42854.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41762.log 2026-02-23T11:15:37.603 INFO:teuthology.orchestra.run.trial191.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.42854.log.gz 2026-02-23T11:15:37.603 INFO:teuthology.orchestra.run.trial191.stderr:/var/log/ceph/ceph-client.admin.40044.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.1.log 2026-02-23T11:15:37.603 INFO:teuthology.orchestra.run.trial191.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.40044.log.gz 2026-02-23T11:15:37.603 INFO:teuthology.orchestra.run.trial191.stderr:/var/log/ceph/ceph-client.admin.41762.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42462.log 2026-02-23T11:15:37.604 INFO:teuthology.orchestra.run.trial191.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.41762.log.gz 2026-02-23T11:15:37.604 INFO:teuthology.orchestra.run.trial191.stderr:/var/log/ceph/ceph-osd.1.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42511.log 2026-02-23T11:15:37.604 INFO:teuthology.orchestra.run.trial191.stderr:/var/log/ceph/ceph-client.admin.42462.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42315.log 2026-02-23T11:15:37.604 INFO:teuthology.orchestra.run.trial191.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.42462.log.gz 2026-02-23T11:15:37.604 INFO:teuthology.orchestra.run.trial191.stderr:/var/log/ceph/ceph-client.admin.42511.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.42511.log.gz 2026-02-23T11:15:37.604 INFO:teuthology.orchestra.run.trial191.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41811.log 2026-02-23T11:15:37.604 INFO:teuthology.orchestra.run.trial191.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40597.log 2026-02-23T11:15:37.604 INFO:teuthology.orchestra.run.trial191.stderr:/var/log/ceph/ceph-client.admin.42315.log: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.42315.log.gz 2026-02-23T11:15:37.604 INFO:teuthology.orchestra.run.trial191.stderr:/var/log/ceph/ceph-client.admin.41811.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41494.log 2026-02-23T11:15:37.604 INFO:teuthology.orchestra.run.trial191.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.41811.log.gz 2026-02-23T11:15:37.604 INFO:teuthology.orchestra.run.trial191.stderr:/var/log/ceph/ceph-client.admin.40597.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41373.log 2026-02-23T11:15:37.604 INFO:teuthology.orchestra.run.trial191.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.40597.log.gz 2026-02-23T11:15:37.604 INFO:teuthology.orchestra.run.trial191.stderr:/var/log/ceph/ceph-client.admin.41494.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41375.log 2026-02-23T11:15:37.605 INFO:teuthology.orchestra.run.trial191.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.41494.log.gz 2026-02-23T11:15:37.605 INFO:teuthology.orchestra.run.trial191.stderr:/var/log/ceph/ceph-client.admin.41373.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42364.log 2026-02-23T11:15:37.605 INFO:teuthology.orchestra.run.trial191.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.41373.log.gz 2026-02-23T11:15:37.605 INFO:teuthology.orchestra.run.trial191.stderr:/var/log/ceph/ceph-client.admin.41375.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41641.log 2026-02-23T11:15:37.605 INFO:teuthology.orchestra.run.trial191.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.41375.log.gz 2026-02-23T11:15:37.605 INFO:teuthology.orchestra.run.trial191.stderr:/var/log/ceph/ceph-client.admin.42364.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40912.log 2026-02-23T11:15:37.605 INFO:teuthology.orchestra.run.trial191.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.42364.log.gz 2026-02-23T11:15:37.605 INFO:teuthology.orchestra.run.trial191.stderr:/var/log/ceph/ceph-client.admin.41641.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42901.log 2026-02-23T11:15:37.605 INFO:teuthology.orchestra.run.trial191.stderr: 49.2% -- replaced with /var/log/ceph/ceph-client.admin.41641.log.gz 2026-02-23T11:15:37.605 INFO:teuthology.orchestra.run.trial191.stderr:/var/log/ceph/ceph-client.admin.40912.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41228.log 2026-02-23T11:15:37.605 INFO:teuthology.orchestra.run.trial191.stderr: 66.7% -- replaced with /var/log/ceph/ceph-client.admin.40912.log.gz 2026-02-23T11:15:37.605 INFO:teuthology.orchestra.run.trial191.stderr:/var/log/ceph/ceph-client.admin.42901.log: gzip -5 --verbose -- /var/log/ceph/ceph-mgr.x.log 2026-02-23T11:15:37.605 INFO:teuthology.orchestra.run.trial191.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.42901.log.gz 2026-02-23T11:15:37.606 INFO:teuthology.orchestra.run.trial191.stderr:/var/log/ceph/ceph-client.admin.41228.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41226.log 2026-02-23T11:15:37.606 INFO:teuthology.orchestra.run.trial191.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.41228.log.gz 2026-02-23T11:15:37.606 INFO:teuthology.orchestra.run.trial191.stderr:/var/log/ceph/ceph-mgr.x.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41374.log 2026-02-23T11:15:37.606 INFO:teuthology.orchestra.run.trial191.stderr:/var/log/ceph/ceph-client.admin.41226.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42658.log 2026-02-23T11:15:37.606 INFO:teuthology.orchestra.run.trial191.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.41226.log.gz 2026-02-23T11:15:37.606 INFO:teuthology.orchestra.run.trial191.stderr:/var/log/ceph/ceph-client.admin.41374.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42560.log 2026-02-23T11:15:37.606 INFO:teuthology.orchestra.run.trial191.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.41374.log.gz 2026-02-23T11:15:37.606 INFO:teuthology.orchestra.run.trial191.stderr:/var/log/ceph/ceph-client.admin.42658.log: gzip -5 --verbose -- /var/log/ceph/ceph.log 2026-02-23T11:15:37.606 INFO:teuthology.orchestra.run.trial191.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.42658.log.gz 2026-02-23T11:15:37.606 INFO:teuthology.orchestra.run.trial191.stderr:/var/log/ceph/ceph-client.admin.42560.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41860.log 2026-02-23T11:15:37.606 INFO:teuthology.orchestra.run.trial191.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.42560.log.gz 2026-02-23T11:15:37.606 INFO:teuthology.orchestra.run.trial191.stderr:/var/log/ceph/ceph.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41495.log 2026-02-23T11:15:37.606 INFO:teuthology.orchestra.run.trial191.stderr: 93.2% -- replaced with /var/log/ceph/ceph.log.gz 2026-02-23T11:15:37.607 INFO:teuthology.orchestra.run.trial191.stderr:/var/log/ceph/ceph-client.admin.41860.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40331.log 2026-02-23T11:15:37.607 INFO:teuthology.orchestra.run.trial191.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.41860.log.gz 2026-02-23T11:15:37.607 INFO:teuthology.orchestra.run.trial191.stderr:/var/log/ceph/ceph-client.admin.41495.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.41495.log.gz 2026-02-23T11:15:37.607 INFO:teuthology.orchestra.run.trial191.stderr:/var/log/ceph/ceph-client.admin.40331.log: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.40331.log.gz 2026-02-23T11:15:37.607 INFO:teuthology.orchestra.run.trial191.stderr: 91.6% -- replaced with /var/log/ceph/ceph-mgr.x.log.gz 2026-02-23T11:15:37.608 INFO:teuthology.orchestra.run.trial191.stderr: 87.9% -- replaced with /var/log/ceph/ceph-mon.c.log.gz 2026-02-23T11:15:37.609 INFO:teuthology.orchestra.run.trial191.stderr: 87.7% -- replaced with /var/log/ceph/ceph-mon.b.log.gz 2026-02-23T11:15:37.619 INFO:teuthology.orchestra.run.trial191.stderr: 88.5% -- replaced with /var/log/ceph/ceph-mon.a.log.gz 2026-02-23T11:15:37.694 INFO:teuthology.orchestra.run.trial191.stderr: 95.0% -- replaced with /var/log/ceph/ceph-osd.2.log.gz 2026-02-23T11:15:37.698 INFO:teuthology.orchestra.run.trial191.stderr: 95.0% -- replaced with /var/log/ceph/ceph-osd.0.log.gz 2026-02-23T11:15:37.705 INFO:teuthology.orchestra.run.trial191.stderr: 95.0% -- replaced with /var/log/ceph/ceph-osd.1.log.gz 2026-02-23T11:15:37.707 INFO:teuthology.orchestra.run.trial191.stderr: 2026-02-23T11:15:37.707 INFO:teuthology.orchestra.run.trial191.stderr:real 0m0.134s 2026-02-23T11:15:37.707 INFO:teuthology.orchestra.run.trial191.stderr:user 0m0.420s 2026-02-23T11:15:37.707 INFO:teuthology.orchestra.run.trial191.stderr:sys 0m0.059s 2026-02-23T11:15:37.708 INFO:tasks.ceph:Archiving logs... 2026-02-23T11:15:37.708 DEBUG:teuthology.misc:Transferring archived files from trial191:/var/log/ceph to /home/teuthworker/mnt/teuthology/shraddhaag-2026-02-23_09:58:48-crimson-rados-main-distro-debug-trial/66185/remote/trial191/log 2026-02-23T11:15:37.710 DEBUG:teuthology.orchestra.run.trial191:> sudo tar c -f - -C /var/log/ceph -- . 2026-02-23T11:15:38.015 DEBUG:teuthology.run_tasks:Unwinding manager install 2026-02-23T11:15:38.117 DEBUG:teuthology.run_tasks:Unwinding manager install 2026-02-23T11:15:38.130 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2026-02-23T11:15:38.130 DEBUG:teuthology.orchestra.run.trial191:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2026-02-23T11:15:38.156 DEBUG:teuthology.run_tasks:Unwinding manager clock 2026-02-23T11:15:38.168 INFO:teuthology.task.clock:Checking final clock skew... 2026-02-23T11:15:38.168 DEBUG:teuthology.orchestra.run.trial191:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2026-02-23T11:15:38.212 INFO:teuthology.orchestra.run.trial191.stderr:bash: line 1: ntpq: command not found 2026-02-23T11:15:38.233 INFO:teuthology.orchestra.run.trial191.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2026-02-23T11:15:38.233 INFO:teuthology.orchestra.run.trial191.stdout:=============================================================================== 2026-02-23T11:15:38.234 INFO:teuthology.orchestra.run.trial191.stdout:^- time4.lshiy.com 2 6 377 16 +9586us[+9586us] +/- 133ms 2026-02-23T11:15:38.234 INFO:teuthology.orchestra.run.trial191.stdout:^* pool-138-89-14-60.mad.ea> 1 6 377 17 -13us[ -16us] +/- 2889us 2026-02-23T11:15:38.234 INFO:teuthology.orchestra.run.trial191.stdout:^- beta.kenyonralph.com 3 6 377 14 -1268us[-1268us] +/- 34ms 2026-02-23T11:15:38.234 INFO:teuthology.orchestra.run.trial191.stdout:^+ ntp0.alb1.inoc.net 2 6 377 17 +4319ns[+4319ns] +/- 3943us 2026-02-23T11:15:38.235 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2026-02-23T11:15:38.248 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2026-02-23T11:15:38.249 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2026-02-23T11:15:38.265 DEBUG:teuthology.orchestra.run.trial191:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2026-02-23T11:15:38.331 DEBUG:teuthology.orchestra.run.trial191:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 2026-02-23T11:15:38.357 DEBUG:teuthology.orchestra.run:got remote process result: 1 2026-02-23T11:15:38.357 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2026-02-23T11:15:38.373 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2026-02-23T11:15:38.397 INFO:teuthology.task.internal:Duration was 577.706807 seconds 2026-02-23T11:15:38.398 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2026-02-23T11:15:38.409 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2026-02-23T11:15:38.409 DEBUG:teuthology.orchestra.run.trial191:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2026-02-23T11:15:38.447 INFO:teuthology.orchestra.run.trial191.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2026-02-23T11:15:38.681 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2026-02-23T11:15:38.681 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@trial191.front.sepia.ceph.com 2026-02-23T11:15:38.682 DEBUG:teuthology.orchestra.run.trial191:> grep -E --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | grep -E -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | grep -E -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2026-02-23T11:15:38.740 INFO:teuthology.task.internal.syslog:Gathering journactl... 2026-02-23T11:15:38.741 DEBUG:teuthology.orchestra.run.trial191:> sudo journalctl > /home/ubuntu/cephtest/archive/syslog/journalctl.log 2026-02-23T11:15:38.843 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2026-02-23T11:15:38.843 DEBUG:teuthology.orchestra.run.trial191:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --max-args=1 --max-procs=0 --verbose --no-run-if-empty -- gzip -5 --verbose -- 2026-02-23T11:15:38.866 INFO:teuthology.orchestra.run.trial191.stderr:gzip -5 --verbose -- /home/ubuntu/cephtest/archive/syslog/misc.log 2026-02-23T11:15:38.867 INFO:teuthology.orchestra.run.trial191.stderr:gzip -5 --verbose -- /home/ubuntu/cephtest/archive/syslog/kern.log 2026-02-23T11:15:38.867 INFO:teuthology.orchestra.run.trial191.stderr:/home/ubuntu/cephtest/archive/syslog/misc.log: gzip -5 --verbose -- /home/ubuntu/cephtest/archive/syslog/journalctl.log 2026-02-23T11:15:38.867 INFO:teuthology.orchestra.run.trial191.stderr: 0.0% -- replaced with /home/ubuntu/cephtest/archive/syslog/misc.log.gz 2026-02-23T11:15:38.867 INFO:teuthology.orchestra.run.trial191.stderr:/home/ubuntu/cephtest/archive/syslog/kern.log: 0.0% -- replaced with /home/ubuntu/cephtest/archive/syslog/kern.log.gz 2026-02-23T11:15:38.878 INFO:teuthology.orchestra.run.trial191.stderr:/home/ubuntu/cephtest/archive/syslog/journalctl.log: 81.9% -- replaced with /home/ubuntu/cephtest/archive/syslog/journalctl.log.gz 2026-02-23T11:15:38.880 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2026-02-23T11:15:38.891 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2026-02-23T11:15:38.891 DEBUG:teuthology.orchestra.run.trial191:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2026-02-23T11:15:38.943 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2026-02-23T11:15:38.960 DEBUG:teuthology.orchestra.run.trial191:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2026-02-23T11:15:39.005 INFO:teuthology.orchestra.run.trial191.stdout:kernel.core_pattern = core 2026-02-23T11:15:39.021 DEBUG:teuthology.orchestra.run.trial191:> test -e /home/ubuntu/cephtest/archive/coredump 2026-02-23T11:15:39.078 DEBUG:teuthology.orchestra.run:got remote process result: 1 2026-02-23T11:15:39.078 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2026-02-23T11:15:39.095 INFO:teuthology.task.internal:Transferring archived files... 2026-02-23T11:15:39.096 DEBUG:teuthology.misc:Transferring archived files from trial191:/home/ubuntu/cephtest/archive to /home/teuthworker/mnt/teuthology/shraddhaag-2026-02-23_09:58:48-crimson-rados-main-distro-debug-trial/66185/remote/trial191 2026-02-23T11:15:39.097 DEBUG:teuthology.orchestra.run.trial191:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2026-02-23T11:15:39.155 INFO:teuthology.task.internal:Removing archive directory... 2026-02-23T11:15:39.155 DEBUG:teuthology.orchestra.run.trial191:> rm -rf -- /home/ubuntu/cephtest/archive 2026-02-23T11:15:39.202 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2026-02-23T11:15:39.230 INFO:teuthology.task.internal:Not uploading archives. 2026-02-23T11:15:39.230 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2026-02-23T11:15:39.247 INFO:teuthology.task.internal:Tidying up after the test... 2026-02-23T11:15:39.248 DEBUG:teuthology.orchestra.run.trial191:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2026-02-23T11:15:39.266 INFO:teuthology.orchestra.run.trial191.stdout: 917522 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Feb 23 11:15 /home/ubuntu/cephtest 2026-02-23T11:15:39.267 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2026-02-23T11:15:39.284 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2026-02-23T11:15:39.323 INFO:teuthology.run:Summary data: description: crimson-rados/singleton/{0-crimson_install all/osd-dump-metrics crimson-supported-all-distro/centos_latest crimson_qa_overrides objectstore/seastore/segmented$/{crimson_seastore_segmented_2q} rados} duration: 577.7068068981171 flavor: debug owner: scheduled_shraddhaag@soko04.front.sepia.ceph.com success: true 2026-02-23T11:15:39.323 DEBUG:teuthology.report:Pushing job info to https://paddles-paddles.apps.pok.os.sepia.ceph.com 2026-02-23T11:15:39.446 INFO:teuthology.run:pass